Redis Authentication using Scrapy and Python.

by Saahyl

Redis can be instructed to require a password before allowing clients to execute commands. This is done using the requirepass directive in the configuration file. You can read more about redis security from the documentation page http://redis.io/topics/security. 

In this blog, I have outlined how to access password protected redis when working with Scrapy framework in Python.

Prerequisites to work with Scrapy and Redis in python are these two packages, which we need to install:
Redis-py
Scrapy-redis

1) Using only the redis-py package to request for authentication we need to make an object of class redis.Redis

Example:

import redis
connection_object = redis.Redis(host='localhost', port=6379, db=0, password=’redis_server_password’)

This connection_object is used to perform operations on redis like push, pop, etc.

2) Using Scrapy-redis the process of authentication, when scrapy-redis needs to perform operations on a password protected redis is handled using a redis-url scheme provided in scrapy-redis version 0.5 and above.
To enable this we need to add the following attribute in our scrapy project’s settings.py file:

REDIS_URL = 'redis://user:pass@hostname:port'

where,
user -> is redis-server user name
Pass -> is redis-server password
Hostname -> ip address of server where redis is running
Port -> is the port no. on which redis is accepting connections. Default is 6379

Most of the time there is no user created on a redis-server, in which case we will omit user parameter of redis_url.

Example of a redis_url without a user parameter:
REDIS_URL = "redis://:abc123@45.79.199.102:6379"

Note: Do not use @ sign in your redis password because some version of redis-py packages are not able to parse @ inside your password string correctly, this may be because it is the separator between pass and hostname in the redis_url. Don’t confuse yourself if I say redis-py package when using REDIS_URL, it is because scrapy_redis package’s connection.py calls redis-py package’s from_url( ) method of the ConnectionPool class written in connection.py of redis-py package. This is where the password string is parsed.

5 thoughts on “Redis Authentication using Scrapy and Python.”

  1. This post is in fact a nice one it assists new net people, who are wishing in favor of blogging.

  2. Fastidious response in return of this issue with real arguments and explaining everything on the topic of that.

  3. Thanks for a marvelous posting! I certainly enjoyed reading
    it, you will be a great author.I will be sure to bookmark your
    blog and will come back in the foreseeable future. I want to encourage you to ultimately continue your great writing, have a nice weekend!

  4. If some one wishes expert view on the topic of blogging and site-building after that i
    propose him/her to pay a visit this blog, Keep up the
    nice job.

Leave a Reply

Your email address will not be published. Required fields are marked *

Tools & Practices

Tools and Technologies we use at Applied

Contact us now

Popular Posts