Redis Authentication using Scrapy and Python.by Saahyl
Redis can be instructed to require a password before allowing clients to execute commands. This is done using the requirepass directive in the configuration file. You can read more about redis security from the documentation page http://redis.io/topics/security.
In this blog, I have outlined how to access password protected redis when working with Scrapy framework in Python.
1) Using only the redis-py package to request for authentication we need to make an object of class redis.Redis
connection_object = redis.Redis(host='localhost', port=6379, db=0, password=’redis_server_password’)
This connection_object is used to perform operations on redis like push, pop, etc.
2) Using Scrapy-redis the process of authentication, when scrapy-redis needs to perform operations on a password protected redis is handled using a redis-url scheme provided in scrapy-redis version 0.5 and above.
To enable this we need to add the following attribute in our scrapy project’s settings.py file:
REDIS_URL = 'redis://user:pass@hostname:port'
user -> is redis-server user name
Pass -> is redis-server password
Hostname -> ip address of server where redis is running
Port -> is the port no. on which redis is accepting connections. Default is 6379
Most of the time there is no user created on a redis-server, in which case we will omit user parameter of redis_url.
Example of a redis_url without a user parameter:
REDIS_URL = "redis://:firstname.lastname@example.org:6379"
Note: Do not use @ sign in your redis password because some version of redis-py packages are not able to parse @ inside your password string correctly, this may be because it is the separator between pass and hostname in the redis_url. Don’t confuse yourself if I say redis-py package when using REDIS_URL, it is because scrapy_redis package’s connection.py calls redis-py package’s from_url( ) method of the ConnectionPool class written in connection.py of redis-py package. This is where the password string is parsed.