If we imagine that, a simple task like sending emails to all friends of a user in the database involves a long process, it becomes necessary to separate this from the main flow of an application so as not to have user wait for entire process to finish.
Celery allows the addition of asynchronous functionality in Django,
What is Celery?
Celery is an asynchronous task queue/job queue based on distributed message passing. Message passing is a method which program components can use to communicate and exchange information. It can be implemented synchronously or asynchronously and can allow discrete processes to communicate without problems.
Celery requires a solution to send and receive messages, usually this comes in the form of a separate service called a message broker. We will be using Redis as a broker in our example.
Installations
- pip install celery
- pip install -U celery-with-redis
We can also use a monitoring tool for Celery.
- pip install-U flower
This has an inbuilt web server that provides a web based monitoring interface for all the workers that you will be running.
Using celery with Django
First of all, we will add broker settings in settings.py file of the project.
BROKER_URL = 'redis://localhost:6379' CELERY_RESULT_BACKEND = 'redis://localhost:6379' CELERY_ACCEPT_CONTENT = ['application/json'] CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_SERIALIZER = 'json'
To use Celery with your Django project you must first define an instance of Celery library (called an “app”).
The modern Django project layout looks like the below:
- proj/ - proj/__init__.py - proj/settings.py - proj/urls.py - manage.py
The recommended way is to create a new proj/proj/celery.py module that defines the Celery instance:
from __future__ import absolute_import import os from celery import Celery # set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings') from django.conf import settings app = Celery('proj') app.config_from_object('django.conf:settings') app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) @app.task(bind=True) def debug_task(self): print('Request: {0!r}'.format(self.request))
In order to initiate Celery app every time our Django application is started, we can add the following lines to the __init__.py file in our Django proj.proj module,
proj/proj/__init__.py:
from __future__ import absolute_import from .celery import app as celery_app
Now, we will add a sendemailapp to our Django project and add a celery task to it. This can be done by the below command,
python manage.py startapp sendemailapp
After adding this app, we have to make sure this is added in INSTALLED_APPS in proj/settings.py.
Now our django project structure is like this
proj ├── proj │ ├── celery.py │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ ├── manage.py └── sendemailapp ├── __init__.py ├── models.py ├── tasks.py ├── tests.py └── views.py # views.py #This view gets called when the user clicks a button def myview(request): # Schedule a task to send email later user_id = request.user.id send_email_to_users.delay(user_id) return render_to_response(...) # tasks.py @task def send_email_to_users(user_id): user = User.objects.get(id=user_id) friends = #get emails of all friends of user from db #sendemail to those friends
And we are done. We have implemented the functionality where we send email to all friends of a user asynchronously.
For more information about Celery please visit celeryproject.org.
Asynchronous, or non-blocking, processing is a method of separating the execution of certain tasks from the main flow of a program. This provides you with several advantages, including allowing your user-facing code to run without interruption.