Periodic Tasks and Queue Processing in Django

Periodic Tasks and Queue Processing in Django

In Python/Django Celery with RabbitMQ is widely used for background processing or distributed
task queue. Although Celery is really focused on being a distributed task queue, it can
also be used as scheduler using it's periodic tasks feature celerybeat which kicks
off tasks at regular intervals.


Installing Celery and RabbitMQ is quite easy.

First install RabbitMQ, add a new user and remove guest user. To use celery we need
to create a virtual host and allow new user access to that virtual host:

sudo apt-get install rabbitmq-server
sudo rabbitmqctl add_user new_user new_password
sudo rabbitmqctl delete_user guest
sudo rabbitmqctl add_vhost myvhost
sudo rabbitmqctl set_permissions -p myvhost new_user ".*" ".*" ".*"

If you don't want to use RabbitMQ then you can use Redis, MangoDB or Django Database.
For more information see other broker types.

Then install, Celery and django-celery,

sudo pip install celery django-celery

Using Celery in Djngo

Add djcelery to INSTALLED_APPS. Then add the following lines to

import djcelery

Synchronize the celery database tables::

python syncdb

Configure the broker settings, by adding the following to your

BROKER_HOST = "localhost"
BROKER_USER = "new_user"
BROKER_PASSWORD = "new_password"
BROKER_VHOST = "myvhost"
# List of modules to import when celery starts.
CELERY_IMPORTS = ("myapp.tasks", )

Use corresponding commands to manage the tasks,

python celeryd
python celerybeat
python camqadm
python celeryev

If you are using mod_wsgi in production then you should also add following to .wsgi

import os
os.environ["CELERY_LOADER"] = "django"

Use Cases

1. djnago-notification

djnago-notification [] in blocking mode can make your application very slow. To queue all notifications add following in


Normally you can use,

python emit_signals

2. django-mailer

django-mailer is asynchronous.After putting mail on the queue you need to periodically
tell it to clear the queue and actually send the mail using,

python send_mail

Solution using Celery and RabbitMQ

A better way will be to run Celery tasks in beat mode, so create a in
project root folder (same level to

from celery.task.schedules import crontab
from celery.decorators import periodic_task
# Get django-mailer management function to fire notification queue
from notification.engine import send_all
# Get django-mailer management function to fire mail queue
from mailer.engine import send_all as send_all_mail

# this will run every minute, * is every
@periodic_task(run_every=crontab(hour="*", minute="*", day_of_week="*"))
def project_tasks():    

Now run the celery daemon celeryd in "beat" mode (-B). You can also run celery daemon in
background mode with celeryd_detach

sudo ./ celeryd -v 2 -B -s celery -E -l INFO
sudo ./ celeryd_detach -B -s celery -E -l INFO

Running celeryd as a daemon using init script

Although you can use celeryd_detach to run celeryd daemon in background, but in
production environment you are better with init scripts as you can start and stop
the deamon when you want. Download init script for celeryd and create the file
/etc/init.d/celeryd add content of downloaded script to it and make it

sudo nano /etc/init.d/celeryd
sudo chmod +x /etc/init.d/celeryd
Usage:	/etc/init.d/celeryd {start|stop|restart|status}

Now create configuration file: /etc/default/celeryd and add following,

# Name of nodes to start, here we have a single node
# or we could have multiple nodes:
#CELERYD_NODES="w1 w2 w3"

# Where to chdir at start.

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"

# How to call " celeryd_multi"

# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=8 -B -s celery -E -l INFO"

# Name of the celery config module.

# %n will be replaced with the nodename. You should create corresponding folders

# Workers should run as an unprivileged user. Avoid running as sudo user.

Abhishek Tiwari

Abhishek Tiwari

Tech Savant, Servant Leader

View Comments
Next Post

Scala for the Python Geeks

Previous Post

Knowing your data vs relying on it