Tag python

Celery with RabbitMQ 3.1.0 on Webfaction

A while ago Mark Liu wrote about setting up django-celery on a Webfaction host. I took the tutorial and matched it to the current version of RabbitMQ that is 3.1.0.

Please read this article entirely before installing anything!

Basically follow Mark's tutorial and install Erlang as well as RabbitMQ but use the current RabbitMQ binary *.tar.gz from the download page.

Attention: Depending on what applications you are already running on Webfaction, the 256MB RAM plan may be insufficient to run Celery and RabbitMQ and Erlang. You can give it a try, for me it did not work, I added some more memory to my plan. Monitor your memory usage while installing. I'm using the following script to do so (replace the [username] with your username):

# memory_top.sh
# lists memory usage for given user every 5 seconds
while true
do
    ps -U [username] --no-headers -o rss | (tr '\n' +; echo 0) | bc
    sleep 5
done

Do not configure rabbitmq-server and the rabbitmq-env file as written. Instead do the following (I extracted RabbitMQ to \$HOME/rabbitmq/rabbitmq_server-3.1.0 and will use this paths):

Edit \$HOME/rabbitmq/rabbitmq_server-3.1.0/sbin/rabbitmq-defaults:

...
# comment these lines:
#CONFIG_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq
#LOG_BASE=${SYS_PREFIX}/var/log/rabbitmq
#MNESIA_BASE=${SYS_PREFIX}/var/lib/rabbitmq/mnesia
# add these lines:
CONFIG_FILE=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/
LOG_BASE=/home/[username]/logs/user/rabbitmq # create this directory!
MNESIA_BASE=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/
...

Edit \$HOME/rabbitmq/rabbitmq_server-3.1.0/sbin/rabbitmq-env:

...
# add to the end
export ERL_EPMD_PORT=[Erlang-Custom-App-Port]
export RABBITMQ_NODE_PORT=[RabbitMQ-Custom-App-Port]
export ERL_INETRC=$HOME/.erl_inetrc

Create the \$HOME/hosts and \$HOME/.erl_inetrc files as written in Mark's blog and continue as he wrote.

After testing Erlang and Rabbit kill the processes to integrate them into supervisord. I use it to control the processes, you can simply install it into your global Webfaction Python interpreter.
Here are the (quite simple) configurations (adjust the paths to your environment):

erlang.conf

[program:erlang]
command=epmd -port [erlang-port]
user=[username]
autorestart=true
directory=/home/[username]/
stdout_logfile=/home/[username]/[a-log-dir]/erlang.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]/erlang.stderr.log
stderr_logfile_maxbytes=10MB

rabbitmq.conf

[program:rabbit]
command=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/rabbitmq-server
user=[username]
autorestart=true
directory=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/
stdout_logfile=/home/[username]/[username]/rabbit.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]/rabbit.stderr.log
stderr_logfile_maxbytes=10MB

celery_beat.conf

[program:celerybeat]
command=/home/[username]/webapps/[webapp-root-with-manage.py]/virtualenv/bin/python2.7 manage.py celery beat
user=[username]
autorestart=true
directory=/home/[username]/webapps/[webapp-root-with-manage.py]/
stdout_logfile=/home/[username]/[a-log-dir]//celery_beat.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]//celery_beat.stderr.log
stderr_logfile_maxbytes=10MB

celery_worker.conf

[program:celeryworker]
command=/home/[username]/webapps/[webapp-root-with-manage.py]/virtualenv/bin/python2.7 manage.py celery worker
user=[username]
autorestart=true
directory=/home/[username]/webapps/[webapp-root-with-manage.py]/
stdout_logfile=/home/[username]/[a-log-dir]/celery_beat.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]/celery_beat.stderr.log
stderr_logfile_maxbytes=10MB

Django 1.5 on Webfaction

The setup of a Django application on Webfaction is really easy using their one-click-installer. But this setup is really static and surely not the best option to use for Django. Luke Plant wrote about setting up Django with gunicorn on Webfaction in his blog, I will write about setting up Django 1.5 with Apache and mod_wsgi without the Django one-click-installer but the mod_wsgi one (mod_wsgi 3.4 / Python 2.7). The setup won't be the fastest Django setup in the world, but it will work.
It's basically an update for my old blog post covering Django 1.3:

Set up a mod_wsgi application and create a website in the Webfaction control panel. Log into shell and goto your newly created app. Create a virtualenv and a folder for your project files (maybe you want to check out from a VCS), activate the virtualenv and install Django an whatever you need. Edit the default wsgi.py create by the django-admin.py startproject command:

"""
WSGI config for tools_project project.

This module contains the WSGI application used by Django's development server
and any production WSGI deployments. It should expose a module-level variable
named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
this application via the ``WSGI_APPLICATION`` setting.

Usually you will have the standard Django WSGI application here, but it also
might make sense to replace the whole Django WSGI application with a custom one
that later delegates to the Django one. For example, you could introduce WSGI
middleware here, or combine a Django application with an application of another
framework.

"""
import os

# set paths correctly on webfaction
if os.environ['STAGE'] == 'webfaction':
    import site
    import sys

    site.addsitedir('/home/<username>/webapps/<app>/<virtualenv-folder>/lib/python2.7/site-packages')
    sys.path.append('/home/<username>/webapps/<app>/project')

# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
# if running multiple sites in the same mod_wsgi process. To fix this, use
# mod_wsgi daemon mode with each site in its own daemon process, or use
# os.environ["DJANGO_SETTINGS_MODULE"] = "tools_project.settings"
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "tools_project.settings")

# This application object is used by any WSGI server configured to use this
# file. This includes Django's development server, if the WSGI_APPLICATION
# setting points here.
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

# Apply WSGI middleware here.
# from helloworld.wsgi import HelloWorldApplication
# application = HelloWorldApplication(application)

You can leave out the check of the STAGE environment variable if you haven't set it. I use it to run a project on different hosts and identify each host.

Now adjust the Apache2 config by editing /home/<username>/webapps/<app>/apache2/conf/httpd.conf. Remove the following lines on the bottom:

<Directory /home/<username>/webapps/<app>/htdocs>
    AddHandler wsgi-script.py
</Directory>

Instead, add the following line that tell the WSGI daemon to run the wsgi.py of Django:

WSGIScriptAlias / /home/<username>/webapps/<app>/project/<projectname>/wsgi.py

Then restart the Apache and you're done.

Serving static files for Django on Webfaction

In my last post I wrote about the setup of Django with virtualenv on the Webfaction hosts. The post did not cover the serving of static files and will be covered here. It's always better to serve static media through an extra HTTP server/container doing nothing but serving static media files for better scaling. Normally I use nginx for this as it is small and fast. For Webfaction hosting you have two possibilities:

You can add a "Static only" application that is an nginx, configure it for a separate (sub)domain and then configure your Django project to use this domain for serving the files. Wasn't that easy!

Or you can use the Apache for serving the files.

Create the "media" folder somewhere on your server (e.g. within the virtualenv folder: /home/<username>/webapps/<wsgi-app>/<virtualenv-dir>/media). Now adjust the apache.conf to load the modules required for serving the media and set some expiration time if you like (I used 12 hours as seen below). I'm only writing the updates to the apache.conf of the last post:

...
# put after other LoadModule lines
LoadModule alias_module      modules/mod_alias.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule expires_module    modules/mod_expires.so

...

<VirtualHost *:1234>
    ...
    # after WSGIScriptAlias
    Alias /media /home/your_user_name/webapps/your_wsgi_app_name/virtualenv_directory/media/
    <Directory /home/your_user_name/webapps/your_wsgi_app_name/virtualenv_directory/media>
        Order allow,deny
        Allow from all
        ExpiresActive On
        ExpiresDefault "modification plus 12 hours"
    </Directory>
    ...
</VirtualHost>