Tag erlang

Celery with RabbitMQ 3.1.0 on Webfaction

A while ago Mark Liu wrote about setting up django-celery on a Webfaction host. I took the tutorial and matched it to the current version of RabbitMQ that is 3.1.0.

Please read this article entirely before installing anything!

Basically follow Mark's tutorial and install Erlang as well as RabbitMQ but use the current RabbitMQ binary *.tar.gz from the download page.

Attention: Depending on what applications you are already running on Webfaction, the 256MB RAM plan may be insufficient to run Celery and RabbitMQ and Erlang. You can give it a try, for me it did not work, I added some more memory to my plan. Monitor your memory usage while installing. I'm using the following script to do so (replace the [username] with your username):

# memory_top.sh
# lists memory usage for given user every 5 seconds
while true
do
    ps -U [username] --no-headers -o rss | (tr '\n' +; echo 0) | bc
    sleep 5
done

Do not configure rabbitmq-server and the rabbitmq-env file as written. Instead do the following (I extracted RabbitMQ to \$HOME/rabbitmq/rabbitmq_server-3.1.0 and will use this paths):

Edit \$HOME/rabbitmq/rabbitmq_server-3.1.0/sbin/rabbitmq-defaults:

...
# comment these lines:
#CONFIG_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq
#LOG_BASE=${SYS_PREFIX}/var/log/rabbitmq
#MNESIA_BASE=${SYS_PREFIX}/var/lib/rabbitmq/mnesia
# add these lines:
CONFIG_FILE=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/
LOG_BASE=/home/[username]/logs/user/rabbitmq # create this directory!
MNESIA_BASE=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/
...

Edit \$HOME/rabbitmq/rabbitmq_server-3.1.0/sbin/rabbitmq-env:

...
# add to the end
export ERL_EPMD_PORT=[Erlang-Custom-App-Port]
export RABBITMQ_NODE_PORT=[RabbitMQ-Custom-App-Port]
export ERL_INETRC=$HOME/.erl_inetrc

Create the \$HOME/hosts and \$HOME/.erl_inetrc files as written in Mark's blog and continue as he wrote.

After testing Erlang and Rabbit kill the processes to integrate them into supervisord. I use it to control the processes, you can simply install it into your global Webfaction Python interpreter.
Here are the (quite simple) configurations (adjust the paths to your environment):

erlang.conf

[program:erlang]
command=epmd -port [erlang-port]
user=[username]
autorestart=true
directory=/home/[username]/
stdout_logfile=/home/[username]/[a-log-dir]/erlang.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]/erlang.stderr.log
stderr_logfile_maxbytes=10MB

rabbitmq.conf

[program:rabbit]
command=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/rabbitmq-server
user=[username]
autorestart=true
directory=/home/[username]/rabbitmq/rabbitmq_server-3.1.0/sbin/
stdout_logfile=/home/[username]/[username]/rabbit.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]/rabbit.stderr.log
stderr_logfile_maxbytes=10MB

celery_beat.conf

[program:celerybeat]
command=/home/[username]/webapps/[webapp-root-with-manage.py]/virtualenv/bin/python2.7 manage.py celery beat
user=[username]
autorestart=true
directory=/home/[username]/webapps/[webapp-root-with-manage.py]/
stdout_logfile=/home/[username]/[a-log-dir]//celery_beat.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]//celery_beat.stderr.log
stderr_logfile_maxbytes=10MB

celery_worker.conf

[program:celeryworker]
command=/home/[username]/webapps/[webapp-root-with-manage.py]/virtualenv/bin/python2.7 manage.py celery worker
user=[username]
autorestart=true
directory=/home/[username]/webapps/[webapp-root-with-manage.py]/
stdout_logfile=/home/[username]/[a-log-dir]/celery_beat.stdout.log
stdout_logfile_maxbytes=10MB
stderr_logfile=/home/[username]/[a-log-dir]/celery_beat.stderr.log
stderr_logfile_maxbytes=10MB