17

I am using Fabric to deploy a Celery broker (running RabbitMQ) and multiple Celery workers with celeryd daemonized through supervisor. I cannot for the life of me figure out how to reload the tasks.py module short of rebooting the servers.

/etc/supervisor/conf.d/celeryd.conf

[program:celeryd]
directory=/fab-mrv/celeryd
environment=[RABBITMQ crendentials here]
command=xvfb-run celeryd --loglevel=INFO --autoreload
autostart=true
autorestart=true

celeryconfig.py

import os

## Broker settings
BROKER_URL = "amqp://%s:%s@hostname" % (os.environ["RMQU"], os.environ["RMQP"])

# List of modules to import when celery starts.
CELERY_IMPORTS = ("tasks", )

## Using the database to store task state and results.
CELERY_RESULT_BACKEND = "amqp"

CELERYD_POOL_RESTARTS = True

Additional information

  • celery --version 3.0.19 (Chiastic Slide)
  • python --version 2.7.3
  • lsb_release -a Ubuntu 12.04.2 LTS
  • rabbitmqctl status ... 2.7.1 ...

Here are some things I have tried:

  • The celeryd --autoreload flag
  • sudo supervisorctl restart celeryd
  • celery.control.broadcast('pool_restart', arguments={'reload': True})
  • ps auxww | grep celeryd | grep -v grep | awk '{print $2}' | xargs kill -HUP

And unfortunately, nothing causes the workers to reload the tasks.py module (e.g. after running git pull to update the file). The gist of the relevant fab functions is available here.

The brokers/workers run fine after a reboot.

pztrick
  • 3,741
  • 30
  • 35
  • 2
    `supervisorctl restart` usually does the trick for me. – Paulo Scardine Jun 05 '13 at 03:06
  • Alas, not for me. I have added a gist for my `fabfile.py` with the relevant fab functions excerpted. – pztrick Jun 05 '13 at 13:34
  • wide guess here, but have you tried to delete `*.pyc` before reloading? – Paulo Scardine Jun 08 '13 at 23:15
  • 1
    Have you tried the [reload](http://docs.python.org/2/library/functions.html#reload) function? – NT3RP Jun 09 '13 at 02:52
  • I usually just sudo into supervisorctl and then do `restart `. as @PauloScardine mentioned, you may want to clear old *.pyc files as these can cause an issue from time to time. – Hodson Jun 11 '13 at 10:14
  • 1
    Deleting the *.pyc files, in combination with the usual `supevisorctl` commands, the `-HUP` kill signal, or `celery.control.broadcast` did not work. I now **have it working** by sending a `-9` kill signal to **both** `celeryd` and `Xvfb` so it seems to be related to the headless X server process. A `-9` signal isn't exactly graceful, but nothing else is working. – pztrick Jun 11 '13 at 14:13
  • @pztrick: I'm curious, why do you need X? – Paulo Scardine Jun 11 '13 at 15:28
  • I'm using Selenium on my worker nodes and need Xvfb to emulate Firefox (since they have no GUI) – pztrick Jun 11 '13 at 19:59
  • Possible duplicate: http://stackoverflow.com/questions/9642669/how-to-restart-celery-gracefully-without-delaying-tasks – Dwight Gunning Apr 17 '15 at 22:55
  • Have you tried to restart RabbitMQ also? – Most Wanted May 25 '16 at 14:09

2 Answers2

3

I faced a similar problem and was able to use Watchdog to reload the tasks.py tasks modules when there are changes detected. To install:

pip install watchdog

You can programmatically use the Watchdog API, for example, to monitor for directory changes in the file system. Additionally Watchdog provides an optional shell utility called watchmedo that can be used to execute commands on event. Here is an example that starts the Celery worker via Watchdog and reloads on any changes to .py files including changes via git pull:

watchmedo auto-restart --directory=./ --pattern="*.py" --recursive -- celery worker --app=worker.app --concurrency=1 --loglevel=INFO

Using Watchdog's watchmedo I was able to git pull changes and the respective tasks.py modules were auto reloaded without any reboot of the container or server.

andrew
  • 4,991
  • 5
  • 24
  • 27
2

Just a shot in the dark, with the celeryd --autoreload option did you make sure you have one of the file system notification backends? It recommends PyNotify for linux, so I'd start by making sure you have that installed.

The.Anti.9
  • 43,474
  • 48
  • 123
  • 161
  • 1
    No dice. :( `Requirement already satisfied (use --upgrade to upgrade): pyinotify in /usr/local/lib/python2.7/dist-packages` **(Version 0.9.4)** – pztrick Jun 17 '13 at 13:13