Anže's Blog

Python, Django, and the Web

02 Jan 2024

No Downtime Deployments with Gunicorn

Suppose you’re hosting your Django, Flask, or FastAPI application on your server instead of using a platform like Heroku or Fly. You want to continue serving requests while your Gunicorn process restarts to load your application code updates, and you want to avoid setting up a complicated code deployment process. Gunicorn supports this workflow out of the box with the HUP signal.

The HUP Signal

When the Gunicorn process receives the HUP signal, it will start new worker processes, stop routing requests to the old workers and shut them down once their request queues clear. This way, no request is lost during the upgrade process.

You can use the kill command to send the hup signal to your Gunicorn process:

kill -HUP $(cat gunicorn.pid)

Gunicorn doesn’t create the gunicorn.pid file by default, so you’ll have to add the pidfile parameter to your gunicorn.conf.py config file:

pidfile = "gunicorn.pid"

Here is a sample output of the Gunicorn process after sending the HUP signal:

[2024-01-02 17:29:43 +0000] [22791] [INFO] Handling signal: hup
[2024-01-02 17:29:43 +0000] [22791] [INFO] Hang up: Master
[2024-01-02 17:29:43 +0000] [22800] [INFO] Booting worker with pid: 22800
[2024-01-02 17:29:43 +0000] [22801] [INFO] Booting worker with pid: 22801
[2024-01-02 17:29:43 +0000] [22802] [INFO] Booting worker with pid: 22802
[2024-01-02 17:29:43 +0000] [22804] [INFO] Booting worker with pid: 22804
[2024-01-02 17:29:43 +0000] [22806] [INFO] Booting worker with pid: 22806
[2024-01-02 17:29:43 +0000] [22792] [INFO] Worker exiting (pid: 22792)
[2024-01-02 17:29:43 +0000] [22795] [INFO] Worker exiting (pid: 22795)
[2024-01-02 17:29:43 +0000] [22794] [INFO] Worker exiting (pid: 22794)
[2024-01-02 17:29:43 +0000] [22793] [INFO] Worker exiting (pid: 22793)
[2024-01-02 17:29:43 +0000] [22796] [INFO] Worker exiting (pid: 22796)

SystemD

If you are using systemd to manage your Gunicorn process, you can use the systemctl reload command to send the HUP signal to your Gunicorn process:

systemctl reload gunicorn.service

For this to work, you’ll need to ensure that the ExecReload parameter in your unicorn.service file is set to ExecReload=/bin/kill -s HUP $MAINPID. See the example service file in the Gunicorn docs.

This way, systemd will send the HUP signal to the correct process, so you don’t have to worry about the process id (PID). The only downside is that systemd doesn’t recommend using async commands like kill -HUP in the ExecReload parameter, because it might trigger reloads of depending services before the main service has finished reloading. It’s not a huge issue, though, since having other services depend on Gunicorn is rare.

Example output output of systemctl reload gunicorn.service. Note that the systemd Reloaded event is logged before the Gunicorn process has finished reloading:

Jan 03 13:01:00 raspberrypi systemd[1]: Reloading Gunicorn.
Jan 03 13:01:00 raspberrypi gunicorn[201219]: [2024-01-03 13:01:00 +0000] [201219] [INFO] Handling signal: hup
Jan 03 13:01:00 raspberrypi gunicorn[201219]: [2024-01-03 13:01:00 +0000] [201219] [INFO] Hang up: Master
Jan 03 13:01:00 raspberrypi systemd[1]: Reloaded Gunicorn.
Jan 03 13:01:00 raspberrypi gunicorn[201241]: [2024-01-03 13:01:00 +0000] [201241] [INFO] Booting worker with pid: 201241
Jan 03 13:01:01 raspberrypi gunicorn[201242]: [2024-01-03 13:01:00 +0000] [201242] [INFO] Booting worker with pid: 201242
Jan 03 13:01:01 raspberrypi gunicorn[201244]: [2024-01-03 13:01:01 +0000] [201244] [INFO] Booting worker with pid: 201244
Jan 03 13:01:01 raspberrypi gunicorn[201243]: [2024-01-03 13:01:01 +0000] [201243] [INFO] Booting worker with pid: 201243
Jan 03 13:01:01 raspberrypi gunicorn[201245]: [2024-01-03 13:01:01 +0000] [201245] [INFO] Booting worker with pid: 201245
Jan 03 13:01:01 raspberrypi gunicorn[201223]: [2024-01-03 13:01:01 +0000] [201223] [INFO] Worker exiting (pid: 201223)
Jan 03 13:01:01 raspberrypi gunicorn[201224]: [2024-01-03 13:01:01 +0000] [201224] [INFO] Worker exiting (pid: 201224)
Jan 03 13:01:01 raspberrypi gunicorn[201220]: [2024-01-03 13:01:01 +0000] [201220] [INFO] Worker exiting (pid: 201220)
Jan 03 13:01:01 raspberrypi gunicorn[201221]: [2024-01-03 13:01:01 +0000] [201221] [INFO] Worker exiting (pid: 201221)
Jan 03 13:01:01 raspberrypi gunicorn[201222]: [2024-01-03 13:01:01 +0000] [201222] [INFO] Worker exiting (pid: 201222)

Upgrading Gunicorn itself

The kill -HUP method upgrades your application and its dependencies (unless you preload your app!), but it never closes the main Gunicorn process, so Gunicorn itself doesn’t get upgraded. This isn’t a big deal since Gunicorn updates are rare (there were almost two years between 20.0.4 and 21.0.0 releases), and it’s probably not a big deal to restart your app once in a while even if it means losing a few requests, but if you want to avoid downtime there is a way to do it.

First, start a new Gunicorn process alongside the old one:

kill -USR2 $(cat gunicorn.pid) 

Once you see the new Gunicorn process is running, tell the old process to stop serving requests:

kill -WINCH $(cat gunicorn.pid)

Finally, when you confirm that the old process has stopped serving requests, tell it to terminate:

kill -TERM $(cat gunicorn.pid)

More info about this is available in the official Gunicorn docs, along with instructions on how to revert the process if you encounter problems.

Uvicorn

Uvicorn only runs a single process and isn’t recommended for production alone, so the official Uvicorn docs recommend using a process manager like Gunicorn to overcome this limitation. This is great because we have just learned how to restart Gunicorn without downtime!

To get Gunicorn to run your Uvicorn processes, you only need to define the worker_class variable in the config file:

worker_class = "uvicorn.workers.UvicornWorker"

Conclusion

I hope this showed that managing your Gunicorn process is not as complicated as it seems and that you don’t have to reach for Kubernetes to achieve no-downtime deployments.

Enjoyed the read or have a different perspective?

Let me know on Twitter, Mastodon, or email. 🩵