Possible causes of WORKER TIMEOUT

[dbartenstein]

Hello Gunicorn community! 👋

Today WORKER TIMEOUTs happened on our server. There was pretty high load due to many webhook events being processed and after a while Gunicorn workers started to fail with WORKER TIMEOUT (see log below).

Our Gunicorn configuration:
gunicorn --workers 5 --worker-connections=1000 --worker-class=gevent --bind localhost:8000 --disable-redirect-access-to-syslog

Questions:

• Should timeout be set explicitly when using gevent, e.g. --timeout 600?
• Is it wise to set max-requests as recommended by Adam? https://adamj.eu/tech/2019/09/19/working-around-memory-leaks-in-your-django-app/
• Gunicorn + NGINX are running on a virtual server and there was enough memory available - so "Perhaps out of memory?" probably does not point to the the possible solution
• When using gevent + Gunicorn: is the following paragraph of the Gunicorn documentation still relevant? For full greenlet support applications might need to be adapted. When using, e.g., Gevent and Psycopg it makes sense to ensure psycogreen is installed and setup.
  Last update to psycogreen was in 2022.

Grateful for any hints and recommendations! 🙇

/var/log/syslog:

Sep  5 09:38:38 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:38 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043765)
Sep  5 09:38:39 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:39 +0200] [413510] [ERROR] Worker (pid:2043765) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:39 prod-main-zemtu gunicorn[2043946]: [2023-09-05 09:38:39 +0200] [2043946] [INFO] Booting worker with pid: 2043946
Sep  5 09:38:58 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:58 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043479)
Sep  5 09:38:58 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:58 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043840)
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043858)
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [ERROR] Worker (pid:2043479) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:59 prod-main-zemtu gunicorn[2044005]: [2023-09-05 09:38:59 +0200] [2044005] [INFO] Booting worker with pid: 2044005
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [ERROR] Worker (pid:2043840) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:59 prod-main-zemtu gunicorn[2044007]: [2023-09-05 09:38:59 +0200] [2044007] [INFO] Booting worker with pid: 2044007
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [ERROR] Worker (pid:2043858) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:59 prod-main-zemtu gunicorn[2044008]: [2023-09-05 09:38:59 +0200] [2044008] [INFO] Booting worker with pid: 2044008
Sep  5 09:39:10 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:39:10 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043842)
Sep  5 09:39:11 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:39:11 +0200] [413510] [ERROR] Worker (pid:2043842) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:39:11 prod-main-zemtu gunicorn[2044099]: [2023-09-05 09:39:11 +0200] [2044099] [INFO] Booting worker with pid: 2044099

[dbartenstein]

Hello 👋!

After doing quite some research, we today deployed the following configuration on our servers:
gunicorn --workers=5 --worker-connections=1000 --worker-class=gevent --bind=localhost:8000 --name=ZEMTU zemtu.wsgi --disable-redirect-access-to-syslog --timeout=600 --max-requests=5000 --max-requests-jitter=250 --config=gunicorn.conf.py

Additional notes:

• gunicorn.conf.py is almost exactly like https://gist.github.com/jaddison/4506070

I’ll keep you posted about the success of this new configuration.

[akshaybaj]

Hi!.

Did this configuration work without any error?

[dbartenstein]

Yes, for us it this configuration has been working. We only tweaked the config a bit:

gunicorn --env DJANGO_SETTINGS_MODULE=zemtu.settings.production --workers=5 --worker-connections=1000 --worker-class=gevent --bind=localhost:8000 --name=ZEMTU zemtu.wsgi --disable-redirect-access-to-syslog --timeout=600 --max-requests=50000 --max-requests-jitter=3000 --config=zemtu/settings/gunicorn.conf.py

And yes, we have https://github.com/psycopg/psycogreen/ installed.

[mneuwirth2020]

Hi,
I found the reason for that behavior as I was facing the same problem.
In "arbiter.py", in function "spawn_worker", the line after worker.init_process() must be changed from sys.exit(0) to:
os._exit(os.EX_OK).
Otherwise the forked process will not be terminated by sys.exit(0), but keeps running and will be killed after timeout is reached by murder_workers function, which leads to the WORKER TIMEOUT log.

EDIT (Oct 21, 2025):
In my scenario the worker class "gthread" is used. In my case the WORKER TIMEOUT appeared on every Autorestart of the workers.
After @pajod 's comment I modified the spawn_worker function like this:

    def spawn_worker(self):
        self.worker_age += 1
        worker = self.worker_class(self.worker_age, self.pid, self.LISTENERS,
                                   self.app, self.timeout / 2.0,
                                   self.cfg, self.log)
        self.cfg.pre_fork(self, worker)
        pid = os.fork()
        if pid != 0:
            worker.pid = pid
            self.WORKERS[pid] = worker
            return pid

        # Do not inherit the temporary files of other workers
        for sibling in self.WORKERS.values():
            sibling.tmp.close()

        # Process Child
        worker.pid = os.getpid()
        exitcode=os.EX_OK
        try:
            util._setproctitle("worker [%s]" % self.proc_name)
            self.log.info("Booting worker with pid: %s", worker.pid)
            if self.cfg.reuse_port:
                worker.sockets = sock.create_sockets(self.cfg, self.log)
            self.cfg.post_fork(self, worker)
            worker.init_process()
            sys.exit(0)
        except SystemExit:
            raise
        except AppImportError as e:
            self.log.debug("Exception while loading the application",
                           exc_info=True)
            print("%s" % e, file=sys.stderr)
            sys.stderr.flush()
            exitcode = self.APP_LOAD_ERROR
            sys.exit(self.APP_LOAD_ERROR)
        except Exception:
            self.log.exception("Exception in worker process")
            if not worker.booted:
                exitcode = self.WORKER_BOOT_ERROR
                sys.exit(self.WORKER_BOOT_ERROR)
            exitcode = -1
            sys.exit(-1)
        finally:
            self.log.info("Worker exiting (pid: %s)", worker.pid)
            try:
                worker.tmp.close()
                self.cfg.worker_exit(self, worker)
            except Exception:
                self.log.warning("Exception during worker exit:\n%s",
                                 traceback.format_exc())
            finally:
               os._exit(exitcode)

sys.exit() only exits the thread of the child process, but not the child process itself.
This is done by os._exit(exitcode) in the end.
Only the second finally block and the exitcode variable was added.

@pajod, what do you think? Am I missing something?

[pajod]

sys.exit() should work. Skipping ordinary shutdown will merely hide the problem and does not determine or resolve the underlying reason. I suggested eliminating one plausible candidate in #3275

[mneuwirth2020]

I found this remark in the documentation of os._exit (https://docs.python.org/3/library/os.html#os._exit):

Note: The standard way to exit is sys.exit(n). _exit() should normally only be used in the child process after a fork().