Skip to content

snuba-api CPU usage is too high #4005

@tangjian0976

Description

@tangjian0976

Body

Self-Hosted ### Version
24.4.1

CPU ### Architecture
x86_64

Docker ### Version
28.4.0

Docker Compose ### Version
2.33.1

Steps to ### Reproduce
After removing and configuring the use of external Redis, Postgres, and Clickhouse instances, and starting self hosted Sentry, the functionality ran normally. However, after running for about a month, the Snuba API continued to receive “deletings.group” and the CPU usage reached about 200%, and Redis' default queue could not be consumed

Expected ### Result
Snuba API can handle “deletings.group” normally and CPU usage is normal

Actual ### Result

adb6e1b93178 sentry-self-hosted-snuba-api-1 208.34% 221.2MiB / 2.5GiB 8.64% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 208.34% 221.2MiB / 2.5GiB 8.64% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 208.34% 221.2MiB / 2.5GiB 8.64% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 208.34% 221.2MiB / 2.5GiB 8.64% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.66% 208.3MiB / 2.5GiB 8.14% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.66% 208.3MiB / 2.5GiB 8.14% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.66% 208.3MiB / 2.5GiB 8.14% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.66% 208.3MiB / 2.5GiB 8.14% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.48% 203.2MiB / 2.5GiB 7.94% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.48% 203.2MiB / 2.5GiB 7.94% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.48% 203.2MiB / 2.5GiB 7.94% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.48% 203.2MiB / 2.5GiB 7.94% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.71% 196.4MiB / 2.5GiB 7.67% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.71% 196.4MiB / 2.5GiB 7.67% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.71% 196.4MiB / 2.5GiB 7.67% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 204.71% 196.4MiB / 2.5GiB 7.67% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 201.96% 191.4MiB / 2.5GiB 7.48% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 201.96% 191.4MiB / 2.5GiB 7.48% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 201.96% 191.4MiB / 2.5GiB 7.48% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 201.96% 191.4MiB / 2.5GiB 7.48% 3.52GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 202.86% 192MiB / 2.5GiB 7.50% 3.53GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 202.86% 192MiB / 2.5GiB 7.50% 3.53GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 202.86% 192MiB / 2.5GiB 7.50% 3.53GB / 4.07GB 0B / 0B 11
adb6e1b93178 sentry-self-hosted-snuba-api-1 202.86% 192MiB / 2.5GiB 7.50% 3.53GB / 4.07GB 0B / 0B 11

Log output of Snuba API
The Snuba API log shows continuous receipt of 'deletings. group', but in reality, no deletion was performed in Sentry Web or Sunba API was called to perform deletion

snuba-api-1 | 2025-10-17 09:39:09,524 Received referrer: deletions.group
snuba-api-1 | WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x7f75fc2dceb8 pid: 633 (default app)
snuba-api-1 | 2025-10-17 09:39:14,020 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:39:17,794 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:39:30,304 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:39:30,404 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:39:46,172 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:39:58,137 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:02,471 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:02,665 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:09,716 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:12,862 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:13,108 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:26,168 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:29,215 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:35,758 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:39,180 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:40:50,160 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:41:10,421 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:41:17,902 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:41:29,241 Received referrer: deletions.group
snuba-api-1 | ...The work of process 629 is done. Seeya!
snuba-api-1 | 2025-10-17 09:41:39,504 Received referrer: deletions.group
snuba-api-1 | worker 2 killed successfully (pid: 629)
snuba-api-1 | Respawned uWSGI worker 2 (new pid: 639)
snuba-api-1 | 2025-10-17 09:41:40,251 Initializing Snuba...
snuba-api-1 | WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x7f75fc2dceb8 pid: 639 (default app)
snuba-api-1 | 2025-10-17 09:41:44,343 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:41:44,600 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:41:48,409 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:41:54,287 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:42:04,662 Received referrer: deletions.group
snuba-api-1 | ...The work of process 633 is done. Seeya!
snuba-api-1 | worker 1 killed successfully (pid: 633)
snuba-api-1 | Respawned uWSGI worker 1 (new pid: 643)
snuba-api-1 | 2025-10-17 09:42:18,305 Initializing Snuba...
snuba-api-1 | WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x7f75fc2dceb8 pid: 643 (default app)
snuba-api-1 | 2025-10-17 09:42:29,308 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:42:45,111 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:43:17,319 Received referrer: deletions.group
snuba-api-1 | 2025-10-17 09:43:20,321 Received referrer: deletions.group

Event ### ID
No response

Metadata

Metadata

Assignees

No one assigned

    Projects

    Status

    No status

    Status

    Waiting for: Product Owner

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions