CPU and RAM spike caused by `uptime-checker` filling uptime backlogs on service startup after a certain downtime

[aldy505]

Environment

self-hosted 25.2.0, commit cut off: 9574780

Steps to Reproduce

1. Make sure uptime-results Sentry consumer and snuba-uptime-results-consumer Snuba consumer are healthy. Then try to have an uptime backlog of 24 hours by shutting the uptime-checker down, but try to keep any other container alive.
2. After that 24 hours of downtime, start the uptime-checker correctly.
3. Uptime checker will try to fill all the missing backlog. Causing CPU and RAM spikes until the backlogs are complete.

Expected Result

No spikes in CPU and RAM. This is very sensitive in self-hosted environment since we have single node for everything (50-ish containers in total).

Actual Result

CPU and RAM spike, of course. Discord thread link: https://discord.com/channels/621778831602221064/1344588889284935731

[aldy505]

Some suggestion that might resolve this issue:

Suggestion 1: Since we're using tokio, we can specify the number of CPU being used, therefore limiting the spike.

Suggestion 2: If we found backlog that are greater than 1 hour, try to spawn a single thread trying to fill it. While keeping another thread tending to current requests.