Hydra Janitor not working properly #4037
-
|
We were trying to run Janitor on one of our production environments with the following parameters. The batch size is 5M and limit is 40M, this is because we have around 40M tokens created daily. We run this batch once a day after midnight when traffic is low. Eventhough the batch size is 5M, it stops after the first iteration only clearing 4999997 tokens, which stops the batch run due to the exit clause in the code (Ref: https://github.com/ory/hydra/blob/master/persistence/sql/persister_oauth2.go#L565). We are not sure why this happens. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
Hello @qburst-praveen With 40M daily tokens and 5M batch sizes, you're hitting Janitor's architectural limits. The tool stops after one iteration because it reaches the exit clause, likely due to performance degradation or query timeout. Your 40M scale amplifies this. Ory has built a commercial solution for these large scale use cases: The Ory Enterprise License (OEL)- it solves these issues through automatic stale record cleanup and zero-downtime migrations and is explicitly designed for 100GB+ datasets and high traffic. |
Beta Was this translation helpful? Give feedback.
-
|
@vinckr |
Beta Was this translation helpful? Give feedback.
-
|
Hey @qburst-praveen There are no "ideal" values, as these depend on your database size, load, and performance characteristics. |
Beta Was this translation helpful? Give feedback.
Hello @qburst-praveen
With 40M daily tokens and 5M batch sizes, you're hitting Janitor's architectural limits. The tool stops after one iteration because it reaches the exit clause, likely due to performance degradation or query timeout. Your 40M scale amplifies this.
Ory has built a commercial solution for these large scale use cases: The Ory Enterprise License (OEL)- it solves these issues through automatic stale record cleanup and zero-downtime migrations and is explicitly designed for 100GB+ datasets and high traffic.
The OEL is the only version of Ory that guarantees a clean and scalable solution for critical, high-load environments - see this case study with OpenAI.
Feel free to con…