Support rate backoff and retry to handle rate limiting

[thegrahamking]

Description

When rate limits are hit on one or more endpoints those endpoint fail and the resulting reports are incomplete. This leads to subsequent runs which compound the issue.
The tools should support backoff and retry when rate limits are encounted to prevent this.

Reproduction

1. Run auth0checkmate multiple times in quick succession (reasons for this could be incorrect permissions assignments, failed requests etc).
2. Note HTTP 429 response codes
3. Note incomplete report

Environment

Please provide the following:

• Version of this library used: 1.5.1
• Version of the platform or framework used, if applicable: N/A
• Other relevant versions (language, server software, OS, browser): Windows
• Other modules/plugins/libraries that might be involved: N/A

[mathewwoodyard-okta]

Thanks for opening this issue! Essentially this is the general case of another issue we had, #35. Here's where our thinking is currently, and we'd love to hear your view of the decision.

As you observed, the root cause of the problem stems from the Auth0 Management API rate limit policy, which depends on subscription type and configuration.

We have considered implementing logic to either more aggressively retry or to wait for a period or time before running the report again, but ultimately we decided that approach would bake in a footgun that could be dangerous.

However, an approach to add functionality could be:

• Bomb out when receiving a HTTP 429 response by default and maintain conservative default behavior
• Provide a parameter like --rps to allow the user to determine how many request CheckMate makes per second
• Give tenant admins a way to set their risk tolerance with a --max-attempts option telling CheckMate when to fail (e.g. CheckMate stops running after receiving 3 429s)

This is a contribution I’d really love to see from the community, so I’ll leave this issue open and update it if we decide to start work on this issue internally.

[thegrahamking]

Hi @mathewwoodyard-okta, thanks for the background. Here are my thoughts.

Having some checks fail negative because a rate limit meant the tool wasn't able to determine the tenant state means I have to run the tool again (likely multiple times as I have no control over the order or inclusion of any specific requests) and merge the results. This is tedious and error prone.

Ultimately the desire from me, as an end user of the tool, is that I want to be able to run the tool once (per tenant) and have it produce a full and accurate report. No reruns or merging of run results required.

My preference would be automatic backoff and retry, however, "we decided that approach would bake in a footgun that could be dangerous" suggests you see something in this approach I do not. Could you describe why you consider this approach dangerous?

On your specific suggestions:

Bomb out when receiving a HTTP 429 response by default and maintain conservative default behavior

This would make the tool unacceptable IMO. It would be impossible to obtain any report if any rate limit was hit (which in my experience happens on every run).

Provide a parameter like --rps to allow the user to determine how many request CheckMate makes per second

So long as guidance is included on how to determine what value is appropriate based on the subscription level (I've found this difficult to determine from your docs for our subscription type, although I suspect it maybe the "Sustained Request Limit") I believe this would be acceptable.

Give tenant admins a way to set their risk tolerance with a --max-attempts option telling CheckMate when to fail (e.g. CheckMate stops running after receiving 3 429s)

Again this would would make the tool unacceptable IMO. Even if the tool ran to completion the report could be incomplete and I'm back to reruns and merging results.

I hope that is helpful. I look forward to more information on the automatic backoff and retry approach

[aHochstein]

When interacting with the Auth0 Management API there will be headers sent back to the client you can use to determine how many requests you can do - the tool should just try to stay in around half of the available RPS or so.
Imo you should try to stay in a RPS range where you will never violate this limit even if another application is currently doing requests.
You can check the headers:

• x-ratelimit-limit
• x-ratelimit-remaining
• x-ratelimit-remaining

You can also find the official docs about this here