fix(flagd): configurable retry backoff after each sync cycle error (#756)

[alexandraoberaigner]

This PR

adds configurable retry backoff options to the flagd in-process provider, which allows to configure the retry policy of gRPC connections.

Aims to make the implementation consistent with Java. See this change to prevent tight loops when the retry policy doesn't take effect

Related Issues

• it originated from discussions in Infinite retry to establish connection to FlagSyncService in Flagd golang provider #756
• split from fix(flagd): do not retry for certain status codes (#756) #799

Details

Configuration and API Enhancements:

• Added RetryBackoffMs and RetryBackoffMaxMs to ProviderConfiguration, with corresponding default values, environment variable support, and provider options (WithRetryBackoffMs, WithRetryBackoffMaxMs). These allow users to configure retry backoff durations for connection retries. [1] [2] [3] [4] [5] [6]
• Updated the in-process sync and service layers to accept and use the new retry backoff configuration values, passing them through all relevant constructors and methods. [1] [2] [3] [4]

Retry Logic and Policy:

• Modified the gRPC retry policy builder to use the configured backoff values for InitialBackoff and MaxBackoff instead of hardcoded durations, making retry behavior fully configurable.
• Updated the sync retry logic to log and sleep for the configured maximum backoff duration after a failed sync cycle, improving observability and control.

Testing and Validation:

• Added a new unit test (TestBuildRetryPolicy) for the retry policy builder, verifying that the generated JSON policy reflects the configured backoff values and other retry parameters.
• Updated integration and unit tests to include the new retry backoff options, ensuring end-to-end coverage. [1] [2] [3] [4] [5]

[alexandraoberaigner]

DRAFT: rebase on main after #799 is merged -> the PR includes its changes