chore: stop capturing redis oom set failure in sentry (#7528)

I was just looking into one of the errors that triggered our [Critical:
Snuba errors over 30
minutes.](https://sentry.sentry.io/issues/alerts/rules/details/46321/?alert=230878&referrer=metric_alert_slack&detection_type=static&notification_uuid=653cde63-5ca2-4d50-8fa1-933cab7377bd)
It was ResponseError: OOM command not allowed under OOM prevention. This
comes from our redis cache when it reaches max memory. The cache has an
lru eviction policy when it fills up so usually this doesnt happen, but
occasionally it still can.

We have a datadog metric,
[snuba.read_through_cache.redis_cache_set_error](https://app.datadoghq.com/metric/explorer?graph_layout=stacked&start=1762286323568&end=1762891123568&paused=false#N4Ig7glgJg5gpgFxALlAGwIYE8D2BXJVEADxQEYAaELcqyKBAC1pEbghkcLIF8qo4AMwgA7CAgg4RKUAiwAHOChASAtnADOcAE4RNIKtrgBHPJoQaUAbVBGN8qVoD6gnNtUZCKiOq279VKY6epbINiAiGOrKQdpYZAYgUJ4YThr42gDGSsgg6gi6mZaBZnHKGniqyBoieABGGAB0RhhQTkza+JxOmRiZbM1wUBAaPX1saYhOOp3awFoiBVhORjCSIsh4GjyNGKOZ+IsAFACUIDwAulSu7niYoeE3qncYMaXx51cgGnJoOaDyDB-BAIHJJHAwHr3DQQTKJNCiODtBRg9IIqDwxFOehMZQiNweNDnfgjeSYZZyRTKBEiJSXHh8b5k8QAYSkwhgKFqaDQPCAA)
that basically exactly tracks this errors occurrence. In order to reduce
feed-sns noise I will remove this error from sentry and instead use the
datadog metric to monitor the percentage of redis cache writes that
fail, and have a monitor on it that will send in feed-sns if it ever
gets too high.

[Here is the new monitor I
created](https://app.datadoghq.com/monitors/236008510) and here [a new
widget I added in the api
dashboard](https://app.datadoghq.com/dashboard/spg-jqb-tgz/snuba-api?fromUser=true&refresh_mode=paused&tpl_var_sentry_region%5B0%5D=us&from_ts=1762366350809&to_ts=1762370761457&live=false&tile_focus=1728346643271425)

This PR stops capturing the sentry metric

Author: kylemumma

snuba/state/cache/redis/backend.py

@@ -3,6 +3,7 @@
 
 import sentry_sdk
 
+from redis import ResponseError
 from redis.exceptions import ConnectionError, ReadOnlyError
 from redis.exceptions import TimeoutError as RedisTimeoutError
 from snuba import environment, settings
@@ -22,6 +23,11 @@
 RESULT_WAIT = 2
 SIMPLE_READTHROUGH = 3
 
+DONT_CAPTURE_ERRORS = {
+    # if you need to track this error, see datadog metric snuba.read_through_cache.redis_cache_set_error
+    ResponseError("OOM command not allowed under OOM prevention."),
+}
+
 
 class RedisCache(Cache[TValue]):
     def __init__(
@@ -71,7 +77,8 @@ def __get_value_with_simple_readthrough(
         except Exception as e:
             if settings.RAISE_ON_READTHROUGH_CACHE_REDIS_FAILURES:
                 raise e
-            sentry_sdk.capture_exception(e)
+            if e not in DONT_CAPTURE_ERRORS:
+                sentry_sdk.capture_exception(e)
 
         if timer is not None:
             timer.mark("cache_get")
@@ -91,7 +98,8 @@ def __get_value_with_simple_readthrough(
                     )
                 except Exception as e:
                     metrics.increment("redis_cache_set_error", tags=metric_tags)
-                    sentry_sdk.capture_exception(e)
+                    if e not in DONT_CAPTURE_ERRORS:
+                        sentry_sdk.capture_exception(e)
                     return value
                 record_cache_hit_type(RESULT_EXECUTE)
                 if timer is not None: