chore: add cymbal to hobby docker #41841
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds the cymbal service to the hobby Docker Compose configuration to enable processing of $exception events in self-hosted deployments, which were previously not being processed.
- Adds cymbal service definition to docker-compose.base.yml with proper image, build context, and environment configuration
- Integrates cymbal into docker-compose.hobby.yml with build override pointing to the local rust directory
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| docker-compose.base.yml | Defines the base cymbal service configuration with image, build args, restart policy, and required environment variables (DATABASE_URL and KAFKA_HOSTS) |
| docker-compose.hobby.yml | Adds cymbal to hobby deployment by extending the base service definition and overriding the build context to use local posthog/rust directory |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| context: ./posthog/rust | ||
| extends: | ||
| file: docker-compose.base.yml | ||
| service: cymbal |
There was a problem hiding this comment.
The cymbal service requires both database and Kafka to be available (as indicated by DATABASE_URL and KAFKA_HOSTS environment variables in the base configuration). Consider adding depends_on declarations similar to cyclotron-janitor (lines 261-265) to ensure proper service startup order:
depends_on:
db:
condition: service_healthy
kafka:
condition: service_started| service: cymbal | |
| service: cymbal | |
| depends_on: | |
| db: | |
| condition: service_healthy | |
| kafka: | |
| condition: service_started |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| environment: | ||
| DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog' | ||
| KAFKA_HOSTS: 'kafka:9092' |
There was a problem hiding this comment.
logic: missing required environment variables for cymbal. Based on rust/cymbal/src/config.rs:71-141, cymbal requires:
REDIS_URL(used for caching, see line 139-140)- Object storage config:
OBJECT_STORAGE_ENDPOINT,OBJECT_STORAGE_BUCKET,OBJECT_STORAGE_ACCESS_KEY_ID,OBJECT_STORAGE_SECRET_ACCESS_KEY,OBJECT_STORAGE_FORCE_PATH_STYLE(lines 71-87, used for storing symbol sets) MAXMIND_DB_PATHfor geolocation (line 136-137)
compare with the feature-flags service configuration at lines 304-322 which includes all these. without these, cymbal will fail to start or function correctly
Prompt To Fix With AI
This is a comment left during a code review.
Path: docker-compose.base.yml
Line: 291:293
Comment:
**logic:** missing required environment variables for cymbal. Based on `rust/cymbal/src/config.rs:71-141`, cymbal requires:
- `REDIS_URL` (used for caching, see line 139-140)
- Object storage config: `OBJECT_STORAGE_ENDPOINT`, `OBJECT_STORAGE_BUCKET`, `OBJECT_STORAGE_ACCESS_KEY_ID`, `OBJECT_STORAGE_SECRET_ACCESS_KEY`, `OBJECT_STORAGE_FORCE_PATH_STYLE` (lines 71-87, used for storing symbol sets)
- `MAXMIND_DB_PATH` for geolocation (line 136-137)
compare with the `feature-flags` service configuration at lines 304-322 which includes all these. without these, cymbal will fail to start or function correctly
How can I resolve this? If you propose a fix, please make it concise.| cymbal: | ||
| image: ghcr.io/posthog/posthog/cymbal:master | ||
| build: | ||
| context: rust/ | ||
| args: | ||
| BIN: cymbal | ||
| restart: on-failure |
There was a problem hiding this comment.
logic: missing volume mount for GeoLite2 database. cymbal needs access to the MaxMind database file (see config.rs:136-137 and config.rs:174-182). the feature-flags service at line 302-303 includes volumes: - ./share:/share for this purpose. without this mount, MAXMIND_DB_PATH will point to a non-existent file
Prompt To Fix With AI
This is a comment left during a code review.
Path: docker-compose.base.yml
Line: 284:290
Comment:
**logic:** missing volume mount for GeoLite2 database. cymbal needs access to the MaxMind database file (see `config.rs:136-137` and `config.rs:174-182`). the `feature-flags` service at line 302-303 includes `volumes: - ./share:/share` for this purpose. without this mount, `MAXMIND_DB_PATH` will point to a non-existent file
How can I resolve this? If you propose a fix, please make it concise.| cymbal: | ||
| build: | ||
| context: ./posthog/rust | ||
| extends: | ||
| file: docker-compose.base.yml | ||
| service: cymbal |
There was a problem hiding this comment.
logic: missing depends_on declarations. cymbal requires database, kafka, redis, and objectstorage to be available before starting. compare with feature-flags service at lines 312-314 which declares dependencies. without this, cymbal may attempt to start before its dependencies are ready, causing startup failures
Prompt To Fix With AI
This is a comment left during a code review.
Path: docker-compose.hobby.yml
Line: 288:293
Comment:
**logic:** missing `depends_on` declarations. cymbal requires database, kafka, redis, and objectstorage to be available before starting. compare with `feature-flags` service at lines 312-314 which declares dependencies. without this, cymbal may attempt to start before its dependencies are ready, causing startup failures
How can I resolve this? If you propose a fix, please make it concise.
| restart: on-failure | ||
| environment: | ||
| DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog' | ||
| KAFKA_HOSTS: 'kafka:9092' |
There was a problem hiding this comment.
Not sure this works yet because I tested a self-hosted deploy and cymbal threw these errors:
2025-11-20T17:27:57.179241Z ERROR librdkafka: librdkafka: FAIL [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
2025-11-20T17:27:57.179292Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
2025-11-20T17:27:57.182469Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2025-11-20T17:27:58.177420Z ERROR librdkafka: librdkafka: FAIL [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2025-11-20T17:27:58.177442Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2025-11-20T17:28:12.180083Z ERROR common_kafka::transaction: Failed to fetch metadata from Kafka brokers: KafkaError (Metadata fetch error: BrokerTransportFailure (Local: Broker transport failure))
2025-11-20T17:28:12.184030Z ERROR librdkafka: librdkafka: FAIL [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
2025-11-20T17:28:12.184146Z ERROR librdkafka: librdkafka: FAIL [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2025-11-20T17:28:12.184187Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT) thread 'main' panicked at cymbal/src/main.rs:77:59: called Result::unwrap() on an Err value: KafkaError(KafkaError (Metadata fetch error: BrokerTransportFailure (Local: Broker transport failure))) note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Based on some other issues (#41581 and #32607 specifically) I wonder if this is because of a wider issue with Kafka on self-hosted installs
Problem
$exception events do not go through the normal plugin-server ingestion flow. Cymbal is currently not started in the self hosted docker environment so they are never processed
Changes
Add cymbal to the hobby docker compose file