feat(llma): Add OpenTelemetry ingestion supportv for LLMA

[andrewm4894]

Problem

LLM Analytics currently only supports PostHog SDK ingestion. Customers using OpenTelemetry-based frameworks (OpenAI instrumentation, LangChain, Mastra, etc.) cannot send traces to PostHog without switching SDKs or writing custom integration code.

Changes

Foundation:

• OTLP protobuf parser for trace ingestion at /api/projects/:id/ai/otel/traces
• OTLP protobuf parser for log ingestion at /api/projects/:id/ai/otel/logs
• Span-to-AI-event transformer with waterfall attribute extraction (PostHog native > GenAI semantic conventions > OTel built-ins)
• Authentication via Bearer token in Authorization header
• Route transformed events to PostHog capture pipeline

Instrumentation Pattern Detection:

• OtelInstrumentationPattern enum (V1_ATTRIBUTES, V2_TRACES_AND_LOGS) for future extensibility
• Providers declare their pattern via get_instrumentation_pattern() method
• Centralized detect_provider() function for provider detection
• Safe defaults: unknown providers default to V2 (waits for logs rather than sending incomplete)

V1 Instrumentation Support (data in span attributes):

• Detect v1 spans by provider pattern or presence of prompt/completion in attributes
• Send immediately without waiting for logs
• Supports: OpenAI OTEL v1, Mastra, custom frameworks

V2 Instrumentation Support (metadata in spans, content in logs):

• Event merger with Redis cache for bidirectional span+log merging
• Accumulate all logs before merging to handle out-of-order arrival
• Supports: OpenAI OTEL v2 (official instrumentation)

Provider Transformers:

• Base ProviderTransformer class with can_handle(), transform_prompt(), transform_completion(), get_instrumentation_pattern()
• MastraTransformer: Handles Mastra's custom JSON-wrapped message format
  • Input: {"messages": [{"role": "user", "content": [...]}]}
  • Output: {"files": [], "text": "...", "warnings": [], ...}
  • Note: Mastra does NOT accumulate conversation history - each agent.generate() is independent
• OpenAI GenAI semantic conventions handler (no transformer needed - follows standard conventions)

Tool Message Handling:

• Support for gen_ai.tool.message log events in v2 ingestion
• Proper message role/content extraction for tool interactions

Documentation:

• OTEL ingestion architecture and flow
• Provider Reference section with behavior documentation
• Attribute extraction patterns
• Adding new provider transformers guide

How did you test this code?

Automated:

• Event merger tests covering cache behavior, race conditions, bidirectional merging
• Tool message extraction tests
• V1/V2 ingestion parity tests (test_ingestion_parity.py)

Manual:

• Tested with OpenAI OTEL v1 instrumentation (direct attributes)
• Tested with OpenAI OTEL v2 instrumentation (spans + logs)
• Tested with Mastra framework (custom v1 attributes)
• Verified event tree hierarchy in PostHog UI
• Confirmed token counts, costs, latency measurements

Testing with llm-analytics-apps:

To test this PR with various OTEL providers, use the feature branch from PostHog/llm-analytics-apps#24:

# Clone and checkout the feature branch
git clone https://github.com/PostHog/llm-analytics-apps.git
cd llm-analytics-apps
git checkout feat/add-otel-v1-v2-providers

# Configure .env with your local PostHog instance
cp .env.example .env
# Edit .env and set:
#   POSTHOG_API_KEY=
#   POSTHOG_HOST=http://localhost:8000
#   POSTHOG_PROJECT_ID=1

Node.js OTEL providers:

make run-node
# Select provider 18: "Mastra (OpenAI) - OTEL Instrumentation"
# Then select mode 2 (Tool Call Test) or mode 3 (Message Test)

Python OTEL providers:

make run-python
# Select one of:
#   12: OpenAI with OpenTelemetry v1
#   13: OpenAI with OpenTelemetry v2
#   14: LlamaIndex with OpenTelemetry (RAG)
# Then select mode 2 (Tool Call Test) or mode 3 (Message Test)

The llm-analytics-apps branch includes:

• Mastra OTEL provider (Node.js Dashboard of Action Trend views #18) - sends v1-style traces with attributes
• OpenAI OTEL v1/v2 providers (Python Action trends doesn't load when no actions defined #12, /people page won't load & issue with funnel not updating (related?) #13) - official OpenAI instrumentation
• LlamaIndex OTEL provider (Python Pre HN/big launch #14) - RAG queries with tracing

Changelog: (features only) Is this feature complete?

No - behind feature flag, not ready for public announcement yet.

[github-actions]

Size Change: 0 B

Total Size: 3.41 MB

ℹ️ View Unchanged

Filename	Size
frontend/dist/toolbar.js	3.41 MB

_{compressed-size-action}

General fix:
Avoid returning exception string representations directly to the user in API responses. Instead, return a generic error message or only error details that are safe for user consumption. Optionally, log the detailed error on the server with the request context for debugging, as is already done.

Best way to fix this code:
Change the except ValidationError as e block in otel_traces_endpoint so that the API response contains a generic error (e.g., "Validation failed") and, optionally, a list of error messages if and only if these are stripped of sensitive detail and intended for user consumption. Given we cannot be sure, the safest is to remove str(e) and only say "Validation failed". The server already logs the full error.

Region to change:
File: products/llm_analytics/backend/api/otel/ingestion.py, in the block for except ValidationError as e.

Required changes:

• Replace {"error": str(e)} in the API response with something like {"error": "Validation failed"}.
• Ensure the logger still records the full exception for developers.

To fix the issue, we should not include the raw str(e) from the exception directly in the API response to external clients. Instead, return a generic error message such as "Validation failed" (already present in the similar block above at line 536). Optionally, if the exception's message is known to contain only user input/field issues and has been sanitized, we could expose field-level errors, but by default just log the exception and return a generic message to the user.

Thus, in the block at line 563, replace:

{"error": str(e)}

with:

{"error": "Validation failed"}

Alternatively, if e.detail is field errors and safe, you could return those, but unless the format is strictly controlled and cannot include stack traces, it's safest to exclude them.

No additional methods or imports are needed, as logger already collects the details server-side.

[greptile-apps]

_{19 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: proxy_logs_to_capture_service function is defined in this file but appears to be development/testing infrastructure. This function forwards OTLP logs to a local capture-logs service but is exposed as a production endpoint at /i/v1/logs which could be accessed in production environments.

Issues:

• Hardcoded localhost:4318 will fail in production
• No authentication check on this endpoint
• Uses synchronous requests.post() which blocks Django workers
• Missing @csrf_exempt decorator (though it may inherit from URL config)

Consider moving this to a dev-only utility or documenting that this endpoint should not be exposed in production.

Prompt To Fix With AI

This is a comment left during a code review.
Path: posthog/urls.py
Line: 149:188

Comment:
**logic:** `proxy_logs_to_capture_service` function is defined in this file but appears to be development/testing infrastructure. This function forwards OTLP logs to a local capture-logs service but is exposed as a production endpoint at `/i/v1/logs` which could be accessed in production environments.

Issues:
- Hardcoded `localhost:4318` will fail in production
- No authentication check on this endpoint
- Uses synchronous `requests.post()` which blocks Django workers
- Missing `@csrf_exempt` decorator (though it may inherit from URL config)

Consider moving this to a dev-only utility or documenting that this endpoint should not be exposed in production.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

logic: Logs without trace_id or span_id are grouped under (None, None) key and processed individually. However, this creates a potential issue where multiple unrelated log events accumulate together in logs_by_span[(None, None)].

If you have multiple standalone log events in the same request (not associated with any trace), they'll all be accumulated into the same properties dict, which will cause property overwrites and incorrect merging.

Consider processing these immediately without grouping:

if trace_id and span_id:
    logs_by_span[(trace_id, span_id)].append(log_record)
else:
    # Process standalone logs immediately without grouping
    event = transform_log_to_ai_event(log_record, resource, scope)
    if event is not None:
        events.append(event)

Prompt To Fix With AI

This is a comment left during a code review.
Path: products/llm_analytics/backend/api/otel/ingestion.py
Line: 709:720

Comment:
**logic:** Logs without `trace_id` or `span_id` are grouped under `(None, None)` key and processed individually. However, this creates a potential issue where multiple unrelated log events accumulate together in `logs_by_span[(None, None)]`.

If you have multiple standalone log events in the same request (not associated with any trace), they'll all be accumulated into the same properties dict, which will cause property overwrites and incorrect merging.

Consider processing these immediately without grouping:
```python
if trace_id and span_id:
    logs_by_span[(trace_id, span_id)].append(log_record)
else:
    # Process standalone logs immediately without grouping
    event = transform_log_to_ai_event(log_record, resource, scope)
    if event is not None:
        events.append(event)
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

logic: Race condition in log accumulation logic: between checking if logs exist (line 81), merging them (line 86), re-caching (line 89), and then checking for trace (line 97), another concurrent request could have already merged the logs with a trace and deleted the cache.

This could result in:

1. Line 81: Log exists, retrieve it
2. [Another thread: Trace arrives, merges with logs, deletes both caches]
3. Line 89: Re-cache merged logs (recreates deleted cache)
4. Line 97: Trace is gone (returns None)
5. Result: Merged logs cached but trace already consumed = orphaned cache entry

Consider using Redis transactions (WATCH/MULTI/EXEC) or Lua script for atomic read-modify-write operations.

Prompt To Fix With AI

This is a comment left during a code review.
Path: products/llm_analytics/backend/api/otel/event_merger.py
Line: 81:115

Comment:
**logic:** Race condition in log accumulation logic: between checking if logs exist (line 81), merging them (line 86), re-caching (line 89), and then checking for trace (line 97), another concurrent request could have already merged the logs with a trace and deleted the cache.

This could result in:
1. Line 81: Log exists, retrieve it
2. **[Another thread: Trace arrives, merges with logs, deletes both caches]**
3. Line 89: Re-cache merged logs (recreates deleted cache)
4. Line 97: Trace is gone (returns None)
5. Result: Merged logs cached but trace already consumed = orphaned cache entry

Consider using Redis transactions (WATCH/MULTI/EXEC) or Lua script for atomic read-modify-write operations.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

style: The capture_events function waits for all capture futures synchronously using .result(), blocking the ingestion endpoint until all events are written. This negates the async benefits of capture_batch_internal and increases p99 latency.

For high-throughput scenarios with large batches, this could cause request timeouts. Consider fire-and-forget pattern or background task processing for capture errors.

Prompt To Fix With AI

This is a comment left during a code review.
Path: products/llm_analytics/backend/api/otel/ingestion.py
Line: 429:454

Comment:
**style:** The `capture_events` function waits for all capture futures synchronously using `.result()`, blocking the ingestion endpoint until all events are written. This negates the async benefits of `capture_batch_internal` and increases p99 latency.

For high-throughput scenarios with large batches, this could cause request timeouts. Consider fire-and-forget pattern or background task processing for capture errors.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

logic: JSON parsing of content_input with fallback creates ambiguous behavior. If content_input is a JSON string "[{...}]", it's parsed to a list. But if parsing fails, the string is wrapped as [{"role": "user", "content": content_input}].

This means a valid JSON array string like "[\"hello\", \"world\"]" would be parsed to ["hello", "world"] (not message objects), which will fail validation downstream. The fallback should ensure message object structure.

Prompt To Fix With AI

This is a comment left during a code review.
Path: products/llm_analytics/backend/api/otel/transformer.py
Line: 215:223

Comment:
**logic:** JSON parsing of `content_input` with fallback creates ambiguous behavior. If `content_input` is a JSON string `"[{...}]"`, it's parsed to a list. But if parsing fails, the string is wrapped as `[{"role": "user", "content": content_input}]`.

This means a valid JSON array string like `"[\"hello\", \"world\"]"` would be parsed to `["hello", "world"]` (not message objects), which will fail validation downstream. The fallback should ensure message object structure.

How can I resolve this? If you propose a fix, please make it concise.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Baggage-based distinct_id lost when logs arrive after trace

For v2 ingestion where the trace request arrives first, the span path caches properties and returns None; the eventual event is built in the logs path with extract_distinct_id(resource, span_logs[0].get("attributes", {})) only. Any user context supplied in the trace’s baggage header is never persisted or reused, so merged events become anonymous unless the log payload repeats the distinct ID. This breaks user association for the common trace-first order; the baggage-derived distinct_id needs to be propagated into the cached properties or reused when logs complete the merge.

Useful? React with 👍 / 👎.