Skip to content

Conversation

@cawthorne
Copy link
Contributor

@cawthorne cawthorne commented Oct 28, 2025

Add v2 TXM Metrics with Per-Account Tracking

Metric can be seen for Starknet Testnet nodes at the endpoint:
http://10.13.80.235:6688/plugins/starknet.SN_SEPOLIA/metrics
(Pod IP may rotate).

infra-k8s preview-stage PR is here:
https://github.com/smartcontractkit/infra-k8s/pull/32851/files

ArgoCD Staging is here:
https://argo.main.stage.cldev.sh/applications/cl-ocr-starknet-testnet?view=tree&resource=
https://argo.main.stage.cldev.sh/applications/cl-ocr-starknet-testnet?view=list&resource=&node=%2FPod%2Fchainlink%2Fcl-ocr-starknet-testnet-0-67b97d78b5-pkkg5%2F0

Summary

Implements comprehensive transaction manager metrics following Chainlink-EVM patterns, with per-account granularity and dual telemetry support (Prometheus + OpenTelemetry/Beholder).

Metrics

All metrics are labeled by chainID and accountAddress for per-account monitoring:

  • txm_num_broadcasted_transactions: Counter for successful transaction broadcasts
  • txm_num_confirmed_transactions: Counter for confirmed transactions (handles re-orgs)
  • txm_num_nonce_gaps: Counter for nonce gap events requiring resync
  • txm_nonce_rebroadcast: Counter for nonce rebroadcasts (indicates resync frequency)
  • txm_time_until_tx_confirmed: Histogram of confirmation durations
  • txm_enqueue_failed: Counter for enqueue failures (queue full)
  • txm_next_nonce: Gauge for current next nonce per account

Implementation

  • Dual telemetry: Prometheus (via promauto) and OpenTelemetry/Beholder metrics recorded in parallel
  • Thread-safe: Uses sync.Once for safe metric initialization
  • Per-account tracking: All metrics labeled with both chainID and accountAddress
  • Nonce-based tracking: Internal transaction tracking uses nonce instead of hash for reliability

Files Changed

  • relayer/pkg/chainlink/txm/txm.go - Metric integration with per-account tracking
  • relayer/pkg/chainlink/txm/txm_metrics.go - Prometheus and Beholder metrics implementation
  • relayer/pkg/chainlink/txm/txstore.go - Nonce-based transaction tracking
  • Test files - Comprehensive test coverage

@cawthorne cawthorne changed the base branch from develop to df-21990_fix_starknet_txm October 28, 2025 20:48
toblich
toblich previously approved these changes Nov 28, 2025
Copy link

@toblich toblich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I lack a lot of context here, but as far as I can tell it LGTM

jmank88
jmank88 previously approved these changes Nov 28, 2025
jmank88
jmank88 previously approved these changes Nov 28, 2025
@cawthorne cawthorne merged commit bce49d5 into develop Nov 28, 2025
29 of 30 checks passed
@cawthorne cawthorne deleted the feature/add-v2-txm-metrics branch November 28, 2025 22:00
@cl-sonarqube-production
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants