Allow `/metrics` handler output filtering via `name[]` query param #1925

colega · 2025-12-02T15:58:32Z

Context

We have a huge application with thousands of metrics per pod, and we have a Prometheus instance that need to scrape a small subset of those metrics (5 to 10 metrics)

I've set up some relabeling rules, and it works great, but profiles say that this instance spends more than 50% of its CPU time just parsing openmetrics responses and dropping the metrics.

We could save some trees if there was a way to signal Prometheus that we just want a subset of them. I considered running a dumb nginx proxy that would strip out the unneeded metrics, but with native histograms being exposed in proto that won't be possible anymore, plus we'd still be spending precious CPU cycles encoding that response.

I have considered exposing those metrics on a different registry/handler, but:

It adds complexity to the application, those metrics are deep in the business logic
It would leak the details of how I operate the application to the application itself: the app shouldn't know whether I'm scraping with one Prometheus, two, or N.

What this PR does

This PR adds support for filtering metrics by name using the name[] query parameter in HandlerForTransactional (and hence in HandlerFor and promhttp.DefaultHandler). The default behavior is unchanged.

Related issue

I see this has been discussed already in #135
I don't see much consensus there, however I found out that this is something already implemented in client_python ref and client_java ref through the name[] query param, so I hope there isn't much discussion about whether this is a good fit here.

Example

# Get all metrics (existing behavior)
curl http://localhost:8080/metrics

# Get specific metrics
curl 'http://localhost:8080/metrics?name[]=http_requests_total&name[]=process_cpu_seconds_total'

This forks the promhttp.Handler's code to serer/promhttpfork, and applies prometheus/client_golang#1925 on top of it. I'll remove the fork when it's merged and we can go back upstream. Signed-off-by: Oleg Zaytsev <[email protected]>

This commit adds support for filtering metrics by name using the name[] query parameter in HandlerForTransactional. Multiple metric names can be specified by providing the parameter multiple times. Example usage: /metrics?name[]=http_requests_total&name[]=process_cpu_seconds_total Implementation details: - Query parameters are parsed and converted to a map for O(1) lookup - Filtering happens inline during the encoding loop to avoid allocating a new slice - When no name[] parameters are provided, all metrics are returned (backward compatible) - When name[] parameters don't match any metrics, an empty response is returned with HTTP 200 Tests include: - Single and multiple metric filtering - Backward compatibility (no filter returns all) - Non-matching filters - Empty and duplicate values - Verification that transactional gather lifecycle is maintained Signed-off-by: Oleg Zaytsev <[email protected]>

aknuds1

I lack experience with the code in question, but the new functionality looks sensible to me. Leaving some suggestions. Also, shouldn't there be a changelog entry?

prometheus/promhttp/http.go

prometheus/promhttp/http_test.go

Signed-off-by: Oleg Zaytsev <[email protected]>

aknuds1

LGTM, except I wonder if there shouldn't be a changelog entry. Thanks for applying my feedback :)

@duricanikolic

…s` endpoint filtering by `name[]` (#13746)  #### What this PR does Switches to a fork of `client_golang` that contains the PR prometheus/client_golang#1925 which supports `/metrics` endpoint handler filtering through the use of `name[]` query param. I added a test to make sure this functionality doesn't disappear in a possible future removal of this replace directive. Also updates the costattribution package as this version now has changes introduced by @duricanikolic in prometheus/client_golang#1902 #### Which issue(s) this PR fixes or relates to Fixes an internal issue #### Checklist - [x] Tests updated. - [ ] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features.  --- > [!NOTE] > Switch to a forked `client_golang` to add `/metrics` filtering via `name[]`, update cost-attribution validation and tests, adapt API usage, and bump `compress`. > > - **Server/HTTP metrics**: > - Enable metric filtering on `/metrics` via `name[]` query param (promhttp handler change). Added test `TestMetricsEndpointSupportsMetricFiltering`. > - **Dependencies**: > - Replace `github.com/prometheus/client_golang` with fork `github.com/colega/prometheus-client_golang` (pulls in filtering support) and vendor updates. > - Bump `github.com/klauspost/compress` to `v1.18.2` and vendor changes. > - **Cost attribution**: > - Simplify metric descriptor validation using `prometheus.Desc.Err()`; adjust error expectations in tests. > - **Integration tests client**: > - Adapt `LabelNames` to new `model.LabelNames` return type and convert to `[]string`. > - **Changelog**: > - Document `/metrics` filtering enhancement. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 94aa553. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>  --------- Signed-off-by: Oleg Zaytsev <[email protected]>

Signed-off-by: Oleg Zaytsev <[email protected]>

colega · 2025-12-05T09:27:44Z

Oh, right, sorry, I forgot about the CHANGELOG.md, updated now.

colega force-pushed the feature/promhttp-metric-filtering branch from da23a04 to 9bcabdc Compare December 2, 2025 18:04

colega changed the title ~~Add metric filtering support to promhttp.HandlerForTransactional~~ Add metric filtering support to promhttp.Handler Dec 2, 2025

colega force-pushed the feature/promhttp-metric-filtering branch from 9bcabdc to 80d1536 Compare December 2, 2025 18:08

colega marked this pull request as ready for review December 3, 2025 09:24

colega requested review from ArthurSens, bwplotka, kakkoyun and vesari as code owners December 3, 2025 09:24

colega force-pushed the feature/promhttp-metric-filtering branch 3 times, most recently from 1c76a92 to ba1b97a Compare December 3, 2025 11:26

colega changed the title ~~Add metric filtering support to promhttp.Handler~~ Allow /metrics filtering via name[]= query param Dec 3, 2025

colega mentioned this pull request Dec 3, 2025

Allow filtering via query parameters #135

Open

colega changed the title ~~Allow /metrics filtering via name[]= query param~~ Allow /metrics handler output filtering via name[] query param Dec 3, 2025

mohsanabbas approved these changes Dec 3, 2025

View reviewed changes

colega mentioned this pull request Dec 4, 2025

feat(server): allow filtering /metrics endpoint using name[] query param grafana/dskit#844

Closed

colega force-pushed the feature/promhttp-metric-filtering branch from ba1b97a to 11cda20 Compare December 4, 2025 14:34

colega mentioned this pull request Dec 4, 2025

Switch prometheus/client_golang to a fork with support for /metrics endpoint filtering by name[] grafana/mimir#13746

Merged

4 tasks

aknuds1 reviewed Dec 4, 2025

View reviewed changes

prometheus/promhttp/http.go Show resolved Hide resolved

prometheus/promhttp/http_test.go Outdated Show resolved Hide resolved

prometheus/promhttp/http_test.go Outdated Show resolved Hide resolved

colega added 3 commits December 4, 2025 19:27

Repeat comment on all methods

13f3eb9

Signed-off-by: Oleg Zaytsev <[email protected]>

Fail if can't create the request in the test

499e6f4

Signed-off-by: Oleg Zaytsev <[email protected]>

Merge tests

0fbe04e

Signed-off-by: Oleg Zaytsev <[email protected]>

aknuds1 approved these changes Dec 4, 2025

View reviewed changes

Update CHANGELOG.md

6ec4e43

Signed-off-by: Oleg Zaytsev <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow `/metrics` handler output filtering via `name[]` query param #1925

Allow `/metrics` handler output filtering via `name[]` query param #1925

Uh oh!

colega commented Dec 2, 2025 •

edited

Loading

Uh oh!

aknuds1 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aknuds1 left a comment

Uh oh!

colega commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Allow /metrics handler output filtering via name[] query param #1925

Are you sure you want to change the base?

Allow /metrics handler output filtering via name[] query param #1925

Uh oh!

Conversation

colega commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

What this PR does

Related issue

Example

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

colega commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Allow `/metrics` handler output filtering via `name[]` query param #1925

Allow `/metrics` handler output filtering via `name[]` query param #1925

colega commented Dec 2, 2025 •

edited

Loading