[Bug]: `RequestHandlerAvgIdlePercent` metric on the Strimzi Metrics Reporter

[fabiobaiao]

Bug Description

The RequestHandlerAvgIdlePercent metric isn't being collected with the Strimzi Metrics Reporter. This is happening because it is being set with the _total suffix on the default metrics allow list:

strimzi-kafka-operator/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaCluster.java

Line 131 in eb50698

"kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent_total",

Setting it on the metricsConfig allowList without the _total suffix works (and it gets created with the _total suffix)

However its usage on the grafana dashboard example doesn't seem to be correct, since it's not showing a percentage but an ever increasing value.

strimzi-kafka-operator/examples/metrics/strimzi-metrics-reporter/grafana-dashboards/strimzi-kafka.json

Line 2315 in c643d5d

"expr": "sum(kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent_total{namespace=\"$kubernetes_namespace\",strimzi_io_cluster=\"$strimzi_cluster_name\",kubernetes_pod_name=~\"$strimzi_cluster_name-$kafka_broker\"}*100) by (kubernetes_pod_name)",

I suppose this is because it's being created as a counter. Maybe another one should be created from the MeanRate as it seems to happen with the JMX config:

strimzi-kafka-operator/packaging/examples/metrics/kafka-metrics.yaml

Lines 143 to 146 in e6c39dd

	# Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
	- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
	name: kafka_$1_$2_$3_percent
	type: GAUGE

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.48.0

Kubernetes version

Kubernetes 1.34

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[fvaleri]

@OwenCorrigan76 can you take a look?

[OwenCorrigan76]

@fabiobaiao Thanks for reporting. I will take a look at it.

[fvaleri]

@fabiobaiao this is related to the following metrics-reporter issue: strimzi/metrics-reporter#101. As you figured out, the plugin is exporting this and other metrics as counters, breaking the dashboard. These metrics are of type Meter, which is a compound metric including a counter and various rates. Here we would need the mean rate. A fix for the operator dashboard depends on the solution to the metrics reporter issue.

@OwenCorrigan76 in the meantime, we can remove the _total suffix from all default metrics in KafkaCluster as it is automatically appended by the the Prometheus client to counter metrics. Maybe leave a comment about that for future reference.

[fabiobaiao]

Thank you for the explanation @fvaleri. Given the comment on that issue about leaving the rate computation to the users, I think the dashboard query can be changed to use something like:

rate(kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent_total{}/1e9*100)

(the division by 1e9 is because the counter is in nanoseconds: https://github.com/apache/kafka/blob/710aa678b7a6409296fb3c852a47c1876d8fa8e9/core/src/main/scala/kafka/server/KafkaRequestHandler.scala#L103)

[fvaleri]

@fabiobaiao the metrics-reporter issue linked above has to be triaged in one of the next community calls. You are welcomed to join. Once we have consensus, we proceed with the fix. Are you interested to contribute?