Add cleanup status and alerting on cleanup status to monitoring

[pdbossman]

Please make sure that this is a feature request.

System information

• Scylla version (you are using): 2025.1
• Are you willing to contribute it (Yes/No): No

Describe the feature and the current behavior/state.
With enablement of consistent topology, scylla maintains cleanup status in system.topology table. If a cleanup is needed and a node is decommissioned or removed, a cleanup is initiated, and the decommission/removenode operation waits until the cleanup completes.

This behavior may be a surprise and unexpected by the user if they were not aware cleanup was needed.

The fact nodes are in cleanup needed state should be more visible (add to monitoring), additionally we should add alerting if a node remains in cleanup needed for extended period of time.

This will inform and encourage the user to address the issue and reclaim storage, as well as complete an expensive background operation at a more operationally convenient time (during off peak hours).

Who will benefit with this feature?
All administrators using Scylla 2024.2 and higher.

Any Other info.

[amnonh]

@mykaul @pdbossman, to my understanding, the information is not exposed as a metric, am I right?

[mykaul]

@mykaul @pdbossman, to my understanding, the information is not exposed as a metric, am I right?

It isn't. Moreover, I think it's the wrong direction, in the sense that we are not moving our monitoring dashboard to become an operation dashboard (if we were, why not add a 'cleanup now' button while at it to each node / cluster ?)

[amnonh]

@mykaul, and from the cloud perspective, is there something the user needs to do?

[mykaul]

@mykaul, and from the cloud perspective, is there something the user needs to do?

Nope, we handle it.

[amnonh]

@mykaul so not even sure what we need to show in the cloud

[pdbossman]

We show information on backups and repairs. We should have alerting on this operationally important element of Scylla.

If people miss it, there will be an unpleasant and unexpected expensive operation that may may their scaling take much longer then expected.

You would show that cleanup is needed.

[mykaul]

We show information on backups and repairs. We should have alerting on this operationally important element of Scylla.

We show information on Manager actions. If you do either or both backup or repair manually, we don't show anything.

[pdbossman]

It's an operational state of unhealthy that should be alerted.

We want to help people manage their clusters responsibly.

We have alerts for backups not run and repair not run.

@avikivity fyi