[Questions] 4.2 Missing khepri files after join_cluster #14748
-
Community Support Policy
RabbitMQ version usedother (please specify) 4.2.0-rc.1 Erlang version used27.2.x Operating system (distribution) usedLinux/MacOS How is RabbitMQ deployed?Debian package rabbitmq-diagnostics status outputN/A Logs from node 2 aka rabbit-2Logs from node 3 (if applicable, with sensitive values edited out)N/A rabbitmq.confN/A Steps to deploy RabbitMQ clusterSteps to reproduce the behavior in questionFrom a rabbitmq-server git checkout start to independent brokers Then follow steps in https://www.rabbitmq.com/docs/clustering#creating to join rabbit-2 to rabbit-1 shell logs showing when the files disappear advanced.configN/A Application codeKubernetes deployment fileWhat problem are you trying to solve?Tested on 4.2.0-rc.1. Let's call the remote node as rabbit-1 and the node that joins the remote node as rabbit-2 When creating a cluster manually (not by peer discovery, but by rabbitmqctl commands) after calling After this the cluster will still work as khepri ra_server process has every data in memory, so it is possible to eg define a queue which will be added to khepri. However other operations will fail: eg WAL file force rollover on rabbit-2 will crash as the WAL file on disk is missing. We noticed this because manually removing rabbit-2 from the cluster (as described in https://www.rabbitmq.com/docs/clustering#removal-of-a-reachable-node) also fails. rabbitmq logs added under "Logs from node 2" Logs from a forced WAL roll-over |
Beta Was this translation helpful? Give feedback.
Answered by
michaelklishin
Oct 20, 2025
Replies: 4 comments 10 replies
-
|
What I found so far is that during the according to the logs at this point khepri_db is not enabled on rabbit-2 hence it looks like at the time of reset all stable feature flags are disabled on rabbit-2 |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
That’s expected: the idea is that the joining node will start with the same feature flags state as the node it joins. |
Beta Was this translation helpful? Give feedback.
-
|
The solution seems to be to not call |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@gomoripeti another solution is to not use |
Beta Was this translation helpful? Give feedback.
-
|
Per @dumbbell, the explicit However, |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
That’s what I’m investigation in fact. The second reset deletes the files for sure. I’m looking for a way to determine if an explicit reset was executed before |
Beta Was this translation helpful? Give feedback.
-
|
Here is an attempt at a fix: #14768. We’ll see if the CI detects a change of behaviour I didn’t anticipate. |
Beta Was this translation helpful? Give feedback.
-
|
I’m not satisfied with the initial patch and work on another implementation. |
Beta Was this translation helpful? Give feedback.
-
|
With more investigation, I find that the problem is deeper than that. It won’t be fixed for 4.2.0. I have a draft patch that correctly deletes files, like we do with Mnesia. This uncovered a few places where at least the Ra However there is another issue: some of the clustering knowledge was added to the CLI commands, not RabbitMQ… Things like shrinking the quorum queue and stream Ra clusters, or updating the stream coordinator. These actions are not executed when there is a reset or "forget member" performed programmatically. This will need a more significant refactor. |
Beta Was this translation helpful? Give feedback.
-
|
But given the updated docs, that should be fine: just |
Beta Was this translation helpful? Give feedback.
-
|
@gomoripeti I have updated the docs: rabbitmq/rabbitmq-website@90881de, rabbitmq/rabbitmq-website@7fc0833, rabbitmq/rabbitmq-website@6237ad3. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
thanks a lot Michael |
Beta Was this translation helpful? Give feedback.
Answer selected by
michaelklishin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@gomoripeti I have updated the docs: rabbitmq/rabbitmq-website@90881de, rabbitmq/rabbitmq-website@7fc0833, rabbitmq/rabbitmq-website@6237ad3.