Improve rebalance shards algorithm #6018
Conversation
quickwit/quickwit-control-plane/src/ingest/ingest_controller.rs
Outdated
Show resolved
Hide resolved
| ); | ||
| } | ||
| shards_to_move | ||
| shards_to_rebalance |
There was a problem hiding this comment.
This function returns the list of shards to move... It does not move stuff itself.
This is nice because it means any sophisticated logic downstream will be applied to place the shard on the right server.
HOWEVER
The code downstream does not enforce your new condition.
This means that as a set : this function + the function that attributes your stuff does not necessarily ever converges. I'm thinking in particular of the code that tries to place shards from the same index on the same indexer.
There was a problem hiding this comment.
This logic only exists in the indexing controller. In the ingest controller, the placement logic is much simpler: we allocate shards to the ingesters with the least number of open shards. Right?
guilload
force-pushed
the
guilload/improve-shard-rebalance-algorithm
branch
from
d38cc07 to
c0c4a8d
Compare
guilload
force-pushed
the
guilload/improve-shard-rebalance-algorithm
branch
from
c0c4a8d to
6489c80
Compare
No description provided.