Node Annotations are Only Evaluated on Startup and not Actively Watched During Runtime #676
Description
On an already existing kubernetes cluster with kube-router successfully running in route reflector mode (rr.server annotation on 3 master nodes, rr.client annotation on all the workers), my workflow for joining new worker nodes is like this (the cluster was created with kubeadm):
for each new node:
- run kubeadm join on the node
- kubectl annotate node kube-router.io/rr.client=23
- restart kube-router on the new node to put them in RR mode
Turns out the nodes previously joined to the cluster will receive routes for all the new nodes, but the new nodes will only receive routes for the "old" nodes. Restarting kube-router on the route reflector nodes solved this issue.
My troubleshooting suggested that whenever a new node joins, the rr server kube-router daemons will peer with that node right away, even before the rr.client annotation gets added to it. Thus, the new peer is assumed to be in full mesh mode even if in fact it will be in RR mode and will only peer with RR servers. BGP route reflector allows for BGP daemons in the same AS to peer in either full-mesh or RR mode. Export policies are such that RR servers will reflect advertisements:
- from RR peers to RR peers
- from full-mesh peers to RR peers
- from RR peers to full-mesh peers
- but NOT from full-mesh peers to other full-mesh peers (they're supposed to peer with each other in full-mesh, duh)
So the new nodes, assumed to be forming full-mesh, don't get advertisements about other new nodes.
Restarting RR servers forces them to reload the node list and annotations, and correctly peer with the new nodes in RR mode.
To fix this, kube-router should watch for annotation changes on nodes, and update its internal information about which nodes are forming full-mesh, and which are joined to a RR cluster.
While waiting for a fix to be implemented, I suggest the workaround to restart RR servers be documented for newcomers!