Skip to content

Commit 14440b1

Browse files
gdaskbruelea
gdask
and
bruelea
authored
Add multi-cluster utilities & Readme (#457)
* Add multi-cluster utilities & Readme * Add copy info on script * Apply suggestions from code review Apply Lea's suggestions for better README readability Co-authored-by: bruelea <[email protected]> * Add support for cluster deployment * fix cluster names in readme * Add comment on copied file from multicluster runtime project Co-authored-by: bruelea <[email protected]> * adding new lines at the end of files * fix spelling mistakes * fix yaml lint error --------- Co-authored-by: bruelea <[email protected]>
1 parent 1c6e7ca commit 14440b1

File tree

5 files changed

+765
-0
lines changed

5 files changed

+765
-0
lines changed

‎multi-cluster-utilities/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Multicluster Configuration
2+
The create-kubeconfig-secret.sh is copied from [multicluster-runtime project](https://github.com/kubernetes-sigs/multicluster-runtime/tree/main), especially, from the `Kubeconfig Provider Example`.
3+
4+
This Readme cover only what is relevant for setting up a `Management cluster` which hosts the kubernetes operator, and `Resource Clusters` which host the Netbox Operator Resources. For more information over the scripts, and how a multicluster setup could be setup with Kubeconfig provider please read [this](https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/examples/kubeconfig/README.md)
5+
6+
## 1. Create Management Cluster
7+
8+
Follow the guide in project's root folder.
9+
If all the prerequisites are in place, then `make create-kind` creates the cluster which is going to be used for the all the controller's dependencies (netbox backend, databases etc).
10+
This cluster is also configured with the netbox-operator CRDs but the CRs are hosted and reconciled only in Resource Clusters.
11+
12+
## 2. Create Resource Clusters
13+
14+
Create your Resource Clusters & Provision Netbox-Operator Custom Resource Definitions. For each resource cluster you want to create, execute:
15+
16+
- `kind create cluster --name <res-cluster-name>`
17+
- `make install`
18+
19+
Sidenote: kind create command changes the default context in which the kubectl commands point to. When you are done from this step, the kubectl context will be set at the last cluster you created.
20+
21+
## 3. Establish cross-cluster access
22+
23+
Set kubectl context back to `management cluster`
24+
`kubectl config use-context kind-kind`
25+
26+
Execute RBAC scripts for kubeconfig provider, towards each cluster you created in the previous step.
27+
- `./create-kubeconfig-secret.sh -c kind-<res-cluster-name>`
28+
29+
For each cluster that gets configured as a 'Resource' cluster, a secret is populated in the 'Management' cluster.
30+
Make sure that the appropriate secrets are populated in the kind-kind cluster, with names `kind-<res-cluster-name>`.
31+
32+
## 4-1. Execute manager process locally
33+
At this point, you should be able to successfully start netbox operator process locally after:
34+
- Establishing a port forward from Management cluster to your host for the Netbox service: `kubectl port-forward deploy/netbox 8080:8080 -n default`
35+
- Setting environment variable `export NETBOX_HOST=localhost:8080`
36+
37+
## 4-2. Execute manager process on management cluster
38+
Deploying the manager in the management cluster involves additional manual steps.
39+
40+
From the project's parent directory, execute `make deploy-kind`
41+
42+
### Patch cluster role of netbox operator manager
43+
ClusterRole `manager-role` needs to allow reading, listing and watching secrets.
44+
- Execute `patch-netbox-clusterrole.sh`
45+
46+
### Update secret in controller cluster
47+
The secrets generated from the `create-kubeconfig-secret` needs to refer to the correct ip:port for each resource cluster. Currently it's pointing a localhost ip, which is only reachable from the host machine.
48+
- Execute script `create-kubeconfig-secret-cluster.sh -c kind-<res-cluster-name> --skip-create-rbac`
49+
- This script updates the secret on management cluster, to use the IP of control-plane node of the resource cluster, retrieved from ``docker inspect <res-cluster-name>-control-plane | jq '.[0].NetworkSettings.Networks.kind.IPAddress'``
50+
- The port of the K8s API server is assumed to be `6443`. You can check it with `docker inspect <res-cluster-name>-control-plane | jq '.[0].NetworkSettings.Ports'`
51+
52+
## 5. Test Reconciliation
53+
Apply an example CR in resource cluster and check if it's getting reconciled.
54+
`kubectl --context kind-<res-cluster-name> apply -f config/samples/netbox_v1_ipaddress.yaml`

‎multi-cluster-utilities/create-kubeconfig-secret-cluster.sh‎

Lines changed: 324 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,324 @@
1+
# File copied from https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/examples/kubeconfig/README.md
2+
// Modified by Swisscom (Schweiz) AG.
3+
#!/bin/bash
4+
5+
# Script to create a kubeconfig secret for the pod lister controller
6+
7+
set -e
8+
9+
# Default values
10+
NAMESPACE="default"
11+
SERVICE_ACCOUNT="multicluster-kubeconfig-provider"
12+
KUBECONFIG_CONTEXT=""
13+
SECRET_NAME=""
14+
ROLE_TYPE="clusterrole"
15+
RULES_FILE=""
16+
CREATE_RBAC="true"
17+
18+
# Check for yq
19+
if ! command -v yq &>/dev/null; then
20+
echo "ERROR: 'yq' is required but not installed. Please install yq (https://mikefarah.gitbook.io/yq/) and try again."
21+
exit 1
22+
fi
23+
24+
# Function to display usage information
25+
function show_help {
26+
echo "Usage: $0 [options]"
27+
echo " -c, --context CONTEXT Kubeconfig context to use (required)"
28+
echo " --name NAME Name for the secret (defaults to context name)"
29+
echo " -n, --namespace NS Namespace to create the secret in (default: ${NAMESPACE})"
30+
echo " -a, --service-account SA Service account name to use (default: ${SERVICE_ACCOUNT})"
31+
echo " -t, --role-type TYPE Create Role or ClusterRole (role|clusterrole) (default: clusterrole)"
32+
echo " -r, --rules-file FILE Path to rules file (default: rules.yaml in script directory)"
33+
echo " --skip-create-rbac Skip creating RBAC resources (Role/ClusterRole and bindings)"
34+
echo " -h, --help Show this help message"
35+
echo ""
36+
echo "Examples:"
37+
echo " $0 -c prod-cluster"
38+
echo " $0 -c prod-cluster -t role -r ./custom-rules.yaml"
39+
echo " $0 -c prod-cluster -t clusterrole"
40+
echo " $0 -c prod-cluster --skip-create-rbac"
41+
}
42+
43+
# Function to create Role or ClusterRole
44+
function create_rbac {
45+
local role_type="$1"
46+
local rules_file="$2"
47+
local role_name="$3"
48+
local namespace="$4"
49+
50+
if [ ! -f "$rules_file" ]; then
51+
echo "ERROR: Rules file not found: $rules_file"
52+
exit 1
53+
fi
54+
55+
echo "Creating ${role_type} '${role_name}'..."
56+
57+
if [ "$role_type" = "role" ]; then
58+
# Create Role
59+
ROLE_YAML=$(cat <<EOF
60+
apiVersion: rbac.authorization.k8s.io/v1
61+
kind: Role
62+
metadata:
63+
name: ${role_name}
64+
namespace: ${namespace}
65+
rules:
66+
$(yq '.rules' "$rules_file")
67+
EOF
68+
)
69+
70+
echo "$ROLE_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
71+
72+
# Create RoleBinding
73+
ROLEBINDING_YAML=$(cat <<EOF
74+
apiVersion: rbac.authorization.k8s.io/v1
75+
kind: RoleBinding
76+
metadata:
77+
name: ${role_name}-binding
78+
namespace: ${namespace}
79+
subjects:
80+
- kind: ServiceAccount
81+
name: ${SERVICE_ACCOUNT}
82+
namespace: ${namespace}
83+
roleRef:
84+
kind: Role
85+
name: ${role_name}
86+
apiGroup: rbac.authorization.k8s.io
87+
EOF
88+
)
89+
90+
echo "$ROLEBINDING_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
91+
92+
else
93+
# Create ClusterRole
94+
CLUSTERROLE_YAML=$(cat <<EOF
95+
apiVersion: rbac.authorization.k8s.io/v1
96+
kind: ClusterRole
97+
metadata:
98+
name: ${role_name}
99+
rules:
100+
$(yq '.rules' "$rules_file")
101+
EOF
102+
)
103+
104+
echo "$CLUSTERROLE_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
105+
106+
# Create ClusterRoleBinding
107+
CLUSTERROLEBINDING_YAML=$(cat <<EOF
108+
apiVersion: rbac.authorization.k8s.io/v1
109+
kind: ClusterRoleBinding
110+
metadata:
111+
name: ${role_name}-binding
112+
subjects:
113+
- kind: ServiceAccount
114+
name: ${SERVICE_ACCOUNT}
115+
namespace: ${namespace}
116+
roleRef:
117+
kind: ClusterRole
118+
name: ${role_name}
119+
apiGroup: rbac.authorization.k8s.io
120+
EOF
121+
)
122+
123+
echo "$CLUSTERROLEBINDING_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
124+
fi
125+
126+
echo "$(tr '[:lower:]' '[:upper:]' <<< ${role_type:0:1})${role_type:1} '${role_name}' created successfully!"
127+
}
128+
129+
# Function to create service account if it doesn't exist
130+
function ensure_service_account {
131+
local namespace="$1"
132+
local service_account="$2"
133+
134+
echo "Checking if service account '${service_account}' exists in namespace '${namespace}'..."
135+
136+
# Check if service account exists
137+
if ! kubectl --context=${KUBECONFIG_CONTEXT} get serviceaccount ${service_account} -n ${namespace} &>/dev/null; then
138+
echo "Service account '${service_account}' not found in namespace '${namespace}'. Creating..."
139+
140+
# Create the service account
141+
SERVICE_ACCOUNT_YAML=$(cat <<EOF
142+
apiVersion: v1
143+
kind: ServiceAccount
144+
metadata:
145+
name: ${service_account}
146+
namespace: ${namespace}
147+
EOF
148+
)
149+
150+
echo "$SERVICE_ACCOUNT_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
151+
echo "Service account '${service_account}' created successfully in namespace '${namespace}'"
152+
else
153+
echo "Service account '${service_account}' already exists in namespace '${namespace}'"
154+
fi
155+
}
156+
157+
# Parse command line options
158+
while [[ $# -gt 0 ]]; do
159+
key="$1"
160+
case $key in
161+
--name)
162+
SECRET_NAME="$2"
163+
shift 2
164+
;;
165+
-n|--namespace)
166+
NAMESPACE="$2"
167+
shift 2
168+
;;
169+
-c|--context)
170+
KUBECONFIG_CONTEXT="$2"
171+
shift 2
172+
;;
173+
-a|--service-account)
174+
SERVICE_ACCOUNT="$2"
175+
shift 2
176+
;;
177+
-t|--role-type)
178+
ROLE_TYPE="$2"
179+
shift 2
180+
;;
181+
-r|--rules-file)
182+
RULES_FILE="$2"
183+
shift 2
184+
;;
185+
--skip-create-rbac)
186+
CREATE_RBAC="false"
187+
shift
188+
;;
189+
-h|--help)
190+
show_help
191+
exit 0
192+
;;
193+
*)
194+
echo "Unknown option: $1"
195+
show_help
196+
exit 1
197+
;;
198+
esac
199+
done
200+
201+
# Validate required arguments
202+
if [ -z "$KUBECONFIG_CONTEXT" ]; then
203+
echo "ERROR: Kubeconfig context is required (-c, --context)"
204+
show_help
205+
exit 1
206+
fi
207+
208+
# Validate role type if specified
209+
if [ -n "$ROLE_TYPE" ] && [ "$ROLE_TYPE" != "role" ] && [ "$ROLE_TYPE" != "clusterrole" ]; then
210+
echo "ERROR: Invalid role type '$ROLE_TYPE'. Must be 'role' or 'clusterrole'"
211+
show_help
212+
exit 1
213+
fi
214+
215+
# Set default rules file if not specified
216+
if [ -z "$RULES_FILE" ]; then
217+
RULES_FILE="$(dirname "$0")/rules.yaml"
218+
fi
219+
220+
# Set secret name to context if not specified
221+
if [ -z "$SECRET_NAME" ]; then
222+
SECRET_NAME="$KUBECONFIG_CONTEXT"
223+
fi
224+
225+
# Create RBAC resources by default (unless --no-create-role is specified)
226+
if [ "$CREATE_RBAC" = "true" ]; then
227+
create_rbac "$ROLE_TYPE" "$RULES_FILE" "$SECRET_NAME" "$NAMESPACE"
228+
fi
229+
230+
# Get the cluster CA certificate from the remote cluster
231+
CLUSTER_CA=$(kubectl --context=${KUBECONFIG_CONTEXT} config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}')
232+
if [ -z "$CLUSTER_CA" ]; then
233+
echo "ERROR: Could not get cluster CA certificate"
234+
exit 1
235+
fi
236+
237+
# Get the cluster server URL from the remote cluster
238+
CLUSTER_SERVER=$(kubectl --context=${KUBECONFIG_CONTEXT} config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
239+
if [ -z "$CLUSTER_SERVER" ]; then
240+
echo "ERROR: Could not get cluster server URL"
241+
exit 1
242+
fi
243+
244+
# Ensure service account exists
245+
ensure_service_account "$NAMESPACE" "$SERVICE_ACCOUNT"
246+
247+
# Get the service account token from the remote cluster
248+
SA_TOKEN=$(kubectl --context=${KUBECONFIG_CONTEXT} -n ${NAMESPACE} create token ${SERVICE_ACCOUNT} --duration=8760h)
249+
if [ -z "$SA_TOKEN" ]; then
250+
echo "ERROR: Could not create service account token"
251+
exit 1
252+
fi
253+
254+
# Get the cluster's external IP
255+
CONTROL_PLANE_NAME="${SECRET_NAME#kind-}"-control-plane
256+
CLUSTER_EXTERNAL_IP=$(docker inspect ${CONTROL_PLANE_NAME} | jq -r '.[0].NetworkSettings.Networks.kind.IPAddress')
257+
if [ -z "$CLUSTER_EXTERNAL_IP" ]; then
258+
echo "ERROR: Could not get cluster external IP"
259+
exit 1
260+
fi
261+
262+
# Create a new kubeconfig using the service account token
263+
NEW_KUBECONFIG=$(cat <<EOF
264+
apiVersion: v1
265+
kind: Config
266+
clusters:
267+
- name: ${SECRET_NAME}
268+
cluster:
269+
server: https://${CLUSTER_EXTERNAL_IP}:6443
270+
certificate-authority-data: ${CLUSTER_CA}
271+
contexts:
272+
- name: ${SECRET_NAME}
273+
context:
274+
cluster: ${SECRET_NAME}
275+
user: ${SERVICE_ACCOUNT}
276+
current-context: ${SECRET_NAME}
277+
users:
278+
- name: ${SERVICE_ACCOUNT}
279+
user:
280+
token: ${SA_TOKEN}
281+
EOF
282+
)
283+
284+
# Save kubeconfig temporarily for testing
285+
TEMP_KUBECONFIG=$(mktemp)
286+
echo "$NEW_KUBECONFIG" > "$TEMP_KUBECONFIG"
287+
288+
# echo "$NEW_KUBECONFIG"
289+
# # Verify the kubeconfig works
290+
# echo "Verifying kubeconfig..."
291+
# if kubectl --kubeconfig="$TEMP_KUBECONFIG" version &>/dev/null; then
292+
# rm "$TEMP_KUBECONFIG"
293+
# echo "ERROR: Failed to verify kubeconfig - unable to connect to cluster."
294+
# echo "- Ensure that the service account '${NAMESPACE}/${SERVICE_ACCOUNT}' on cluster '${KUBECONFIG_CONTEXT}' exists and is properly configured."
295+
# echo "- You may specify a namespace using the -n flag."
296+
# echo "- You may specify a service account using the -a flag."
297+
# exit 1
298+
# fi
299+
# echo "Kubeconfig verified successfully!"
300+
301+
# Encode the verified kubeconfig
302+
KUBECONFIG_B64=$(cat "$TEMP_KUBECONFIG" | base64 -w0)
303+
rm "$TEMP_KUBECONFIG"
304+
305+
# Generate and apply the secret
306+
SECRET_YAML=$(cat <<EOF
307+
apiVersion: v1
308+
kind: Secret
309+
metadata:
310+
name: ${SECRET_NAME}
311+
namespace: ${NAMESPACE}
312+
labels:
313+
sigs.k8s.io/multicluster-runtime-kubeconfig: "true"
314+
type: Opaque
315+
data:
316+
kubeconfig: ${KUBECONFIG_B64}
317+
EOF
318+
)
319+
320+
echo "Creating kubeconfig secret..."
321+
echo "$SECRET_YAML" | kubectl apply -f -
322+
323+
echo "Secret '${SECRET_NAME}' created in namespace '${NAMESPACE}'"
324+
echo "The operator should now be able to discover and connect to this cluster"

0 commit comments

Comments
 (0)