Script for automating the upgrade to the dashboard v2 components from `kubeflow/kubeflow`

[kimwnasptd]

Checks

• I have searched the existing issues.
• My request is related to one of the components in the kubeflow/dashboard repository.

Motivation

Since we'll also be doing some changes to component names (ptal in #118 (comment)) and also be using the v2.0.0 tag for the dashboard components this will introduce some friction for users that will want to upgrade between the components of V1 that were hosted in kubeflow/kubeflow and the components hosted in this GH repo.

To alleviate this friction we could do the following:

1. Have a dedicated folder here regarding the needed artifacts for the upgrade
2. Create a script that will:
   1. Scale down resources of V1
   2. Apply the resources of this repo
   3. Delete the resources from V1
3. Create a dedicated README.md in the above folder, documenting how users can use the above script and how it works

With this we'll make it simpler for users to migrate between V1 and V2

Implementation

The main items to discuss are:

1. What folder to use
2. More details on the logic of the script above

Regarding the folder name, one approach is to use /hack like KServe has. I personally am not in favor of this name as it can tip users off, if we have the upgrade process in a hack folder. Another approach would be to have a dedicated /upgrade or /migration folder and put those files in.

Regarding the scripts logic, an initial brainstorm of ideas and attention points I could think of are:

1. We need to not delete any CRDs, as this will result in deleting the CRs in user namespaces as well
2. We will need to delete the older resources with labels, and need to ensure this is possible without deleting objects from V2

I'll be posting more updates on this as I'll be looking deeper into our existing manifests

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[kimwnasptd]

WDYT @juliusvonkohout @thesuperzapper @andyatmiami ?

[kimwnasptd]

/assign @kimwnasptd

[juliusvonkohout]

In kubeflow manifests we have /scripts and https://github.com/kubeflow/manifests#upgrading-and-extending

[kimwnasptd]

Ah nice! Then let's also follow this pattern

[kimwnasptd]

The script is straight forward. One thing I noticed is that we have a kustomization in kubeflow/manifests for an app that doesn't live in the app repo
https://github.com/kubeflow/manifests/blob/v1.10-branch/applications/centraldashboard/overlays/oauth2-proxy/kustomization.yaml

This is problematic now the manifests deviate. @juliusvonkohout let's ensure we first try to make changes to the manifests in the apps instead of having extra overlays in manifests.

In this case, we should make the default logout URL to align with oauth2-proxy by default here, or with an overlay.

[kimwnasptd]

The more concerning part though is that after I remove the V1 manifests and apply the V2 manifests, in an existing Kubeflow cluster, then requests from the Istio Gateway to the Dashboard are timing out.

If I port-forward the SVC of the dashboard though the request completes as expected and returns the index.html. So this would mean that the error would be from the communication between the Gateway <> SVC.

Looking into it.

[kimwnasptd]

The reason the request between the IngressGateway and the dashboard SVC was because of the NetworkPolicy that gets applied from the manifests repo
https://github.com/kubeflow/manifests/blob/v1.10-branch/common/networkpolicies/base/centraldashboard.yaml

This is the same problem where we should be keeping any manifests that update other components in the components repo. In this case the NetworkPolicies should be moved to the component's repo, for cases like this.

For now I'll create a PR to move the NetworkPolicy in this repo, and have it allow requests to the new dashboard component.

cc @juliusvonkohout

[kimwnasptd]

After setting the NetworkPolicy with the expected label selector, when connecting to the dashboard I would get an RBAC: Access Denied error in a UI popup.

Looking a bit around it looks like the culprit is the AuthorizationPolicy that we currently have, that needs to be updated to point to dashboard instead of centraldashboard

dashboard/components/profile-controller/config/overlays/kubeflow/authorizationpolicy.yaml

Line 11 in 6eb7103

- cluster.local/ns/kubeflow/sa/centraldashboard

[kimwnasptd]

Following up, after I try to create a Notebook and mount a PodDefault the StatefulSet never manages to create the Pod.

I see the following K8s Event to be associated with the STS:

statefulset/test: create Pod test-0 in StatefulSet test failed error: Internal error occurred: failed calling webhook "deployment.kubeflow.org": failed to call webhook: Post "https://poddefaults-webhook-service.kubeflow.svc:443/apply-poddefault?timeout=10s": tls: failed to verify certificate: x509: certificate is valid for $(podDefaultsServiceName).$(podDefaultsNamespace).svc, $(podDefaultsServiceName).$(podDefaultsNamespace).svc.cluster.local, not poddefaults-webhook-service.kubeflow.svc

After investigating the created Certificate CR, I see that it's pointing to the webhook-secret K8s Secret (we should change this also to be poddefaults-webhook-secret). Which in turn has a tls.crt file that indeed points to the following fields:

Common Name: $(podDefaultsServiceName).$(podDefaultsNamespace).svc
Subject Alternative Names: $(podDefaultsServiceName).$(podDefaultsNamespace).svc, $(podDefaultsServiceName).$(podDefaultsNamespace).svc.cluster.local
Organization:
Organization Unit:
Locality:
State:
Country:
Valid From: October 10, 2025
Valid To: January 8, 2026
Serial Number: 0e2c5f6b0a647b8fec2348352f073633

This most probably means that the kustomize substitution didn't work as expected. As previously it had the Common Name to be admission-webhook-service.kubeflow.svc.

Looking into it

[kimwnasptd]

The root cause of the above was this PR #94

It aimed to move away from vars, which are being deprecated from kustomize, in favour of replacements. But the PR did not address the use of vars that we have in the overlays/cert-manager

dashboard/components/poddefaults-webhooks/manifests/overlays/cert-manager/certificate.yaml

Lines 7 to 10 in 6eb7103

	commonName: $(podDefaultsServiceName).$(podDefaultsNamespace).svc
	dnsNames:
	- $(podDefaultsServiceName).$(podDefaultsNamespace).svc
	- $(podDefaultsServiceName).$(podDefaultsNamespace).svc.cluster.local

[juliusvonkohout]

The reason the request between the IngressGateway and the dashboard SVC was because of the NetworkPolicy that gets applied from the manifests repo https://github.com/kubeflow/manifests/blob/v1.10-branch/common/networkpolicies/base/centraldashboard.yaml

This is the same problem where we should be keeping any manifests that update other components in the components repo. In this case the NetworkPolicies should be moved to the component's repo, for cases like this.

For now I'll create a PR to move the NetworkPolicy in this repo, and have it allow requests to the new dashboard component.

cc @juliusvonkohout

But lets carefully align on this. We have to make sure to not break the kubeflow/manifests master and there are also general per namespace network policies. But just copying (not moving) should be fine right now. Also regarding #154 (comment) lets discuss how we upstream what first.