Support Kubernetes Sidecars for Katib Metrics Collectors

[andreyvelich]

Recently Kubernetes added native support for sidecars containers as part of KEP 753: kubernetes/enhancements#3761

We need to discuss if we can improve our architecture to run Katib Metrics Collectors as Kubernetes sidecars.
It requires to use initContainer with restartPolicy: Always to run the container as a sidecar.

This feature will be available only in Kubernetes 1.28, but we can start the design discussions now.

Existing problems with Katib Metrics Collectors Sidecar:

• If training container is finished before Metrics Collector is started, the Trial will fail.
• Since we override Trial start command, the Trial might fail. E.g. Add support for entire kubeflow pipelines as trial target (in addition to containers) #1914 (comment).

cc @kubeflow/wg-automl-leads @tenzen-y @gaocegege @votti

[tenzen-y]

@andreyvelich Thank you for raising this proposal!
I agree with you because by supporting the sidecars pattern, we can avoid manually managing the termination of the metrics collector.

https://github.com/kubeflow/katib/blob/master/pkg/metricscollector/v1beta1/common/pns.go.

Also, we might be able to support Istio.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tenzen-y]

/lifecycle freeze

[tenzen-y]

/lifecycle frozen

[Electronic-Waste]

/remove-label lifecycle/frozen

[google-oss-prow]

@Electronic-Waste: The label(s) /remove-label lifecycle/frozen cannot be applied. These labels are supported: tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, lifecycle/needs-triage. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/remove-label lifecycle/frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.