TTL for TrainJobs

[stivanov-intercom]

What you would like to be added?

Kubernetes Jobs and JobSets support the TTL field that enables automatic resource cleanup. The TrainJob resource allows passing these TTLs down to the JobSet/Job level, which works for cleaning up pods, but the TrainJob resource itself remains indefinitely. It would be nice if there would be a TTL option implemented for the TrainJob resource, so we don't have to create additional systems on top to clean them up.

Why is this needed?

Without this, all the trainjob resources live on the cluster indefinitely after they have finished running and users are required to either manually clean them up or create cron jobs to clean them up. Would be nice to have parity with k8s jobs/jobsets.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[kannon92]

I believe this is possible via ClusterTrainingRuntimes or TrainingRuntimes.

Have you tried modifying ttl feature in JobSet for https://www.kubeflow.org/docs/components/trainer/operator-guides/job-template/?

[stivanov-intercom]

Yes, we are using the TTL at TrainingRuntimes - but that's a TTL set at the JobSet/Job level which doesn't clean up the TrainJob resource. So ultimately most of the resources are indeed cleaned up, except for the TrainJob.

[kannon92]

Ah, that makes total sense.

Yea, I think this is a valid feature request.

[andreyvelich]

Thanks for creating this @stivanov-intercom!

Yeah, this is a good point, that other resources that TrainJob controller creates won't be cleaned up.
For example, Secret or ConfigMap for MPI-based runtimes. Additionally, for non-JobSet runtimes like Flux that might be useful: #2841 or Slurm: #2249
cc @vsoch

I am curious whether we should introduce several APIs for it, like:

TTLSecondsAfterFinished
ActiveDeadlineSeconds

@kubeflow/kubeflow-trainer-team Any thoughts ?

/remove-label lifecycle/needs-triage
/area api
/area controller