Skip to content

[feature] Support display_name for Kubernetes Pod Names #12434

@wassimbensalem

Description

@wassimbensalem

Enable set_display_name() to affect Kubernetes pod names for container tasks,
allowing human-readable pod names instead of auto-generated ones. This improves
cluster-side debugging and resource monitoring, especially for tasks within loops.

Benefits:

  • Easier debugging: Pod names match display names, making it easier to identify
    tasks in kubectl, logs, and monitoring tools
  • Resource monitoring: In ParallelFor loops, each iteration's pod has a clear
    name, enabling per-component resource usage tracking
  • Better observability: Metrics and logs can be easily correlated with pipeline tasks

Changes:

  • Use display_name (sanitized) as DAG task key for container tasks
  • Support loop variables: task.set_display_name(f"task-{loop_var.field}")
  • Ensure uniqueness and enforce Kubernetes naming constraints
  • Fix producer task references in groups to use correct DAG keys

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions