-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Open
Labels
Description
Enable set_display_name() to affect Kubernetes pod names for container tasks,
allowing human-readable pod names instead of auto-generated ones. This improves
cluster-side debugging and resource monitoring, especially for tasks within loops.
Benefits:
- Easier debugging: Pod names match display names, making it easier to identify
tasks in kubectl, logs, and monitoring tools - Resource monitoring: In ParallelFor loops, each iteration's pod has a clear
name, enabling per-component resource usage tracking - Better observability: Metrics and logs can be easily correlated with pipeline tasks
Changes:
- Use display_name (sanitized) as DAG task key for container tasks
- Support loop variables: task.set_display_name(f"task-{loop_var.field}")
- Ensure uniqueness and enforce Kubernetes naming constraints
- Fix producer task references in groups to use correct DAG keys