Use intstr.IntOrString for cores in SparkApplication

[rmannibucau]

What feature you would like to be added?

I'd like to be able to give to the driver container < 1 core (100m)

Why is this needed?

Currently cores is an int(32)

Describe the solution you would like

intstr.IntOrString or Value sounds like options to have int-or-string support like in any k8s container

Describe alternatives you have considered

mutation/webhooks but not frictionless

Additional context

No response

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[ChenYi015]

Value of sparkapplication.spec.[driver|executor].cores will be passed to spark-submit script as parameters --driver-cores or --executor-cores. Currently, these two parameters are required to be integers.

$ ${SPARK_HOME}/bin/spark-submit --help
...
 Cluster deploy mode only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: 1).
...
 Spark standalone, YARN and Kubernetes only:
  --executor-cores NUM        Number of cores used by each executor. (Default: 1 in
                              YARN and K8S modes, or all available cores on the worker
                              in standalone mode).

[rmannibucau]

@ChenYi015 guess it is fine to pass 0.125 or 100m and --driver-cores 1 (ceiled value)

[ChenYi015]

@ChenYi015 guess it is fine to pass 0.125 or 100m and --driver-cores 1 (ceiled value)

Yes. It is possible to ahchieve this with the webhook. When a driver/executor pod is created, the webhook will mutate the cores to what sparkapp spec specifies.