Snakemake resources does not trigger GPU node-pool on kubernetes

[mrburke00]

Using snakemake/snakemake:v8.20.3 for Kubernetes jobs.

I am trying to set GPU resources for a rule to execute on GKE, but the GPU node pool is never triggered.

snakemake --snakefile workflow/snakefile_stage3_totalvi.smk \
          --executor kubernetes \
          --software-deployment-method conda \
          --default-storage-provider gcs \
          --default-storage-prefix $BUCKET \
          --storage-gcs-project $PROJECT \
          --set-resources totalvi_train.smk:gpu=1 \
          --jobs 1

Ive also tried resources gpu=1 and default-resources gpu=1 as well as setting the resources gpu=1 directly in the rule itself.

Changing the mem and cores resources does in fact trigger the respective node pools so the problem is gpu specific.