node pools for the DRA instructions don't scale down because the nvidia-driver-installer manual install doesn't have safe-to-evict annotation.
Adding it like so fixed the problem:
kubectl patch daemonset nvidia-driver-installer -n kube-system -p
'{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.
kubernetes.io/safe-to-evict": "true"}}}}}'