wip: Fix image loading with podman driver #21932

obnoxxx · 2025-11-19T19:53:28Z

This fixes the image loading for the podman driver removing the annoying message

Resolves: #8426

without this patch

$ git branch --show-current
master
$ git fetch origin
...
$  git reset --hard origin/master
HEAD is now at bac7021992 Merge pull request #21933 from medyagh/fix_update_golang
$ make clean && make
$ ./out/minikube  profile list
┌───────────────────┬────────┬─────────┬──────────────┬─────────┬─────────┬───────┬────────────────┬────────────────────┐
│      PROFILE      │ DRIVER │ RUNTIME │      IP      │ VERSION │ STATUS  │ NODES │ ACTIVE PROFILE │ ACTIVE KUBECONTEXT │
├───────────────────┼────────┼─────────┼──────────────┼─────────┼─────────┼───────┼────────────────┼────────────────────┤
│ functional-865055 │ docker │ crio    │ 192.168.49.2 │ v1.34.1 │ Stopped │ 1     │                │                    │
│ functional-981737 │ docker │ crio    │ 192.168.58.2 │ v1.34.1 │ Stopped │ 1     │                │                    │
│ minikube          │ kvm2   │ docker  │              │ v1.33.0 │ Stopped │ 1     │ *              │                    │
└───────────────────┴────────┴─────────┴──────────────┴─────────┴─────────┴───────┴────────────────┴────────────────────┘
11:21:42 obnox@cone minikube ±|master ✗|→ ./out/minikube delete --all
🔥  Deleting "functional-865055" in docker ...
🔥  Removing /home/obnox/.minikube/machines/functional-865055 ...
💀  Removed all traces of the "functional-865055" cluster.
🔥  Deleting "functional-981737" in docker ...
🔥  Removing /home/obnox/.minikube/machines/functional-981737 ...
💀  Removed all traces of the "functional-981737" cluster.
🔥  Deleting "minikube" in kvm2 ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles
 $ time ./out/minikube start -d podman 
😄  minikube v1.37.0 on Fedora 43
✨  Using the podman driver based on user configuration
📌  Using Podman driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.48-1763507788-21924 ...
💾  Downloading Kubernetes v1.34.1 preload ...
    > preloaded-images-k8s-v18-v1...:  337.01 MiB / 337.01 MiB  100.00% 20.71 M
    > gcr.io/k8s-minikube/kicbase...:  496.38 MiB / 496.38 MiB  100.00% 6.45 Mi
E1120 11:24:14.124011   40829 cache.go:238] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=7900MB) ...
🐳  Preparing Kubernetes v1.34.1 on Docker 29.0.2 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

real    2m25.750s
user    0m10.176s
sys     0m4.657s
$  echo $?
0
$ ./out/minikube  profile list
┌──────────┬────────┬─────────┬──────────────┬─────────┬────────┬───────┬────────────────┬────────────────────┐
│ PROFILE  │ DRIVER │ RUNTIME │      IP      │ VERSION │ STATUS │ NODES │ ACTIVE PROFILE │ ACTIVE KUBECONTEXT │
├──────────┼────────┼─────────┼──────────────┼─────────┼────────┼───────┼────────────────┼────────────────────┤
│ minikube │ podman │ docker  │ 192.168.49.2 │ v1.34.1 │ OK     │ 1     │ *              │ *                  │
└──────────┴────────┴─────────┴──────────────┴─────────┴────────┴───────┴────────────────┴────────────────────┘
$ kubectl get no
NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   96s   v1.34.1
$ kubectl get ns
NAME              STATUS   AGE
default           Active   99s
kube-node-lease   Active   99s
kube-public       Active   99s
kube-system       Active   99s

so there is the error message but it seems to have succeeded.

with the patch

$ git fetch obnox 
From github.com:obnoxxx/minikube
 * [new branch]            podman-fix-image-load -> obnox/podman-fix-image-load
$ git checkout podman-fix-image-load 
branch 'podman-fix-image-load' set up to track 'obnox/podman-fix-image-load'.
Switched to a new branch 'podman-fix-image-load'
$ make clean && make
$ ./out/minikube  profile list
┌──────────┬────────┬─────────┬──────────────┬─────────┬────────┬───────┬────────────────┬────────────────────┐
│ PROFILE  │ DRIVER │ RUNTIME │      IP      │ VERSION │ STATUS │ NODES │ ACTIVE PROFILE │ ACTIVE KUBECONTEXT │
├──────────┼────────┼─────────┼──────────────┼─────────┼────────┼───────┼────────────────┼────────────────────┤
│ minikube │ podman │ docker  │ 192.168.49.2 │ v1.34.1 │ OK     │ 1     │ *              │ *                  │
└──────────┴────────┴─────────┴──────────────┴─────────┴────────┴───────┴────────────────┴────────────────────┘
$ ./out/minikube  delete --all
🔥  Deleting "minikube" in podman ...
🔥  Removing /home/obnox/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles
$ time ./out/minikube start -d podman 
😄  minikube v1.37.0 on Fedora 43
✨  Using the podman driver based on user configuration
📌  Using Podman driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.48-1761985721-21837 ...
🔥  Creating podman container (CPUs=2, Memory=7900MB) ...
🐳  Preparing Kubernetes v1.34.1 on Docker 28.5.1 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

real    0m35.878s
user    0m3.116s
sys     0m1.015s
11:44:11 obnox@cone minikube ±|podman-fix-image-load ✗|→ echo $?
0
11:45:38 obnox@cone minikube ±|podman-fix-image-load ✗|→ ./out/minikube  profile list
┌──────────┬────────┬─────────┬──────────────┬─────────┬────────┬───────┬────────────────┬────────────────────┐
│ PROFILE  │ DRIVER │ RUNTIME │      IP      │ VERSION │ STATUS │ NODES │ ACTIVE PROFILE │ ACTIVE KUBECONTEXT │
├──────────┼────────┼─────────┼──────────────┼─────────┼────────┼───────┼────────────────┼────────────────────┤
│ minikube │ podman │ docker  │ 192.168.49.2 │ v1.34.1 │ OK     │ 1     │ *              │ *                  │
└──────────┴────────┴─────────┴──────────────┴─────────┴────────┴───────┴────────────────┴────────────────────┘
$ kubectl get no
NAME       STATUS   ROLES           AGE    VERSION
minikube   Ready    control-plane   113s   v1.34.1
$ kubectl get ns
NAME              STATUS   AGE
default           Active   117s
kube-node-lease   Active   117s
kube-public       Active   117s
kube-system       Active   117s

So this is working fine.

k8s-ci-robot · 2025-11-19T19:53:38Z

Hi @obnoxxx. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-11-19T19:57:03Z

Can one of the admins verify this patch?

medyagh

can you plz provide before/after this PR output?

medyagh · 2025-11-19T21:03:18Z

pkg/minikube/node/cache.go

-				return fmt.Errorf("not yet implemented, see issue #8426")
-			}
-			if driver.IsDocker(cc.Driver) && err == nil {
+			if err == nil && (driver.IsDocker(cc.Driver) || (cc.Driver == driver.Podman)) {


you can use driver.IsKIC(cc.Driver) here

done. Thanks for the hint!

obnoxxx · 2025-11-19T21:09:18Z

can you plz provide before/after this PR output?

Will do, tomorrow.

obnoxxx · 2025-11-20T12:07:32Z

@medyagh wrote:

can you plz provide before/after this PR output?

done.

obnoxxx · 2025-11-20T13:31:58Z

/assign @medyagh

medyagh · 2025-11-20T19:39:27Z

/ok-to-test

medyagh

plz run make gomodtidy

obnoxxx · 2025-11-21T11:07:58Z

plz run make gomodtidy

See the first commit. I had done that first.

k8s-ci-robot · 2025-11-21T12:29:53Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: obnoxxx
Once this PR has been reviewed and has the lgtm label, please ask for approval from medyagh. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

obnoxxx · 2025-11-21T12:33:53Z

plz run make gomodtidy

See the first commit. I had done that first.

I ave done it again now and squashed with the first commit.

obnoxxx · 2025-11-21T12:36:13Z

/retest-required

This fixes issue 8426 Signed-off-by: Michael Adam <[email protected]>

k8s-ci-robot · 2025-11-24T12:54:06Z

@obnoxxx: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
integration-kvm-containerd-linux-x86-64	`84a896b`	link	true	`/test integration-kvm-containerd-linux-x86-64`
integration-kvm-crio-linux-x86-64	`84a896b`	link	true	`/test integration-kvm-crio-linux-x86-64`
integration-docker-crio-linux-x86-64	`84a896b`	link	true	`/test integration-docker-crio-linux-x86-64`
integration-docker-containerd-linux-x86-64	`84a896b`	link	true	`/test integration-docker-containerd-linux-x86-64`
integration-none-docker-linux-x86-64	`84a896b`	link	true	`/test integration-none-docker-linux-x86-64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

obnoxxx · 2025-11-24T12:55:32Z

for some reason, when I checked it now, the gomodtidy commit had an empty diff.

So I removed it from the PR.

Locally, make gomodtidy produces no diff for me.

Not sure what was different before.

minikube-pr-bot · 2025-11-24T13:42:38Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21932 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 42.5s    │ 42.3s                  │
│ enable ingress │ 16.7s    │ 16.4s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 42.7s 43.6s 40.8s 44.4s 40.8s
Times for minikube (PR 21932) start: 41.7s 40.9s 42.4s 43.2s 43.2s

Times for minikube (PR 21932) ingress: 15.8s 15.8s 18.8s 15.8s 15.8s
Times for minikube ingress: 15.8s 19.8s 15.8s 16.3s 15.8s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21932 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.9s    │ 22.5s                  │
│ enable ingress │ 11.8s    │ 12.2s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 24.7s 20.9s 26.7s 21.5s 20.5s
Times for minikube (PR 21932) start: 24.9s 20.5s 20.7s 22.3s 24.2s

Times for minikube (PR 21932) ingress: 11.6s 12.6s 13.6s 12.7s 10.6s
Times for minikube ingress: 12.6s 10.6s 10.6s 12.6s 12.6s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21932 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 21.5s    │ 21.6s                  │
│ enable ingress │ 22.7s    │ 22.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 21.1s 21.2s 22.8s 21.7s 20.8s
Times for minikube (PR 21932) start: 22.7s 22.9s 19.5s 20.6s 22.5s

Times for minikube ingress: 22.1s 23.1s 23.1s 23.1s 22.1s
Times for minikube (PR 21932) ingress: 22.1s 23.1s 23.1s 23.1s 22.2s

minikube-pr-bot · 2025-11-24T14:31:10Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_containerd_arm64 (4 failed)	TestStartStop/group/old-k8s-version/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd_arm64 (4 failed)	TestStartStop/group/default-k8s-diff-port/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd_arm64 (4 failed)	TestStartStop/group/embed-certs/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd_arm64 (4 failed)	TestStartStop/group/no-preload/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd (4 failed)	TestStartStop/group/old-k8s-version/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd (4 failed)	TestStartStop/group/no-preload/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd (4 failed)	TestStartStop/group/embed-certs/serial/DeployApp(gopogh)	0.00% (chart)
Docker_Linux_containerd (4 failed)	TestStartStop/group/default-k8s-diff-port/serial/DeployApp(gopogh)	2.00% (chart)

Besides the following environments also have failed tests:

KVM_Linux_crio: 2 failed (gopogh)
Docker_Linux_crio: 37 failed (gopogh)
Docker_Linux_crio_arm64: 36 failed (gopogh)

To see the flake rates of all tests by environment, click here.

afbjorklund · 2025-11-24T17:11:13Z

pkg/minikube/node/cache.go

+
+				if driver.IsKIC(cc.Driver) {
+					klog.Infof("Loading %s from local cache", img)
+					if finalImg, err = download.CacheToDaemon(img); err == nil {


This function needs to add support for podman, currently it only supports docker...

The easiest is probably forking a call to exec, had do to that anyway (for the digest)

@afbjorklund wrote:

This function needs to add support for podman, currently it only supports docker...

The easiest is probably forking a call to exec, had do to that anyway (for the digest)

good point, but the ToDaemon part of the function's name does not really apply to podman, right?

You can either rename the function to more a more generic name, or make another one just for Podman.

Calling it "daemon" in the first place, was focusing on the wrong part of "Docker daemon" anyway...

daemon.Write (pkg/v1/daemon)

github.com/docker/docker/client

obnoxxx · 2025-11-26T16:02:18Z

My current thinking is to maybe just stop this effort anc close PR.

wdyt, @medyagh , @afbjorklund ?

afbjorklund · 2025-11-26T17:28:11Z

My current thinking is to maybe just stop this effort anc close PR.

You can close the PR, if you don't want to work on it further right now

It can either be reopened later, or you can open a new PR instead

Another approach would be to just change the wording of the message...

Since the download when just fine, it was the image load that was missing.
And also because the reference issue is closed, so nothing to "wait on"?

Error downloading kic artifacts: not yet implemented, see issue #8426

afbjorklund · 2025-11-26T17:53:15Z

@obnoxxx

If you do decide to work on it, you have the building blocks in pkg/drivers/kic/oci

func LoadImageFromPath(ctx context.Context, ociBin string, p string) error {
        rr, err := runCmd(exec.CommandContext(ctx, ociBin, "load", "-i", p))

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 19, 2025

k8s-ci-robot requested review from medyagh and prezha November 19, 2025 19:53

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 19, 2025

medyagh reviewed Nov 19, 2025

View reviewed changes

obnoxxx changed the title ~~ix image loading with podman driver~~ Fix image loading with podman driver Nov 19, 2025

medyagh requested changes Nov 19, 2025

View reviewed changes

obnoxxx force-pushed the podman-fix-image-load branch from d0996ef to 00018e5 Compare November 20, 2025 10:59

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 20, 2025

k8s-ci-robot assigned medyagh Nov 20, 2025

obnoxxx mentioned this pull request Nov 20, 2025

Minikube Is Failing To Find and Add Kic Artifact To Cache #21818

Open

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 20, 2025

medyagh reviewed Nov 20, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

obnoxxx force-pushed the podman-fix-image-load branch from 9c2833f to 922b90d Compare November 21, 2025 12:29

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 21, 2025

obnoxxx requested a review from medyagh November 21, 2025 12:53

This comment has been minimized.

Sign in to view

Fix image loading with podman driver

84a896b

This fixes issue 8426 Signed-off-by: Michael Adam <[email protected]>

obnoxxx force-pushed the podman-fix-image-load branch from 922b90d to 84a896b Compare November 24, 2025 12:52

afbjorklund requested changes Nov 24, 2025

View reviewed changes

medyagh changed the title ~~Fix image loading with podman driver~~ wip: Fix image loading with podman driver Nov 24, 2025

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 24, 2025

wip: Fix image loading with podman driver #21932

Are you sure you want to change the base?

wip: Fix image loading with podman driver #21932

Conversation

obnoxxx commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

without this patch

with the patch

Uh oh!

k8s-ci-robot commented Nov 19, 2025

Uh oh!

minikube-bot commented Nov 19, 2025

Uh oh!

medyagh left a comment

Choose a reason for hiding this comment

Uh oh!

medyagh Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

obnoxxx Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

obnoxxx commented Nov 19, 2025

Uh oh!

obnoxxx commented Nov 20, 2025

Uh oh!

obnoxxx commented Nov 20, 2025

Uh oh!

medyagh commented Nov 20, 2025

Uh oh!

medyagh left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

obnoxxx commented Nov 21, 2025

Uh oh!

k8s-ci-robot commented Nov 21, 2025

Uh oh!

obnoxxx commented Nov 21, 2025

Uh oh!

obnoxxx commented Nov 21, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

k8s-ci-robot commented Nov 24, 2025

Uh oh!

obnoxxx commented Nov 24, 2025

Uh oh!

minikube-pr-bot commented Nov 24, 2025

Uh oh!

minikube-pr-bot commented Nov 24, 2025

Uh oh!

afbjorklund Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

obnoxxx Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

afbjorklund Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

obnoxxx commented Nov 26, 2025

Uh oh!

afbjorklund commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

obnoxxx commented Nov 19, 2025 •

edited

Loading

afbjorklund Nov 26, 2025 •

edited

Loading

afbjorklund commented Nov 26, 2025 •

edited

Loading

afbjorklund commented Nov 26, 2025 •

edited

Loading