builder-noble-java-tiny:0.0.56 produces zero length layers which can cause tooling issuess

[philwebb]

Expected Behavior

When using builder-noble-java-tiny:0.0.56 to generate an image, all included layers should have a non-zero length. Zero byte layers seem to cause issues with some publishing tools and with tools such as dive.

This issue was first reported at spring-projects/spring-boot#47022.

Current Behavior

Running pack with builder-noble-java-tiny:0.0.56 produces a docker image that has zero byte layers.

Steps to Reproduce

A sample project to reproduce the issue is available in demo-image-issue.zip. It needs to be built with Apache Maven.

Once the project has been built, run pack to generate an image:

$ pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:latest --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar

Save the image and extract and view the manifest to inspect the JSON:

$ docker image save demo-image-issue | tar -xf - manifest.json && cat manifest.json | jq

The result is:

[
  {
    "Config": "blobs/sha256/0a05309c5285563eb3091f85153f5347c021a4b0d91ae9ec39583d0aa3be6014",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/3e74b876fa888a7265efcbe547e07018ccda0421797d9ae6cbf185d6d1d5508c",
      "blobs/sha256/e13f32b89503082d9d076df31b0ab4fe367f17308145e2e9777332f52351305f",
      "blobs/sha256/5f4c1d5773eb3a00ebec6070263004ec60faf7311ea4db1ac5ff045aecd165f8",
      "blobs/sha256/a4124ebfe5c4a63a1f5020c530f176f0fb62c2edc2e0c87220cb0bba6d159b74",
      "blobs/sha256/b8578b8a2ac51f7abdfdd4eea28bc34a73920771ba01df61f9f6d30f2bbc0dfe",
      "blobs/sha256/d6ecce2bcb621e3b99b3fadd213faf56a32a4d221855c6726d317a931b97401e",
      "blobs/sha256/2cdf5b962e8ece0e546450ea779424e3420db80186dfafc0a0f44d172cee1c2f",
      "blobs/sha256/bc7f82c871ffa12af89cd62c3047e5289ce24fbbe07f7adeeb840ccd43140f20",
      "blobs/sha256/8e326708c473538285f07c789947ca89fa19fbbd7ae08333ec883c0fcba4d7cc",
      "blobs/sha256/cc3c0e569c3cd1ea2ca983256c6dc1c4ef10fe3c381b1fa0c450925c257dde94",
      "blobs/sha256/24d5d2eaea1e2fc53ad9e081541acb105a2ae62a6dcffb875dec06b573149c0a",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/891c5ef4572e1828d36c43534cacffd20bda39314b768e995bc327f475e2be96",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/34061cc4ad826866fb162f006f0a8e7b98cf4c6d4ae8644335d5fce00c345027",
      "blobs/sha256/f7f72ce47a0b7ddbe653dbe7ae8304e39f8ac3f29a8f16cd0623223373b4936e",
      "blobs/sha256/b39b18afe74ec7ee652aed10c5c3e637af7e7daad3da9380043a93fe7d212d44"
    ]
  }
]

The top three layers (with sha e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) all have 0 bytes.

Running dive gives the following error:

$ dive  demo-image-issue
Image Source: docker://demo-image-issue-l
Extracting image from docker-engine... (this can take a while for large images)
cannot fetch image
EOF

Some Docker registries may also reject this manifest

Running the same pack command with 0.0.55 of the builder produces a different result:

$ pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:0.0.55 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar

$ docker image save demo-image-issue | tar -xf - manifest.json && cat manifest.json | jq

Now gives:

[
  {
    "Config": "blobs/sha256/5d6e19b261e7fec3ae6ac29665eeaea85acf8063f2ae7a26b63ddc05fc541583",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/8637476602cf2504187f2da7bc4b67fe11ea77b97d8506d33b791aaeac110e76",
      "blobs/sha256/72c283668f14198365eace6c60701babd1fa267eac2b647fbad0352d0c97af86",
      "blobs/sha256/aac046430871674bd6f2219bb468681c5b3338f1e06ab4c14197bb5885968ad3",
      "blobs/sha256/5b6f8eaa5fccd62d95fffce6fecfd06b35f5fd04417d0412e61339005d028490",
      "blobs/sha256/9f6e2a03234306101ca35e7e361bd3f08fc4d55beeeacd8c351204106e29aae3",
      "blobs/sha256/bea0a3dc2651cac7c9c567a5cb4e7536107b357cb9113e8806f690f050500012",
      "blobs/sha256/03160ce98d2ab327ffbb1a513d2d758f57214b22ffafc035151801da7e84ca5c",
      "blobs/sha256/417e5bfc3c82b9373cf6804206e071d2fc74560df867d0f39cb21ac3d15231b6",
      "blobs/sha256/e8dc1c799ea8bf3d9a939276b9b4a6ae15843738f1c1ef94239dc51b2ea62f3c",
      "blobs/sha256/f0e9078fd509b2493bcef1215b4de825d53f118ef0648b850edec03dc6e3cfd4",
      "blobs/sha256/366ce7d1a7f90f2e4ad08752f87510eee3ffca18736fa63c03823c8c4ebf2925",
      "blobs/sha256/a7dca6c9996468c5b91aec53520bc781895168f949c93815ac501062411c153f",
      "blobs/sha256/f0d0405d31af85d016a44d855faf4fad2bcf23e3b31ebb3c313f4714c7ffeea7",
      "blobs/sha256/92b2ae71f144e493fd8c6172dfd184b430c316ddbda3a565cdfaa47fc667fbc0",
      "blobs/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
      "blobs/sha256/e22c76e803cb583614fe12bc6090b68f2cfdabc0700dca0a97627bbf4226c464",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/787c1f3510bd8ba56dd3944d217ed6f8a53c6eda5d55a11fbeee1c189dec9a3f",
      "blobs/sha256/dc62fc7eb1a822b0d70fc11a4cbef7a7c804c699374c6c312695cbffd6a995b9",
      "blobs/sha256/1dc94a70dbaa2171fb086500a5d27797f779219b126b0a1eebb9180c2792e80e"
    ]
  }
]

Using dive on this image works.

Motivations

It's possible that zero length layers are valid, however, it does appear to cause problems with certain tools and registries so it would be nice to know if the change was intentional.

[dmikusa]

Thanks for the detailed report!

Looking at 0.0.55 and 0.0.56, the only difference is a bump to the lifecycle. paketo-buildpacks/builder-noble-java-tiny@v0.0.55...v0.0.56

You could try to isolate this more by running pack build with --lifecycle-image buildpacksio/lifecycle:0.20.12. That will downgrade the lifecycle to the one that's included with 0.0.55, and since there are no other changes between those versions you should effectively have the same image (just created dynamically on-the-fly).

If downgrading fixes the issue, then this is something we'd need to report upstream to the CNB project.

For what it's worth, I tried your app and a typical maven app I use from our samples repo but wasn't able to reproduce this. I tried using the :latest builder as well as specifically 0.0.56 builder. Neither had the issue. I also tried using mvn spring-boot:build-image, but didn't see the issue.

My system: MacOS 15.6.1, Podman Desktop 1.21.0/Podman 5.6.0.

One thing that's interesting is that with my images, the first three layers of the app image are layers from the run image itself. The first being the layer with the base OS, the second adding the cnb user, and the third updating os-release. After that is where the buildpack generated layers start coming in. Makes me wonder if there's something weird going on with the run image, but idk what that would be. When you run with the builder version pinned to 0.0.56, it gives me the 0.0.25 run-image and that is working on my machine.

Can you run docker images and just check what it's pulled down? Check that you have the same digets.

> podman images | grep 'paketobuildpacks/ubuntu-noble-run'
docker.io/paketobuildpacks/ubuntu-noble-run-tiny               0.0.29          3e214c1045fb  8 days ago     32.8 MB
docker.io/paketobuildpacks/ubuntu-noble-run-tiny               0.0.25          f8f19345a385  3 weeks ago    32.8 MB

Maybe also try deleting your local run images & pull them back down.

I did note on the other ticket that this only seems to happen when publishing to the local docker daemon. It was indicated that using the --publish flag, the problem did not occur. That could point to an issue with Docker/Docker daemon or the code talking to it, because that takes a different code path, publishing to the daemon vs publishing to the registry. That could also be why I'm not seeing the issue, since I'm using Podman and a lot of folks are still using Docker Desktop. If it was an issue with the lifecycle itself, then I'd expect the problem to go away when you downgrade the lifecycle version (see command above).

In summary:

1. Are you using Docker Desktop? If so, what version? Has that version been updated recently?
2. Do you see the issue still when using newer builders, but downgrading the lifecycle to 0.20.12 (verison from last known working builder)?
3. Do your run images have the same digests as mine (above)?

[philwebb]

I've not yet had a chance to try the downgraded lifecycle, but to answer the other questions:

1. I'm using an M1 Mac with docker desktop (Docker version 28.3.2, build 578ccf6)
2. No, different digests:

docker images | grep 'paketobuildpacks/ubuntu-noble-run'
paketobuildpacks/ubuntu-noble-run-tiny           0.0.29           e355a14c9107   8 days ago      43.9MB
paketobuildpacks/ubuntu-noble-run-tiny           0.0.25           e67cf2d4a60a   3 weeks ago     43.9MB

[dmikusa]

Hmm, that's very interesting. I am not sure how it would have different digests there. The digests showing in my screenshot seem consistent with what is on the server.

> crane manifest paketobuildpacks/ubuntu-noble-run-tiny:0.0.29
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 748,
         "digest": "sha256:2ff811377e89613516e21e40d3c3d833194c38a5ee95b87a2b49a660f9599566",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 748,
         "digest": "sha256:1a5ce84e629a46033c2cd1d2e0f73449ff9ac686ee0d229eada2114842c68cda",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

Picking the arm64 image since we're both on M1's...

> crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:1a5ce84e629a46033c2cd1d2e0f73449ff9ac686ee0d229eada2114842c68cda | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1108,
    "digest": "sha256:3e214c1045fb2184b3af20d74042a7874e08eb84f95d2b43878fea7093203abb"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 8972014,
      "digest": "sha256:bbd2f12f185bcb0f5a94383dbde64ad3959d213bd1f9112cc65ef2fbbd5b3648"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 298,
      "digest": "sha256:8de9ac64a34a63e98947c8fdb64a10cec7c27a50b08867ba1fa24c02b405345c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 434,
      "digest": "sha256:938c0a9ed6a0694029ab8230905e8cffc4557959b472fc57835de4d33ad853ae"
    }
  ]
}

You can see the config digest matches what podman is showing in my screenshot: sha256:3e214c1045fb2184b3af20d74042a7874e08eb84f95d2b43878fea7093203abb.

and pulling that config out gives me this...

> crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:3e214c1045fb2184b3af20d74042a7874e08eb84f95d2b43878fea7093203abb | jq .
{
  "architecture": "arm64",
  "created": "2025-08-28T04:21:49.088615135Z",
  "history": [
    {
      "created": "2025-08-28T04:21:49.088615135Z",
      "created_by": "COPY /tiny/ / # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {                                                                                                                       "created": "0001-01-01T00:00:00Z"
    },
    {
      "created": "0001-01-01T00:00:00Z"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:9fa3fe91e4485e13ab0d4c9a09632b1de726a3347647a5e4181f6ebb72e676f7",
      "sha256:6ad1a069972005fdb1f19026aaf8538fcf25346225ca5a2032ba4da638a79100",
      "sha256:4dc4c8a4a130809caec7b08963873e310b03206570d0f91dfe0f721b055fed02"
    ]
  },
  "config": {                                                                                                             "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Labels": {
      "io.buildpacks.stack.description": "distroless-like noble",
      "io.buildpacks.stack.distro.name": "ubuntu",
      "io.buildpacks.stack.distro.version": "24.04",
      "io.buildpacks.stack.homepage": "https://github.com/paketo-buildpacks/noble-tiny-stack",
      "io.buildpacks.stack.id": "io.buildpacks.stacks.noble",
      "io.buildpacks.stack.maintainer": "Paketo Buildpacks",
      "io.buildpacks.stack.metadata": "{}",
      "io.buildpacks.stack.released": "2025-08-28T04:25:18Z"
    },
    "User": "1002:1000",
    "WorkingDir": "/"
  }
}

Can you try repeating those crane commands? See if you see the same thing remotely. I'd be pretty surprised if you don't, but it seems like a good baseline.

Then try running docker inspect e355a14c9107, which should give you the config for what you have locally. I'm really interested to see how that compares to what is remote. Also, see if you can get the "ID" value from that, you can then take the image ID and run crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:<img-id> and see if the ID you have remotely still exists. The image index might not reference it any more, but perhaps at once time it did? If that blob still exists, it might give more clues.

Totally wild guess, but I'm wondering if we had published an invalid image for some period of time and then it got updated and fixed?

[philwebb]

I've applied the latest Docker Desktop update [4.45.0 (203075)]

$ docker --version 
Docker version 28.3.3, build 980b856

$ docker system prune --all
$ pack build demo-image-issue-l --builder paketobuildpacks/builder-noble-java-tiny:0.0.60 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar

$ docker images | grep 'paketobuildpacks/ubuntu-noble-run'
paketobuildpacks/ubuntu-noble-run-tiny     0.0.29    e355a14c9107   8 days ago     43.9MB

$ crane manifest paketobuildpacks/ubuntu-noble-run-tiny:0.0.29 | jq .

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 748,
      "digest": "sha256:2ff811377e89613516e21e40d3c3d833194c38a5ee95b87a2b49a660f9599566",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 748,
      "digest": "sha256:1a5ce84e629a46033c2cd1d2e0f73449ff9ac686ee0d229eada2114842c68cda",
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

$ crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:1a5ce84e629a46033c2cd1d2e0f73449ff9ac686ee0d229eada2114842c68cda | jq .

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1108,
    "digest": "sha256:3e214c1045fb2184b3af20d74042a7874e08eb84f95d2b43878fea7093203abb"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 8972014,
      "digest": "sha256:bbd2f12f185bcb0f5a94383dbde64ad3959d213bd1f9112cc65ef2fbbd5b3648"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 298,
      "digest": "sha256:8de9ac64a34a63e98947c8fdb64a10cec7c27a50b08867ba1fa24c02b405345c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 434,
      "digest": "sha256:938c0a9ed6a0694029ab8230905e8cffc4557959b472fc57835de4d33ad853ae"
    }
  ]
}

Trying the other arch, just in case:

$ crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:2ff811377e89613516e21e40d3c3d833194c38a5ee95b87a2b49a660f9599566 | jq .

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1108,
    "digest": "sha256:153db738b09a8535faaf317c70465cb2295baefcf8158845254fd266050d2bb6"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 9680852,
      "digest": "sha256:c2769f0cc4876f9434bfba8b6f741b0c72b4c88cbc405419fa8cb80834eb151d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 303,
      "digest": "sha256:a5716e59c4bc0dbda564499e4ca691a9d4146d479bd4bebd6767e8134864c7cc"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 453,
      "digest": "sha256:44f4c4e7c6f41b7f3447abe65494b5103eff312c4b9b53e6d8e0402ff5b90d92"
    }
  ]
}

$ docker inspect e355a14c9107 | jq

[
  {
    "Id": "sha256:e355a14c910731fcad0ab3dd8615b6048b3c3d81ff10d8026eac35393058af48",
    "RepoTags": [
      "paketobuildpacks/ubuntu-noble-run-tiny:0.0.29"
    ],
    "RepoDigests": [
      "paketobuildpacks/ubuntu-noble-run-tiny@sha256:e355a14c910731fcad0ab3dd8615b6048b3c3d81ff10d8026eac35393058af48"
    ],
    "Parent": "",
    "Comment": "",
    "Created": "2025-08-28T04:21:49.088615135Z",
    "DockerVersion": "",
    "Author": "",
    "Architecture": "arm64",
    "Os": "linux",
    "Size": 8975325,
    "VirtualSize": 8975325,
    "GraphDriver": {
      "Data": null,
      "Name": "overlayfs"
    },
    "RootFS": {
      "Type": "layers",
      "Layers": [
        "sha256:9fa3fe91e4485e13ab0d4c9a09632b1de726a3347647a5e4181f6ebb72e676f7",
        "sha256:6ad1a069972005fdb1f19026aaf8538fcf25346225ca5a2032ba4da638a79100",
        "sha256:4dc4c8a4a130809caec7b08963873e310b03206570d0f91dfe0f721b055fed02"
      ]
    },
    "Metadata": {
      "LastTagTime": "2025-09-05T18:58:25.374508504Z"
    },
    "Config": {
      "AttachStderr": false,
      "AttachStdin": false,
      "AttachStdout": false,
      "Cmd": null,
      "Domainname": "",
      "Entrypoint": null,
      "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      ],
      "Hostname": "",
      "Image": "",
      "Labels": {
        "io.buildpacks.stack.description": "distroless-like noble",
        "io.buildpacks.stack.distro.name": "ubuntu",
        "io.buildpacks.stack.distro.version": "24.04",
        "io.buildpacks.stack.homepage": "https://github.com/paketo-buildpacks/noble-tiny-stack",
        "io.buildpacks.stack.id": "io.buildpacks.stacks.noble",
        "io.buildpacks.stack.maintainer": "Paketo Buildpacks",
        "io.buildpacks.stack.metadata": "{}",
        "io.buildpacks.stack.released": "2025-08-28T04:25:18Z"
      },
      "OnBuild": null,
      "OpenStdin": false,
      "StdinOnce": false,
      "Tty": false,
      "User": "1002:1000",
      "Volumes": null,
      "WorkingDir": "/"
    }
  }
]

$ crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:e355a14c910731fcad0ab3dd8615b6048b3c3d81ff10d8026eac35393058af48 | jq .

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 748,
         "digest": "sha256:2ff811377e89613516e21e40d3c3d833194c38a5ee95b87a2b49a660f9599566",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 748,
         "digest": "sha256:1a5ce84e629a46033c2cd1d2e0f73449ff9ac686ee0d229eada2114842c68cda",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

[philwebb]

Full pack output:

$ pack build demo-image-issue-l --builder paketobuildpacks/builder-noble-java-tiny:0.0.60 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar
Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded.
For more information, see https://github.com/buildpacks/pack/issues/2272.
0.0.60: Pulling from paketobuildpacks/builder-noble-java-tiny
30414fa31a83: Pull complete 
89732bc75041: Pull complete 
891bb4e1d0e5: Pull complete 
bab7275ca528: Pull complete 
fb4a3447f25a: Pull complete 
c1c89b3eeb92: Pull complete 
1c3dd7ac6a46: Pull complete 
d02484a7762d: Pull complete 
b981ffeba613: Pull complete 
d4816a461430: Pull complete 
f42198ab7e2c: Pull complete 
afc061aba914: Pull complete 
1b787c71f27f: Pull complete 
f88b0398bfaf: Pull complete 
d8bb7f89ba22: Pull complete 
61f19ec79d2a: Pull complete 
0bec5faa4682: Pull complete 
558b652b1517: Pull complete 
905d63e9595e: Pull complete 
be9d588eb786: Pull complete 
90052e314e91: Pull complete 
3da71729ec4f: Pull complete 
3986aaaa0d04: Pull complete 
4a94837ba8c4: Pull complete 
efec52ad8379: Pull complete 
9dd650a95521: Pull complete 
902d785f0447: Pull complete 
bbcdb25ea663: Pull complete 
3e6b54c7d575: Pull complete 
b3971e026370: Pull complete 
ea1d22415541: Pull complete 
45b7b4c41ffe: Pull complete 
2c584b6f97fd: Pull complete 
a34927795b65: Pull complete 
2015ea6964b0: Pull complete 
da45776cd4c2: Pull complete 
db5253a1f577: Pull complete 
96d1b73abb65: Pull complete 
fb3cc7e0411e: Pull complete 
Digest: sha256:c61c58cc747adcefb6948bf45aa7ae403200e121381efeda5b06388019744915
Status: Downloaded newer image for paketobuildpacks/builder-noble-java-tiny:0.0.60
0.0.29: Pulling from paketobuildpacks/ubuntu-noble-run-tiny
938c0a9ed6a0: Pull complete 
8de9ac64a34a: Pull complete 
bbd2f12f185b: Pull complete 
Digest: sha256:e355a14c910731fcad0ab3dd8615b6048b3c3d81ff10d8026eac35393058af48
Status: Downloaded newer image for paketobuildpacks/ubuntu-noble-run-tiny:0.0.29
0.20.13: Pulling from buildpacksio/lifecycle
4eff9a62d888: Pull complete 
7c12895b777b: Pull complete 
3214acf345c0: Pull complete 
bfb59b82a9b6: Pull complete 
5664b15f108b: Pull complete 
60d1f4385046: Pull complete 
62de241dac5f: Pull complete 
0bab15eea81d: Pull complete 
a62778643d56: Pull complete 
4aa0ea1413d3: Pull complete 
ddf74a63f7d8: Pull complete 
da88a3b79f2c: Pull complete 
da7816fa955e: Pull complete 
Digest: sha256:80c34d889d5db82bd58de6fe0d6b885fa479ff96fbbde20b0f80c3c2843d8913
Status: Downloaded newer image for buildpacksio/lifecycle:0.20.13
===> ANALYZING
[analyzer] Image with name "demo-image-issue-l" not found
===> DETECTING
[detector] target distro name/version labels not found, reading /etc/os-release file
[detector] 6 of 26 buildpacks participating
[detector] paketo-buildpacks/ca-certificates   3.10.4
[detector] paketo-buildpacks/bellsoft-liberica 11.2.7
[detector] paketo-buildpacks/syft              2.19.0
[detector] paketo-buildpacks/executable-jar    6.13.3
[detector] paketo-buildpacks/dist-zip          5.10.3
[detector] paketo-buildpacks/spring-boot       5.33.4
===> RESTORING
[restorer] Restoring metadata for "paketo-buildpacks/syft:syft" from cache
[restorer] Restoring data for "paketo-buildpacks/syft:syft" from cache
[restorer] Restoring data for SBOM from cache
===> BUILDING
[builder] target distro name/version labels not found, reading /etc/os-release file
[builder] 
[builder] Paketo Buildpack for CA Certificates 3.10.4
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Build Configuration:
[builder]     $BP_EMBED_CERTS                    false  Embed certificates into the image
[builder]     $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
[builder]     $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder] 
[builder] Paketo Buildpack for BellSoft Liberica 11.2.7
[builder]   https://github.com/paketo-buildpacks/bellsoft-liberica
[builder]   Build Configuration:
[builder]     $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
[builder]     $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
[builder]     $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
[builder]     $BP_JVM_VERSION              21                                                           the Java version
[builder]   Launch Configuration:
[builder]     $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
[builder]     $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
[builder]     $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
[builder]     $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
[builder]     $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
[builder]     $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
[builder]     $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
[builder]     $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
[builder]     $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
[builder]     $BPL_JMX_PORT                5000                                                         configure the JMX port
[builder]     $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
[builder]     $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
[builder]     $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
[builder]     $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
[builder]     Using Java version 21 from BP_JVM_VERSION
[builder]   BellSoft Liberica JRE 21.0.8: Contributing to layer
[builder]     Downloading from https://github.com/bell-sw/Liberica/releases/download/21.0.8+12/bellsoft-jre21.0.8+12-linux-aarch64.tar.gz
[builder]     Verifying checksum
[builder]     Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
[builder]     Adding 146 container CA certificates to JVM truststore
[builder]     Writing env.launch/BPI_APPLICATION_PATH.default
[builder]     Writing env.launch/BPI_JVM_CACERTS.default
[builder]     Writing env.launch/BPI_JVM_CLASS_COUNT.default
[builder]     Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
[builder]     Writing env.launch/JAVA_HOME.default
[builder]     Writing env.launch/JAVA_TOOL_OPTIONS.append
[builder]     Writing env.launch/JAVA_TOOL_OPTIONS.delim
[builder]     Writing env.launch/MALLOC_ARENA_MAX.default
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/java-opts
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/jvm-heap
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/link-local-dns
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-configurer
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/jmx
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/jfr
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-classpath-9
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/debug-9
[builder]     Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/nmt
[builder]   Java Security Properties: Contributing to layer
[builder]     Writing env.launch/JAVA_SECURITY_PROPERTIES.default
[builder]     Writing env.launch/JAVA_TOOL_OPTIONS.append
[builder]     Writing env.launch/JAVA_TOOL_OPTIONS.delim
[builder] 
[builder] Paketo Buildpack for Syft 2.19.0
[builder]   https://github.com/paketo-buildpacks/syft
[builder]     Downloading from https://github.com/anchore/syft/releases/download/v1.31.0/syft_1.31.0_linux_arm64.tar.gz
[builder]     Verifying checksum
[builder]     Writing env.build/SYFT_CHECK_FOR_APP_UPDATE.default
[builder] 
[builder] Paketo Buildpack for Executable JAR 6.13.3
[builder]   https://github.com/paketo-buildpacks/executable-jar
[builder]   Class Path: Contributing to layer
[builder]     Writing env/CLASSPATH.delim
[builder]     Writing env/CLASSPATH.prepend
[builder]   Process types:
[builder]     executable-jar: java org.springframework.boot.loader.launch.JarLauncher (direct)
[builder]     task:           java org.springframework.boot.loader.launch.JarLauncher (direct)
[builder]     web:            java org.springframework.boot.loader.launch.JarLauncher (direct)
[builder] 
[builder] Paketo Buildpack for Spring Boot 5.33.4
[builder]   https://github.com/paketo-buildpacks/spring-boot
[builder]   Build Configuration:
[builder]     $BPL_JVM_CDS_ENABLED                 false  whether to enable CDS optimizations at runtime
[builder]     $BPL_SPRING_AOT_ENABLED              false  whether to enable Spring AOT at runtime
[builder]     $BP_JVM_CDS_ENABLED                  false  whether to enable CDS & perform JVM training run
[builder]     $BP_SPRING_AOT_ENABLED               false  whether to enable Spring AOT
[builder]     $BP_SPRING_CLOUD_BINDINGS_DISABLED   false  whether to contribute Spring Boot cloud bindings support
[builder]     $BP_SPRING_CLOUD_BINDINGS_VERSION    1      default version of Spring Cloud Bindings library to contribute
[builder]   Launch Configuration:
[builder]     $BPL_SPRING_CLOUD_BINDINGS_DISABLED  false  whether to auto-configure Spring Boot environment properties from bindings
[builder]     $BPL_SPRING_CLOUD_BINDINGS_ENABLED   true   Deprecated - whether to auto-configure Spring Boot environment properties from bindings
[builder]   Creating slices from layers index
[builder]     dependencies (19.8 MB)
[builder]     spring-boot-loader (458.0 KB)
[builder]     snapshot-dependencies (0.0 B)
[builder]     application (41.1 KB)
[builder]   Spring Cloud Bindings 2.0.4: Contributing to layer
[builder]     Downloading from https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-bindings/2.0.4/spring-cloud-bindings-2.0.4.jar
[builder]     Verifying checksum
[builder]     Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
[builder]   Web Application Type: Contributing to layer
[builder]     Servlet web application detected
[builder]     Writing env.launch/BPL_JVM_THREAD_COUNT.default
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
[builder]   4 application slices
[builder]   Image labels:
[builder]     org.opencontainers.image.title
[builder]     org.opencontainers.image.version
[builder]     org.springframework.boot.version
===> EXPORTING
[exporter] Adding layer 'paketo-buildpacks/ca-certificates:helper'
[exporter] Adding layer 'paketo-buildpacks/bellsoft-liberica:helper'
[exporter] Adding layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
[exporter] Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'
[exporter] Adding layer 'paketo-buildpacks/executable-jar:classpath'
[exporter] Adding layer 'paketo-buildpacks/spring-boot:helper'
[exporter] Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
[exporter] Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
[exporter] Adding layer 'buildpacksio/lifecycle:launch.sbom'
[exporter] Added 5/5 app layer(s)
[exporter] Adding layer 'buildpacksio/lifecycle:launcher'
[exporter] Adding layer 'buildpacksio/lifecycle:config'
[exporter] Adding layer 'buildpacksio/lifecycle:process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Adding label 'org.opencontainers.image.title'
[exporter] Adding label 'org.opencontainers.image.version'
[exporter] Adding label 'org.springframework.boot.version'
[exporter] Setting default process type 'web'
[exporter] Saving demo-image-issue-l...
[exporter] *** Images (327d5495bef6):
[exporter]       demo-image-issue-l
[exporter] Adding cache layer 'paketo-buildpacks/syft:syft'
[exporter] Reusing cache layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
[exporter] Adding cache layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
[exporter] Adding cache layer 'buildpacksio/lifecycle:cache.sbom'

[dmikusa]

Hmm, ok. So it looks like docker might be just showing something different in the output from the images command. It looks like it is showing the digest of the index, while podman is showing the digest of the actual image config.

crane manifest paketobuildpacks/ubuntu-noble-run-tiny:0.0.29 is equal to crane blob paketobuildpacks/ubuntu-noble-run-tiny@sha256:e355a14c910731fcad0ab3dd8615b6048b3c3d81ff10d8026eac35393058af48, (that's the ID you're seeing with Docker) and both are pointing to the image index.

That's not a problem, the same images are being used it's just a UI difference.

[dmikusa]

Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded. For more information, see buildpacks/pack#2272.

Please try turning off containd storage and see if the problem goes away. I think docker desktop is enabling this by default now, and it's been the source of a number of issues. Hence the warning :)

[philwebb]

I've turned off containd storage a restarted Docker Desktop.

$ pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:0.0.60 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar 
$ docker image save demo-image-issue | tar -xf - manifest.json && cat manifest.json | jq  

[
  {
    "Config": "blobs/sha256/0a05309c5285563eb3091f85153f5347c021a4b0d91ae9ec39583d0aa3be6014",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/3e74b876fa888a7265efcbe547e07018ccda0421797d9ae6cbf185d6d1d5508c",
      "blobs/sha256/e13f32b89503082d9d076df31b0ab4fe367f17308145e2e9777332f52351305f",
      "blobs/sha256/5f4c1d5773eb3a00ebec6070263004ec60faf7311ea4db1ac5ff045aecd165f8",
      "blobs/sha256/a4124ebfe5c4a63a1f5020c530f176f0fb62c2edc2e0c87220cb0bba6d159b74",
      "blobs/sha256/b8578b8a2ac51f7abdfdd4eea28bc34a73920771ba01df61f9f6d30f2bbc0dfe",
      "blobs/sha256/d6ecce2bcb621e3b99b3fadd213faf56a32a4d221855c6726d317a931b97401e",
      "blobs/sha256/2cdf5b962e8ece0e546450ea779424e3420db80186dfafc0a0f44d172cee1c2f",
      "blobs/sha256/bc7f82c871ffa12af89cd62c3047e5289ce24fbbe07f7adeeb840ccd43140f20",
      "blobs/sha256/8e326708c473538285f07c789947ca89fa19fbbd7ae08333ec883c0fcba4d7cc",
      "blobs/sha256/cc3c0e569c3cd1ea2ca983256c6dc1c4ef10fe3c381b1fa0c450925c257dde94",
      "blobs/sha256/24d5d2eaea1e2fc53ad9e081541acb105a2ae62a6dcffb875dec06b573149c0a",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/891c5ef4572e1828d36c43534cacffd20bda39314b768e995bc327f475e2be96",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/34061cc4ad826866fb162f006f0a8e7b98cf4c6d4ae8644335d5fce00c345027",
      "blobs/sha256/f7f72ce47a0b7ddbe653dbe7ae8304e39f8ac3f29a8f16cd0623223373b4936e",
      "blobs/sha256/b39b18afe74ec7ee652aed10c5c3e637af7e7daad3da9380043a93fe7d212d44"
    ]
  }
]

So it doesn't look like that's the issue.

[philwebb]

The lifecycle downgrade does appear to fix things:

$ pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:0.0.60 --lifecycle-image buildpacksio/lifecycle:0.20.12 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar
$ docker image save demo-image-issue | tar -xf - manifest.json && cat manifest.json | jq

[
  {
    "Config": "blobs/sha256/f3551a212201af597f077137323d8fefbe9595c787ef323d077f0e73f03b5f8a",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/bbd2f12f185bcb0f5a94383dbde64ad3959d213bd1f9112cc65ef2fbbd5b3648",
      "blobs/sha256/8de9ac64a34a63e98947c8fdb64a10cec7c27a50b08867ba1fa24c02b405345c",
      "blobs/sha256/938c0a9ed6a0694029ab8230905e8cffc4557959b472fc57835de4d33ad853ae",
      "blobs/sha256/3c84bd1fca60a6d4642f1214971274ef79e4ffb2b715ba631eaa02c136d3f926",
      "blobs/sha256/6c4babdf3d5a65ff1fa0219a62319b54482b13be036b26c497cfbf1caf522224",
      "blobs/sha256/bea0a3dc2651cac7c9c567a5cb4e7536107b357cb9113e8806f690f050500012",
      "blobs/sha256/03160ce98d2ab327ffbb1a513d2d758f57214b22ffafc035151801da7e84ca5c",
      "blobs/sha256/417e5bfc3c82b9373cf6804206e071d2fc74560df867d0f39cb21ac3d15231b6",
      "blobs/sha256/7b886ae4be4791951200b37169b969f22eb9c3c59bd1bbaf1f761bbb0aa0e83b",
      "blobs/sha256/f0e9078fd509b2493bcef1215b4de825d53f118ef0648b850edec03dc6e3cfd4",
      "blobs/sha256/366ce7d1a7f90f2e4ad08752f87510eee3ffca18736fa63c03823c8c4ebf2925",
      "blobs/sha256/c56661e2d98239670a47dc9a51ec3686cef68bc6bb26b1a7be96016169c89e13",
      "blobs/sha256/f0d0405d31af85d016a44d855faf4fad2bcf23e3b31ebb3c313f4714c7ffeea7",
      "blobs/sha256/92b2ae71f144e493fd8c6172dfd184b430c316ddbda3a565cdfaa47fc667fbc0",
      "blobs/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
      "blobs/sha256/e22c76e803cb583614fe12bc6090b68f2cfdabc0700dca0a97627bbf4226c464",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/787c1f3510bd8ba56dd3944d217ed6f8a53c6eda5d55a11fbeee1c189dec9a3f",
      "blobs/sha256/53bba8729bd616f35d78c72839977ddf1bff11f995e7fb2e2a39913b91806662",
      "blobs/sha256/1dc94a70dbaa2171fb086500a5d27797f779219b126b0a1eebb9180c2792e80e"
    ]
  }
]

[dmikusa]

OK, thanks @philwebb Let me ask one of the CNB folks and see if they have any thoughts.

[dmikusa]

@philwebb - OK, so there have been some lifecycle changes in .13 which might be causing this issue. There has been some more work done in this part of the code & in particular a race condition that was fixed.

Can you try a build with the following lifecycle commit and see if the problem still exists? buildpacksio/lifecycle:c9c7bf5

pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:0.0.60 --lifecycle-image buildpacksio/lifecycle:c9c7bf5 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar

Thanks!

[philwebb]

No luck I'm afraid. Same result:

[
  {
    "Config": "blobs/sha256/ef6f8b992471e7876198260c45d521361b36826f93a50aec9c839f72f3c8b9f6",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/3e74b876fa888a7265efcbe547e07018ccda0421797d9ae6cbf185d6d1d5508c",
      "blobs/sha256/e13f32b89503082d9d076df31b0ab4fe367f17308145e2e9777332f52351305f",
      "blobs/sha256/5f4c1d5773eb3a00ebec6070263004ec60faf7311ea4db1ac5ff045aecd165f8",
      "blobs/sha256/a4124ebfe5c4a63a1f5020c530f176f0fb62c2edc2e0c87220cb0bba6d159b74",
      "blobs/sha256/b8578b8a2ac51f7abdfdd4eea28bc34a73920771ba01df61f9f6d30f2bbc0dfe",
      "blobs/sha256/d6ecce2bcb621e3b99b3fadd213faf56a32a4d221855c6726d317a931b97401e",
      "blobs/sha256/2cdf5b962e8ece0e546450ea779424e3420db80186dfafc0a0f44d172cee1c2f",
      "blobs/sha256/bc7f82c871ffa12af89cd62c3047e5289ce24fbbe07f7adeeb840ccd43140f20",
      "blobs/sha256/41ecf2da6eb2d33ddaf8a9914064af4055c1147989b055aa771d3cc67239f359",
      "blobs/sha256/cc3c0e569c3cd1ea2ca983256c6dc1c4ef10fe3c381b1fa0c450925c257dde94",
      "blobs/sha256/24d5d2eaea1e2fc53ad9e081541acb105a2ae62a6dcffb875dec06b573149c0a",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/891c5ef4572e1828d36c43534cacffd20bda39314b768e995bc327f475e2be96",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/d8ab759432ea7d160c78e38be82eaadbed1749dd0baa732d4ab1b32a474e03b5",
      "blobs/sha256/f7f72ce47a0b7ddbe653dbe7ae8304e39f8ac3f29a8f16cd0623223373b4936e",
      "blobs/sha256/b39b18afe74ec7ee652aed10c5c3e637af7e7daad3da9380043a93fe7d212d44"
    ]
  }
]

[dmikusa]

Thanks