[Feedback Requested] Updated Posit container images and build tooling

[bschwedler]

We are in the process of updating how Posit's public container images are defined and built.

We are currently in internal beta stage with the image builds; we are working with product engineering teams and our internal platforms to identify any major issues with the images and tooling before releasing them into the wild.

High level-goals of this refactor:

• Produce static image definitions that are easier to use and extend for our internal and external stakeholders
• Establish conventions and tooling to reduce maintenance overhead for teams and stakeholders
• Rapidly address security concerns by supporting scanning and rebuilds of previous product images
• Eliminate image dependencies between different products
• Re-brand the images to Posit

---

@bschwedler is the image build refactor being done in the open or behind a private repo?

My workplace runs both connect and workbench and I'm currently reviewing our customised image build processes so any sneak previews you can provide, or timeframes would be appreciated

Originally posted by @harrisony in #506

---

Here is a sneak peek behind the curtain:

• A open-source python CLI (and GitHub Actions) that bind together different tools in an opinionated manner, including
  • Containerfile/Dockerfile linting (hadolint)
  • Different build backends (e.g. docker buildx bake, podman)
  • Artifact testing (dgoss)
  • Container security scanning (e.g. trivy, openscap and other security scanners)
• A single configuration file (manged & integrated with the CLI) can define
  • Multiple images
  • Multiple versions for each image
  • Multiple variants of the images (e.g. minimal vs large)
  • Multiple combinations of dependencies (e.g. R, python)
  • Multiple OSes
  • Multiple platform support (e.g. ARM)
  • Custom, dynamic image tags
  • Multiple image registries

We hope to have something available for preview by the end of the year. 🤞🏻

Our small team's work on this was preempted by other priorities earlier in the year, but we intend to remain focused on this work through at least the end of this year.

[fh-mthomson]

Excited to learn more!

We hope to have something available for preview by the end of the year.

In the meantime, is it fair to assume the current images (this repo) will continue to receive routine app upgrades of Workbench, PM, Connect etc?

[bschwedler]

In the meantime, is it fair to assume the current images (this repo) will continue to receive routine app upgrades of Workbench, PM, Connect etc?

That is correct!
After the updated images and build process are generally available, we will still provide product updates in this repository for a period of time.

[harrisony]

Thanks for sharing—this sounds awesome!

Our setup uses Workbench and Connect with the Kubernetes executor, and it's completely air-gapped from the internet. The CI/CD environment where we build our containers is a bit more flexible, but we still have to pull base images from our internal Artifactory mirror. Thankfully, most build tools support mirror configuration, so that part’s pretty straightforward.

To deal with the air-gap, we pre-install VS Code extensions (plus a few extras requested by users) during the container build process while we still have internet access. We also throw in some system libraries and the AWS CLI tools.

Because of the way our CI/CD environment is configured, I’m limited to using rootless build backends, which unfortunately rules out the docker buildx bake approach used in this repo. At the moment, I’ve set up a GitLab pipeline that queries the latest Connect content image tags, then kicks off a series of builds using buildkit-rootless and generates SBOMs using Trivy. While I'm quite proud of the setup and it's quite stable, it’s fairly complex and be more than happy to retire it

A rootless build backend, ability to add some additional RUN steps and configurable registry would make me a happy camper

[bschwedler]

@harrisony Thank you for your input!

Internally, we are already doing many of these steps with other images, especially the VS Code extensions and AWS CLI. I've added these to [currently internal, public soon-ish] issues so we can track them.

I had a few questions so that I can ensure that I understand your needs and current process

1. What rootless builders did you consider in addition to rootless buildkit?
   We have been thinking about possibly supporting buildah and/or nerdctl (containerd) via python-on-whales

2. Are you installing extensions from Open VSX directly, or do you download the extension file to install it?

3. Would you want to be able to specify registry/repo for the base image at build-time, or would having this as configuration or in a FROM statement work for you?

4. Are you using any other capabilities from Trivy (e.g. scanning for vulnerabilities/secrets)?

[harrisony]

@bschwedler 🙂

1. I probably should have been a little bit more specific in saying we can't use a solution involving a privileged container or passing the host socket to the builder (docker in docker).
   Kaniko was previously used before Google retired it, rootless buildkit (via buildctl-daemonless.sh) was an I don't have the time to properly research this and want to copy and paste something quickly decision. The GitLab docs had examples for rootless buildkit and buildah and buildkit was higher on the list. I suspect buildah would have worked equally as well.

2. Directly from Open VSX.
   When we build r-session-complete, we copy across our own vscode.extensions.conf and then run rstudio-server configure-vs-code --extensions-dir=/opt/code-server/extensions while building the container.

3. Don't mind, either is fine

4. We run a vulnerabilities and secrets scan but don't do anything with them currently.

[jprecursion]

@bschwedler 🙂

1. I probably should have been a little bit more specific in saying we can't use a solution involving a privileged container or passing the host socket to the builder (docker in docker).
   Kaniko was previously used before Google retired it, rootless buildkit (via buildctl-daemonless.sh) was an I don't have the time to properly research this and want to copy and paste something quickly decision. The GitLab docs had examples for rootless buildkit and buildah and buildkit was higher on the list. I suspect buildah would have worked equally as well.
2. Directly from Open VSX.
   When we build r-session-complete, we copy across our own vscode.extensions.conf and then run rstudio-server configure-vs-code --extensions-dir=/opt/code-server/extensions while building the container.
3. Don't mind, either is fine
4. We run a vulnerabilities and secrets scan but don't do anything with them currently.

Curious (coming in late to the conversation), was podman considered? I am using it over docker everywhere that I can, and that is rootless by default.

[bschwedler]

@jprecursion

While our CI currently only runs against docker buildx, we have manually verified that the images can be built using buildah and podman.

Under the hood we are using python-on-whales so we may also be able to support nerdctl (containerd).