🐞 VM fails to start on mac2.metal EC2 (Sequoia & Tahoe 26 images) — headless setup

[camilacodes]

Hi! I'm Camila!

I'm running GitLab Runner with Tart executor on a headless macOS EC2 instance (mac2.metal) hosted on a dedicated Mac host. When attempting to run jobs using newer macOS images (e.g. Sequoia and Tahoe 26), the VM fails to boot with the following error:

2025/10/03 01:28:51 Cloning and configuring a new VM... 2025/10/03 01:28:51 Waiting for the VM to boot and be SSH-able... Error Domain=VZErrorDomain Code=1 "The virtual machine failed to start." UserInfo={NSLocalizedFailure=Internal Virtualization error., NSLocalizedFailureReason=The virtual machine failed to start.} Failed to retrieve IP address of VM "gitlab-4447377" in 60 seconds: tart command returned non-zero exit code: "no IP address found, is your VM running?", will re-try...

This only happens with newer macOS images (Sequoia, Tahoe 26). With sonoma-xcode latest works fine.

I suspect it's related to local network access permissions, which require GUI confirmation.

However, my environment is headless, so I cannot interact with the GUI to approve the virtualization prompt.

Environment

ProductName:    macOS
ProductVersion: 15.6.1
BuildVersion:   24G90

tart --version
2.28.6

gitlab-runner --version
Version:      18.4.0
Git revision: 139a0ac0
Git branch:   18-4-stable
GO version:   go1.25.1
Built:        2025-09-12T20:51:41+00:00
OS/Arch:      darwin/arm64

What I've Tried

• Running GitLab Runner as a LaunchAgent (user context).
• Running GitLab Runner as a LaunchDaemon (system context).
• Enabling autologin for ec2-user and confirming session is active.
• Using launchctl asuser to invoke Tart inside a session.
• Validating permissions, TART_HOME, and sudoers configuration.

Despite all of this, the VM still fails to start when using newer macOS images.

Here’s a simplified version of my user-data script: LaunchAgent mode

#!/bin/bash
exec > /var/log/user-data.log 2>&1
set -euo pipefail
set -x

RUNNER_USER="ec2-user"
RUNNER_HOME="$(eval echo "~$RUNNER_USER")"

# Homebrew + packages
su - "$RUNNER_USER" -c 'if ! command -v brew >/dev/null 2>&1; then
  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi'
su - "$RUNNER_USER" -c 'brew install tart gitlab-tart-executor gitlab-runner awscli jq tmux'

# GitLab Runner registration
AUTH_TOKEN="retrieved-from-ssm"
su - "$RUNNER_USER" -c "gitlab-runner register \
  --non-interactive \
  --url https://gitlab.example.com/ \
  --token \"$AUTH_TOKEN\" \
  --executor custom \
  --custom-prepare-exec gitlab-tart-executor \
  --custom-prepare-args prepare \
  --custom-run-exec gitlab-tart-executor \
  --custom-run-args run \
  --custom-cleanup-exec gitlab-tart-executor \
  --custom-cleanup-args cleanup \
  --builds-dir \"$RUNNER_HOME/builds\" \
  --cache-dir \"$RUNNER_HOME/cache\""

sudo /usr/bin/env HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_FROM_API=1 \
  /opt/homebrew/bin/brew services restart gitlab-runner \
  || sudo /usr/bin/env HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_FROM_API=1 \
       /opt/homebrew/bin/brew services start gitlab-runner

Is there a way to pre-authorize or bypass the GUI prompt for local network access in headless environments?

Alternatively, is there a recommended workaround for running Sequoia/Tahoe images on EC2 macOS without GUI interaction?

Updates: Inside de UI of the EC2 I was able to clone and run the latest image of sequoia, I must be missing something in my user data configuration but I don't know.

some logs:

​Sandbox: com.apple.Virtualization.Virtual(5304) deny(1) iokit-get-properties iokit-class:AppleAVD property:IOGVAHEVCDecode
tccd: AUTHREQ_ATTRIBUTION ... responsible_path=/opt/homebrew/Cellar/gitlab-runner/18.4.0/bin/gitlab-runner
/Library/Logs/DiagnosticReports/com.apple.Virtualization.VirtualMachine-2025-10-03-184431.ips

Thanks in advance!

[edigaryev]

Does tart clone ghcr.io/cirruslabs/macos-sequoia-base:latest macos && tart run macos work if you execute it via SSH on that GitLab Runner machine?

Also, we've just added a new FAQ entry that specifically addresses issues on headless machines.

[camilacodes]

Yes, tart clone ghcr.io/cirruslabs/macos-sequoia-base:latest macos && tart run macos works fine. The issue is with the new Sequoia xcode images and Tahoe — I need to log in through the UI for them to work. This makes auto-scaling difficult because I can’t log in every time a new machine is created, unless I create a new AMI based on one where I’ve already completed the GUI session, but we have plans to use tart AMI. I’ll try the keychain solution today. Thanks for the support!

[camilacodes]

Does tart clone ghcr.io/cirruslabs/macos-sequoia-base:latest macos && tart run macos work if you execute it via SSH on that GitLab Runner machine?

Also, we've just added a new FAQ entry that specifically addresses issues on headless machines.

Hi!
I tested the solution for headless EC2 environments documented at https://tart.run/faq/#headless-machines, but unfortunately it didn’t work for me.

But when using the latest Sequoia-Xcode:26 and Tahoe-Xcode:26 images, my GitLab jobs were unable to access the VMs. The only way I could get them to work was by manually restarting the instance and approving the Local Network Access prompt for gitlab-runner via the GUI — even though the runner was launched as a daemon.

This manual approval step breaks automation and makes it impossible to scale runners dynamically in a headless setup.

Is there a known workaround or something I might be missing in my configuration?

Is there any planned fix or improvement for this behavior in future versions of Tart or macOS?