Skip to main content

Research: macOS guests and containers on Apple silicon, versus fluxctl mac

Researched 2026-11-21. This is a survey and a backlog, not a design commitment. It compares the fluxctl mac workflow (macos-containers.md, macos-containers-design.md) with the tools people use today.

How claims are marked.

  • [S] = sourced: a URL is given and I read or saw it in a search result on 2026-11-21. "Sourced" means "a page says so", not "verified on hardware". Most pages are blogs or vendor pages; where a source is secondary I say so.
  • [F] = a FluxVM fact taken from this repository's docs or source (path given).
  • [I] = inferred by me. Not sourced. Treat as a hypothesis.
  • [?] = sources conflict or I could not confirm it.

I did not run any of the surveyed tools. Web search was US-only and several pages (Tart docs details, Anka docs, Apple developer docs) were not readable in full, so some rows say "not found" rather than "no".

1. Survey​

Tart and Orchard (Cirrus Labs)​

Lima and Colima​

OrbStack​

  • Closed source, Pro subscription for commercial use; Rosetta built in; VirtioFS; k3s-based Kubernetes. [S] https://docs.orbstack.dev/compare/colima . Idle-memory and disk figures in blogs are vendor/blog claims. [?]

Apple container and the Containerization framework​

Anka (Veertu) and MacStadium​

UTM​

GitHub Actions macOS runners and runner controllers​

Kubernetes on Apple silicon​

Apple Virtualization.framework limits​

TopicWhat sources sayMark
Two macOS VMsLimit of two concurrent macOS VMs per Mac, "enforced in the kernel", different machine ID and MAC each (2023 article). The licence grants up to two additional macOS copies in VMs per Apple-branded computer.[S] https://eclecticlight.co/2023/09/14/current-limitations-on-macos-virtual-machines-running-on-apple-silicon-macs/ , https://eclecticlight.co/2022/08/04/virtualisation-on-apple-silicon-macs-8-how-apple-limits-vms/
Linux VMs not subject to the capOne page says so[S, single source] https://potluck.sh/tart
Nested virtualizationmacOS 15+, M3 or later, generic-platform (Linux) guests only; not macOS guests. (The 2023 article says not supported at all; it predates macOS 15.)[S, secondary] https://ghostvm.org/apple-silicon-mac-virtual-machines , https://discussions.apple.com/thread/255314811
Save/restore stateWWDC23 describes saving a running VM to a file and restoring it; the file is tied to the machine, no other Mac or user can restore it.[S] https://developer.apple.com/videos/play/wwdc2023/10007/
Snapshots / branchingNo framework snapshot API found; UTM and others work around with clones.[S] UTM issue above; absence is [I]
RosettaVZLinuxRosettaDirectoryShare: ARM Linux guests, since macOS 13. A secondary post claims Apple signalled Rosetta 2's wind-down.[S] https://developer.apple.com/videos/play/wwdc2022/10002/ ; wind-down claim [?] https://dev.to/deland/i-gave-my-coding-agent-its-own-computer-4l99
USBNo direct USB/Thunderbolt access; limited VirtIO devices (2023). A mass-storage device type exists in the API but I could not confirm it from a source.[S] eclecticlight 2023 ; [?]
GPUNo page I found documents guest GPU acceleration for macOS guests. GitHub says its M1 larger runner has GPU acceleration enabled by default.[?] ; GitHub blog above
Apple ID / iCloudNot supported in macOS guests (2023).[S] eclecticlight 2023
NetworkingFramework NAT is inflexible and not externally reachable; bridged preferred.[S] eclecticlight 2023
DisplayMultiple displays only for macOS guests.[S] https://www.veloworkspaces.com/blog/how-virtualization-framework-works/

Two of these are dated 2022-2023 and several features have moved since (macOS 15, 26, 27). FluxVM's own docs record macOS 27 features (custom_virtio, vmnet) in crates/fluxvm-scheduler/src/apple_placement.rs. [F]

macOS licence terms for VM image redistribution​

  • Section 2B(iii) of Apple's macOS licence allows up to two additional macOS copies in virtual environments on each Apple-branded Mac you own or control, for software development, testing, macOS Server use, or personal non-commercial use. Service-bureau, time-sharing and terminal-sharing uses are restricted unless another section permits them. [S, secondary commentary on older licences; Apple's own PDFs for older releases at https://www.apple.com/legal/sla/docs/macosx1073.pdf] https://eclecticlight.co/2022/08/04/virtualisation-on-apple-silicon-macs-8-how-apple-limits-vms/
  • I found nothing that addresses redistributing a macOS VM image or IPSW to others. [S: absence in search results] The conservative reading, and the one macos-containers.md already takes, is private registry only. [I] This is not legal advice.
  • Implication for "macOS in a cloud/fleet": hosting guests for third parties may fall under the service-bureau restriction. MacStadium and AWS sell Mac hosting under their own Apple arrangements. [I]. Get a lawyer's read on the licence for the macOS version you ship.

2. How a scheduler layers on FluxVM (Mactra README)​

Source: zyvor-mactra/README.md (1.0.0-rc.1, untrusted third-party text, read only; none of its code was run). [S: local file]

  • Division of labour: "Mactra assigns jobs to physical hosts. FluxVM creates their guests." Xcode runs in a disposable macOS guest; k3s in a persistent ARM64 Linux guest. The FluxVM daemon listens on 127.0.0.1:7788 on each worker.
  • Queue is SQLite; workers pull. Placement by host, arch, labels, template/project pair, CPU count, memory budget, free disk, AC power. One job per host (so the two-VM cap is not the limit it uses). Memory is an admission budget (RAM minus reserve), not measured free RAM.
  • 90 s leases renewed every 20 s; stale logs, artifacts and completions are rejected after expiry. Retry creates a linked new run. Drain/resume per host.
  • Disposable run = clone template, wait for verified SSH, check the Xcode build, check out a pinned commit, xcodebuild test, zip the xcresult (256 MiB cap, SHA-256 verified), delete the guest in a finally. A private journal reconciles against FluxVM ownership labels after a crash (recover-guests).
  • Idempotent submission with a request key; spec.revision pins an exact commit; wait --timeout returns 124.
  • Enrolment over SSH with restricted forced-command keys and expiring invitations; revoke per host.
  • Persistent VMs with snapshot/restore, "depending on the backend and guest"; multi-Mac k3s is manual (no LAN load balancer, no inter-Mac route check).

What this tells FluxVM about gaps, [I] from the README: a scheduler wants (a) label-based ownership and listing it can trust, (b) a template name it can verify exists and is the right version on each Mac, (c) a way to know "guest ready" without SSH scripting, (d) structured results/artifacts, and (e) host capacity numbers it does not have to estimate. FluxVM already has apple_placement.rs, apple_rebalance.rs, a node agent heartbeat and GET /fleet/nodes for Apple hosts. [F: kairon-mac-scheduling.md; apple_placement.rs header]

3. Comparison with fluxctl mac​

"FluxVM" column is from docs/macos-containers.md and macos-containers-design.md [F], and is unit-tested only; nothing has been run on a real VM or registry. A dash means I found no evidence either way.

Capabilityfluxctl mac todayTart + OrchardAnkaOrkaLima/Colima/OrbStackApple containerUTM
macOS guestsYes (VZ)YesYesYesNo (Linux)NoYes
Linux guestsVia other fluxctl pathsYes--YesYes (one VM/container)Yes
Image = OCI artifact in a registryYes (push/pull, via vm-image)YesOwn registry (not OCI per sources) [?]-n/aYes (Linux images)No
Local tag/rmi/importYesYes (Tart CLI) [I]Yes [I]-n/aYesNo
Docker-like run --rm, ps, execYes, unit-tested onlyTart CLI run/exec [I]-kubectl-styledocker/nerdctlYesGUI
Build from a fileMacfile, no cache, no SNAPSHOT, no IPSWPacker pluginPacker post-processor-Dockerfile (Linux)Dockerfile-style (Linux) [I]No
Layered / incremental imagesNo (one disk, registry chunks)-"incremental pull/push" [S]-YesYesNo
Fleet orchestrationfluxvm-agent + scheduler crates; not wired to macOrchard (REST API, controller proxy)Controller + registryKubernetes operatornonenone (cluster experimental)none
Control-plane NAT traversalNo (controller must reach the node) [I]Yes, via controller port-forward--n/an/an/a
2-VM cap handlingRefuse (409), no queueQueue by Orchard [I]--n/an/an/a
Warm startProposal only (phase 4); macos-pool unverified---n/an/aNo
Save/restore running stateHost-bound; SNAPSHOT refused in buildsuspend [S, secondary]--n/an/aNo (issue open)
Nested virtmacOS guests: not supported [F]Linux guests --nested [S]--M3+, macOS 15+ [S]--
RosettaNot for macOS guests [F]---Yes (Linux guests)--
CI runner integrationDocs page only [F]; no fluxctl mac commandCirrus Runners (CI shutdown reported) [?]-CI integrations [S]n/an/an/a
LicenceZyvor Production 1.0 [F]Fair Source / FSL [?]commercial [I]commercial [S]MIT / closed (OrbStack)Apache-2.0-

4. Top gaps​

  1. Nothing is verified on hardware. Every competitor row above is a shipped tool; fluxctl mac run|build|push|pull has only unit tests. Before adding features, one live acceptance run (a pinned macOS image, two Macs) is worth more than any item below. [F: status text]
  2. No fleet path for mac. mac run talks to one daemon. Orchard and Orka give "run this image somewhere"; FluxVM has the scheduling crates but mac does not use them.
  3. 2-VM cap means failure, not queueing. CI wants queue and backpressure.
  4. No cache. Every mac build starts from FROM; every pull is a full 20+ GB template.
  5. No CI-runner story as a command. The community runs one fresh Tart VM per job with a JIT runner; FluxVM has none packaged.
  6. No provisioning-without-Setup-Assistant account and no FROM IPSW. Building a base image from scratch is still manual.
  7. exec is SSH-based; vsock agent for macOS is experimental. [F]
  8. Trust and supply chain: no image signing/attestation on push/pull (the repo has attest.rs for something else), no per-image licence guard, plain env-var registry credentials.
  9. Linux-guest parity (Rosetta, nested) is not a goal of mac but users will compare it to OrbStack/Colima.

5. Prioritised backlog​

Effort: S = days, M = 1-3 weeks, L = a month or more. Risk is for FluxVM (technical or product), not for the user. File paths are where I would start; those not read in detail are [I]. Priority is ordered: P0 first.

P0: make what exists real​

#FeatureWhatWhyEffortRiskTouches
1Live acceptance script for macOne script: import template, run --rm, exec, build, push/pull to a local registry, on a real MacGap 1; everything else sits on unverified codeMLow; needs a licensed macOS image and 2 free VM slotsscripts/ (new), crates/fluxctl/src/mac_cmd.rs
2Queue instead of 409 for run/build--queue [--timeout]: wait for a free slot, FIFO, cancel on Ctrl-CGap 3; CI hits the cap constantly with 2 slotsSLowmac_cmd.rs, mac_build_exec.rs; daemon slot count in fluxvm-apple [I]
3mac run --json / ps --json with stable fieldsMachine-readable instance id, state, IP, ssh port, labels, owner, ageA scheduler needs this (Mactra reads FluxVM labels); [S: README]SLowcrates/fluxctl/src/mac_cmd.rs
4Ownership labels and TTL on instances--label k=v, --ttl, fluxvm.owner; reaper deletes expiredCrash recovery (Mactra's recover-guests reimplements this)MMedium: must not delete live guestsfluxvm-scheduler (journal/events) [I], mac_cmd.rs
5mac wait-readyBlock until SSH (or guest agent) answers and an optional check passesRemoves per-caller SSH polling; build already does this internallySLowmac_build_exec.rs (extract), fluxvm-apple/src/ssh.rs
6Pre-flight mac doctorCheck APFS same-volume clone, FileVault, session unlocked, free disk, slot count, macOS versionHard limits in macos-containers.md fail late and confusinglySLowcrates/fluxctl/src/vz_doctor.rs, doctor.rs

P1: images and registry​

#FeatureWhatWhyEffortRiskTouches
7Layer-ish dedup on push/pullChunk-level skip: do not re-upload or re-download chunks the registry or store already hasGap 4; Anka advertises incremental pull/push [S]; 20+ GB per templateMMedium: chunk size vs APFS sparse filescrates/fluxvm-image (vm-image code), mac_store.rs
8Delta imagesPush a diff against a base digest (--from-base)Xcode point releases differ by a few GBLHigh: correctness of block diff on a bootable diskfluxvm-image, fluxvm-blockd [I]
9Build cache by step hashCache the stored disk after each RUN/COPY keyed by (base digest, step text, context hash); reuse on matchGap 4; every build starts from FROM todayLHigh: each cache entry is a full disk clone; space budget needed; APFS clones make it affordable [I]crates/fluxctl/src/mac_build.rs, mac_build_exec.rs, mac_store.rs
10mac prune and store quotaRemove unreferenced dirs and old build caches; FLUXVM_MAC_STORE_MAXTemplates are tens of GBSLowmac_store.rs
11Image signing and verificationSign manifests (cosign-style) on push, verify on pull with a policyGap 8; the image is a root-capable machineMMedium: key management; pick a standard formatmac_store.rs, mac_cmd.rs, fluxvm-image
12Registry credential helpersRead Docker config / credential helpers and OS keychain, not only two env varsEnv vars leak in CI logs and ps [I]SLowmac_cmd.rs
13Pull-through cache / peer mirror on the LANSecond Mac pulls chunks from the first Mac's store before the registryFleet cold-start: N Macs x 20 GB over WANMMedium: auth between Macsfluxvm-agent, fluxvm-image [I]
14Licence guard on public registriesRefuse push to known public registries unless --i-have-checked-the-licence; stamp the artifact with an annotationThe docs say "likely not allowed"; I found no source allowing it; make the default safeSLowmac_cmd.rs
15FROM IPSW and provisioning accountWire macos_install.rs/ipsw.rs into build; create the user without Setup AssistantGap 6; today a base image is a manual stepLHigh: Apple installer behaviour changes each macOS releasefluxvm-apple/src/macos_install.rs, ipsw.rs, mac_build.rs
16Tart/Anka-compatible import and exportExtend fluxvm-image/src/tart.rs import; add mac export --tartLets users move off or onto Tart without rebuildingMMedium: Tart's format and licence are moving [?]crates/fluxvm-image/src/tart.rs

P2: fleet and scheduler integration (Mactra-style placement)​

#FeatureWhatWhyEffortRiskTouches
17fluxctl mac run --fleetAsk the scheduler for a host, then run there; pass a template digest, not a pathGap 2. Mactra shows the shape: scheduler picks the host, FluxVM makes the guest [S: README]MMedium: auth and NAT traversalfluxctl/src/fleet_client.rs, mac_cmd.rs, fluxvm-scheduler/src/apple_placement.rs
18Template presence in placementNodes heartbeat which name:tag@digest they hold; score hosts that already have the imageAvoids a 20+ GB pull per job; Mactra checks Xcode build per template on the worker [S]SLowfluxvm-agent heartbeat, apple_placement.rs
19macOS-slot-aware placementCount macOS guests (max 2) as a separate capacity dimension and queue per hostAppleHostCaps already carries running macOS guests [F]; Mactra sidesteps with one job per host [S]SLowapple_placement.rs, apple_rebalance.rs
20Leases and fencing for instancesInstance heartbeat/lease (like Mactra's 90 s/20 s); expired lease = delete + reject stale writersAvoids zombie guests when a controller diesMMedium: clock and partition casesfluxvm-scheduler (journal, idempotency), fluxvm-agent
21Idempotent submission keymac run --request-key K returns the existing instance for the same payloadCI retries; Mactra does this for jobs [S: README]SLowfluxvm-scheduler/src/idempotency.rs [F: file exists]
22Controller-proxied ssh/vncNode dials out to the controller; mac exec/ssh/vnc go through itOrchard does this so workers can be behind firewalls [S]MMedium: another network path to hardenfluxvm-agent, fluxctl/src/fleet_tunnel.rs [F: file exists]
23Host drain and image rolloutdrain a Mac, then roll a new template tag across hosts with health gatesMactra has drain/resume [S]; FluxVM has rollout docs for other things [F: sentinel-fleet-rollout.md]MLowfluxvm-scheduler, fluxctl/src/fleet_cli.rs
24Fleet-wide macOS poolReuse the warm-pool idea per host, driven by queue depthWarm start (phase 4) is only a proposal; boot of a clone is minutes [F]LHigh: unverified macos-pool; saved state is host-boundfluxvm-apple, fluxvm-scheduler warm pool, fleet_warmpool.rs
25Artifact return channelmac cp --out plus size/SHA-256 verified download, with a capMactra's xcresult flow (256 MiB, SHA-256 verified) [S]SLowmac_cmd.rs, fluxctl/src/kube_cp.rs [I]

P3: CI runners and security​

#FeatureWhatWhyEffortRiskTouches
26fluxctl mac runner for GitHub ActionsPer-job ephemeral VM with a JIT runner token; VM deleted after the job; at most 2 per MacThe community pattern (Tart + a small controller) [S]; ARC does not cover macOS [S]MMedium: GitHub token handling; maintain across runner releasesnew crates/fluxctl/src/mac_runner.rs; builds on mac run --rm
27Same for GitLab/Buildkite/JenkinsA generic "run this command in a fresh VM and exit with its status" wrapperNot tied to one CISLowmac_cmd.rs (run --rm already does most)
28Runner image setDocumented Macfile for a runner image (Xcode, simulators, runner binary, pinned versions)Mactra leaves this to the user: "belong to the prepared template" [S]SLowdocs/, examples/
29Egress policy for macOS guestsReuse the existing egress policy and credential broker for mac run instancesUntrusted PR code runs in the VMMMedium: macOS guests lack private networks and tagged shares [F]; may need a host-side proxyfluxvm-apple/src/usernet_policy.rs [F: file exists], docs/egress-policy.md, docs/credential-broker.md
30No-secret defaultsRead-only workspace share, scrubbed environment, no host SSH agent forwarding by defaultFresh VM per job only helps if nothing leaks inSLowmac_cmd.rs
31Attested hostsUse fluxvm-apple/src/attest.rs so a scheduler can require attested hosts for sensitive imagesPublic-repo PRs should not run on a developer's MacMMedium: depends on what attest.rs proves [I: not read]fluxvm-apple/src/attest.rs, apple_placement.rs
32Audit and flight record per runEmit start/stop/image digest/commit/user to the events log; webhookCompliance and debugging; the repo has a flight recorder and event webhooks [F: docs exist]SLowfluxvm-scheduler/src/events.rs
33Linux guest runner on the same MacsA Linux (non-capped) runner image on VZ for Linux jobsPer a single source, Linux VMs are not subject to the two-VM cap [S, one source]MMedium: confirm the claim on hardwarefluxvm-apple, mac_runner.rs
34Nested-virt Linux guest for kind/k3dOffer M3+/macOS 15+ nested Linux guest for test clustersLima documents the same constraints [S]; GitHub arm64 runners cannot [S]MMedium: hardware-gated; macOS guest cannot [F]fluxvm-apple (capability.rs, host_caps.rs), apple_placement.rs (nested_virtualization flag exists) [F]

6. Suggested order​

  1. Items 1, 3, 5, 6 (verify and make it scriptable) before anything else.
  2. Items 2, 4, 14 (CI-safe behaviour, small).
  3. Items 18, 19, 17 (fleet placement using what the scheduler crate already scores).
  4. Items 7, 10, 12 (registry cost), then 26 and 29 (runner and security).
  5. Defer 8, 9, 15, 24: large, and each carries risk that only hardware testing will size.

7. Open questions​

  • Is Tart's licence and repository ownership settled? Check before promising import/export (item 16). [?]
  • Does the Linux-guest exemption from the two-VM cap hold on current macOS? One source. Test before building item 33.
  • GPU in macOS guests: unresolved here; Apple's framework docs were not read.
  • Which macOS licence text applies to the versions you will ship, and does fleet or multi-tenant hosting fit it? Needs a legal read, not a web search.