Skip to main content

Provisioning a raw disk (first integration slice)

Atlas can turn a raw, unformatted block device (e.g. /dev/sdb) into usable capacity two ways: a new ZFS pool (direct zpool create) or a new Ceph OSD via Rook (patches the CephCluster CR's device list and waits for Rook's own reconciler to claim the disk). Both are async jobs (202 Accepted + job_id, same shape as every other write path) and both require an explicit "confirm": true — formatting a disk is irreversible.

Device selection is backed by two read-only discovery endpoints the Disks UI's picker calls instead of a blind free-text field: GET /api/atlas/v1/zfs/devices (bulk lsblk/wipefs/zpool status scan of the gateway's own host) and GET /api/atlas/v1/ceph/nodes/{node}/devices (passes through Rook's own local-device-<node> discovery ConfigMap for one node). Both are advisory only — the checks below are re-run, authoritatively, against whatever device is actually submitted.

Safety model​

Every real command is preceded by defense-in-depth checks that refuse anything that looks like it could be in use. Nothing is ever marked succeeded until independently verified — never fabricated (matching every other driver's "never fabricate" convention in this codebase).

atlas_common::device::validate_raw_device_path is a cheap first gate, applied at both the route and the job-dispatch layer: the path must be a whole disk (/dev/sdb, /dev/vdc, /dev/nvme1n1 — never a partition like /dev/sdb1), and the conventional first-disk name for each bus (sda, vda, nvme0n1) is refused outright as a static heuristic.

ZFS: POST /api/atlas/v1/zfs/pools/from-device​

{ "pool_name": "tank2", "device_path": "/dev/sdb", "confirm": true }

Local-host-only: the gateway process must be running on the host that owns the device (the same limitation RealZfsDriver itself already has — ZFS has no remote query protocol, unlike Ceph). Requires ATLAS_ZFS_ENABLE=1 at startup. Before running zpool create, the job independently checks (via lsblk, findmnt, wipefs, zpool status) that the device: exists, is a whole disk, isn't read-only, has no partitions/filesystem/partition-table signature, isn't mounted, isn't a member of a currently imported zpool, and isn't the host's actual mounted root/boot disk. zpool create is never passed -f — that flag exists specifically to override zpool's own built-in in-use refusal, which stays as an independent safety net underneath Atlas's own checks.

Found live, then closed (2026-09-28): the root/boot-disk check is dynamic (resolves the host's actual mounted root/boot device) specifically so it isn't fooled by an unconventional layout where the boot disk isn't the first letter/index — but doing that via findmnt alone is blind when the gateway runs in a container without visibility into the host's mount namespace. Confirmed live: on the deploy/k8s/atlas-gateway.yaml lab deploy (before this fix), /dev/sda (this lab host's actual root/boot disk) was not flagged by findmnt alone, and showed up in the Disks UI's device picker as "has data — wipeable" — atlas_common::device::validate_raw_device_path's static name-based refusal (sda/vda/nvme0n1) still independently blocked an actual submit against it regardless, so no real host ever had its root disk formatted by this. Closed by also hostPath-mounting the host's real root-mount-namespace mount table as a single read-only file (/proc/1/mountinfo on the node → /host-mountinfo in the container, ATLAS_HOST_MOUNTINFO_PATH) and resolving root/boot disks from it directly (major:minor → /sys/dev/block/<maj>:<min> → parent whole-disk basename — sysfs block topology isn't mount-namespace-scoped, so this works from inside the container). The plain findmnt check and the static first-disk-name gate both stay in place unconditionally alongside it — this is additive, not a replacement.

Active pool members are a hard refusal (found live, 2026-09-28): a disk that belongs to an imported zpool is refused outright — wipe_existing cannot override it, and the picker hides it (status zpool_member) — because wiping a live pool's disk corrupts a running pool. (Stale ZFS labels from an exported/destroyed pool are the wipeable case: they show as a zfs_member filesystem signature, not pool membership.) The first version of this check missed real pools: a whole-disk pool lists the partition as its vdev (zpool status -P shows /dev/sdb1 for a pool created on /dev/sdb), so an exact whole-disk comparison never matched and the live pool's disk was still offered as "wipeable". Matching now recognizes the device, its partitions (sdb1, nvme1n1p1 — without confusing sdbb or nvme1n11 for them) and by-id -partN aliases. Zero-size devices (unattached /dev/nbd* stubs) are not offered either.

Resolved live (2026-09-28): zpool create (whole-disk mode) writes a new GPT and then immediately reopens the partition device it just asked the kernel to create — but that reopen polls udev's device database (/run/udev/data/b<major>:<minor>), root-caused via strace attached directly to the live zpool process: openat("/run/udev/data/b8:17", …) = -1 ENOENT. The lab manifest only ever hostPath-mounted /dev, never /run/udev, so that file could never appear from inside the container — not a transient race, a permanently missing mount, which is why zpool_create's own 5-attempt retry never helped. Fixed by also hostPath-mounting /run/udev (read-only) in deploy/k8s/atlas-gateway.yaml. The exact same udev-database-propagation lag showed up one step earlier too — the post-wipe recheck (refusal_reason_after_wipe) could see a stale cached pttype for a short window right after wipefs -a genuinely finished — so that recheck also retries a few times with backoff before giving up. With both fixes, a full wipe_existing + zpool create run against a disk carrying a stale Ceph OSD partition table (real bluestore signature, not a fixture) succeeded end-to-end through the console: zpool status confirms tank0/ONLINE, backed by the real device, zero errors.

zpool create is synchronous, so the job succeeds as soon as it returns 0. The new pool/root dataset are written directly into inventory (not left to a follow-up discovery pass) — because RealZfsDriver's configured zpool list is fixed at gateway startup (ATLAS_ZFS_POOLS), a subsequent discovery pass alone would never notice a pool that wasn't in that list when the process started. Known limitation: this pool's capacity/health numbers go stale until the gateway is restarted with the pool name added to ATLAS_ZFS_POOLS.

ZFS: POST /api/atlas/v1/zfs/pools/{name}/destroy​

{ "confirm_pool_name": "tank2" }

Async job (zfs.pool.destroy), admin-only; the Disks console page lists the host's pools with a Destroy… button that asks for the pool name to be typed back. Refusals: the confirmation must equal the pool name; the pool must be in Atlas's inventory (404 otherwise); inventory must show no volume in it other than the pool's own root dataset (409); and on the host zfs list -r must show no dataset below the root. zpool destroy is never passed -f, so zpool's own "pool is busy" refusal stays underneath. The inventory pool/volume rows are dropped only after the destroy succeeded (or after zpool list shows the pool is already gone). The member disk keeps its ZFS labels, so it reappears in the Disks picker as a zfs_member device that can be re-provisioned with the explicit wipe option — that is the destroy → wipe → re-provision cycle.

Status: unit- and fake-mode-tested (tests/disks.rs) and verified live (2026-09-28) through the console on the lab host: destroyed the real pool tank0 (job succeeded, pool gone from inventory), /dev/sdb reappeared in the picker as "has data — wipeable", and re-provisioning it with the wipe option as tank1 succeeded (zpool status: tank1 ONLINE on sdb, zero errors, 928G).

Helm-deployed gateway (disks.enabled) — verified live​

Same cycle through the console of a Helm-installed gateway (deploy/helm/atlas, disks.enabled=true, release atlas-helm), 2026-09-28: while the primary gateway's pool tank1 owned sdb, the Helm gateway's picker listed no usable disks (sdb is an active pool member — hard refusal). After tank1 was destroyed from the primary console, the Helm console offered sdb as "has data — wipeable", formatted it with the wipe option as tankh (zpool status: ONLINE, zero errors), and destroyed it again; tank1 was then re-created from the primary console. Each gateway's inventory only knows the pools it created, so one gateway cannot destroy the other's pool. Two privileged, disk-enabled gateways on one node see the same physical disks — let only one own a pool at a time.

Ceph: POST /api/atlas/v1/ceph/devices​

{ "node_name": "node-1", "device_path": "/dev/sdb", "confirm": true }

Requires a live Kubernetes cluster. Refuses if the target CephCluster's storage.useAllDevices is true — that mode already auto-claims every empty device on every node, so this API only applies when devices are pinned explicitly per node. Patches spec.storage.nodes[].devices via a scoped JSON merge patch (never a full-spec replace, which would clobber unrelated cluster settings) and is idempotent — safe to retry.

Rook's reconciliation (ceph-volume prepare/activate) can take several minutes. Rather than blocking the job engine's single worker (which would stall every other tenant's jobs), each dispatch invocation does one non-blocking check of ceph device ls for a matching OSD and returns; an unfinished check is an Err that the job engine's own retry/backoff machinery reschedules automatically (exponential backoff capped at 64s/attempt, ~25–30 minutes of polling before a final failed). The job only reaches succeeded once ceph device ls actually shows the OSD attached — never on CR-patch-accepted alone.

Dependency: the safety check before patching the CR reads Rook's own device-discovery ConfigMap (local-device-<node>, populated by Rook's discovery DaemonSet). If that ConfigMap is absent, the request fails closed (a clear 503) rather than skipping the check — enable ROOK_ENABLE_DISCOVERY_DAEMON on the Rook operator if this happens.