atlas-native asynchronous replication

Asynchronous replication keeps a read-only copy of an atlas-native filesystem (NATIVE_FS.md) on a second native cluster, typically in another site. It works like ZFS send/receive: the source is snapshotted on a schedule, the changes between the last replicated snapshot and the new one are sent, and the target commits them as a snapshot of its own. If the source site is lost, the replica is promoted and becomes writable; when the source returns, it is demoted onto the last snapshot both sides share and replication runs the other way.

Piece	Role
Source cluster	GET /v1/fs-snapshots/{to}/diff?from= pages the changes between two snapshots of a filesystem; file data is read from the snapshot tree (<fs>@<snapshot>)
Target cluster	A replica filesystem that clients can read but not change (409 read_only, EROFS through FUSE, NFS and SMB) until it is promoted; only increments change it
atlas-native-replicate (crates/atlas-native-fuse, bin)	Snapshots the source, pages the diff, applies it to the target, copies the changed data, commits and prunes old snapshots. Stateless: the replica records where to continue

What is sent

The diff compares the two snapshot trees inode by inode. The replica keeps the source's inode numbers, so after each commit its tree matches the source snapshot inode for inode: the same names, modes, owners, link counts, timestamps, extended attributes (POSIX ACLs included), symlink targets, device numbers and hard links.

• Directories send only their changed entries, and only when their ctime changed (every entry change stamps it).
• Files send their attributes, and data only for the grid cells (extent_bytes, 4 MiB by default) whose extent changed between the snapshots. Rewriting one aligned 4 MiB cell of a 256 MiB file sends 4 MiB; a small write sends its whole cell. Snapshots share extents, so finding the changes reads no data.
• Removed inodes are listed separately; their extents are released on the replica.

The replicator splits large increments into requests of about 512 KiB of metadata and copies file data with --parallel (default 4) concurrent requests of at most --max-io-bytes (default 4 MiB).

Consistency

• Each completed round leaves a snapshot on the replica (its local snapshot, with the same id as the source snapshot). That snapshot always equals a point-in-time snapshot of the source.
• During a round the live replica tree is part-way between two snapshots. To read a consistent view while replication runs, read the latest replica snapshot (<fs>@<snapshot>, or atlas-native-mount --snapshot).
• A round cut short (replicator restart, network failure) leaves the increment pending on the replica. The next round finishes the same increment: it re-sends the whole increment, not just what is missing. A round for a different increment first discards the partial one.
• Promotion never exposes a partial increment: the replica reverts to its last complete snapshot first.
• RPO is roughly the interval plus the time a round takes. RTO is one API call: promotion is a catalog operation (55 ms on the lab).

Install

Replication runs next to the source cluster and connects to the target over its API. Put the target's API token in a Secret in the source namespace, then enable it in the source release of the atlas-native chart:

kubectl -n atlas-native create secret generic dr-token --from-file=token=<target-token-file>

replication:
  enabled: true
  target:
    endpoints: ["https://<dr-cluster-api>:7480"]
    tokenSecret: dr-token
    caSecret: dr-ca            # optional: Secret with ca.crt for a private CA
  intervalSecs: 300
  keep: 24                     # replicated snapshots kept on the target
  jobs:
    - fs: team-share
      targetFs: team-share     # optional, default fs
      intervalSecs: 60         # optional per job

Each job is an unprivileged Deployment (<release>-atlas-native-replicate-<fs>, image Dockerfile.native-replicate). The first round creates the replica on the target (same name and grid as the source) and copies everything. httpTls.requireClientCert (mTLS to the API) isn't supported by the replicator yet.

Without the chart:

atlas-native-replicate \
  --source https://<src-api>:7480 --source-token-file src.token --source-ca-file src-ca.crt \
  --target https://<dr-api>:7480  --target-token-file dr.token  --target-ca-file dr-ca.crt \
  --fs team-share --interval-secs 300 --keep 24

--once runs one round and exits with its status (for cron or a CI drill). Each round logs one line: the snapshot, the snapshot it started from, whether it resumed a partial round, and the inodes, removals and bytes it sent. Tokens are read from files and never logged.

Snapshots and retention

• The replicator's snapshots have ids <prefix><unix-ns> (prefix repl- by default, --prefix). It deletes only snapshots with its prefix.
• On the source it keeps only the latest replicated snapshot: the next increment starts from it. Deleting that snapshot by hand breaks the chain; the replicator then reports that the base is gone and a new replica has to be seeded.
• On the target it keeps the newest --keep. The replica's current base snapshot can't be deleted, by the replicator or anyone else.
• A round runs every interval even when nothing changed. It takes a new snapshot and sends an empty increment.

Source and target must be separate clusters: the replica's snapshots reuse the source snapshot ids.

Failover and failback

Failover (the source site is down or being retired):

curl -X POST -H "Authorization: Bearer $DR_TOKEN" https://<dr-api>:7480/v1/fs/team-share/promote

The replica reverts to its last complete snapshot if a round was in flight, stops being a replica and accepts writes. {"force": true} promotes a replica whose first round never finished, as it is. Stop the replicator first (replication.enabled: false, or scale its Deployment to 0), so it doesn't keep trying to update a filesystem that is now writable.

Failback (the old source is back): the old source has to become a replica of the promoted side. Run the replicator the other way (source = DR, target = old source). It refuses a writable target, and its error names the newest snapshot both sides have:

target filesystem team-share is writable, not a replica; to fail back, demote it onto the snapshot
both sides have: POST /v1/fs/team-share/demote {"snapshot":"repl-…","base":"repl-…"}

curl -X POST -H "Authorization: Bearer $SRC_TOKEN" https://<src-api>:7480/v1/fs/team-share/demote \
  -H 'content-type: application/json' -d '{"snapshot":"repl-…","base":"repl-…"}'

Demotion reverts the old source to that snapshot, discarding anything written there after it (writes that never replicated before the failover), and makes it a replica. The reverse round then sends only what changed on the DR side since. To move back to the original direction, promote the original source and demote the DR side onto its latest replicated snapshot the same way.

API

Method	Path
POST	/v1/fs {"id","name","replica":true,"extent_bytes"}	Create an empty replica (idempotent)
GET	/v1/fs	Each filesystem's replica state: base (source snapshot it matches), local (its own snapshot of that), pending (increment in flight)
GET	/v1/fs-snapshots/{to}/diff?from=&after=&limit=	One page of changes from snapshot from (none: everything) to to, in inode order; pass next as after
POST	/v1/fs/{fs}/replica/apply {"from","to","inodes","removed"}	Apply one part of an increment
PUT	/v1/fs/{fs}/replica/inodes/{ino}/data?offset=	Write file data of the increment
POST	/v1/fs/{fs}/replica/commit {"from","to","snapshot","next_ino"}	Complete the increment and snapshot the replica (idempotent)
POST	/v1/fs/{fs}/promote {"force"?}	Make a replica writable
POST	/v1/fs/{fs}/demote {"snapshot","base"}	Revert a writable filesystem to its snapshot, which matches source snapshot base, and make it a replica

Every change is a Raft-replicated catalog operation on the target, validated before it changes anything: an increment that doesn't continue the replica's base, a type change of an existing inode, or removing the root is refused whole.

Not replicated

• Quotas: the replica's quota is its own (PUT /v1/fs/{fs}/quota).
• Client sessions, file locks and cache leases: they belong to the clients of one cluster.
• Tiering placement and erasure-coding layout: the target writes the data with its own settings.
• Snapshots other than the replicator's: only the chain of replicated states arrives.

Verified

• Unit tests (cargo test -p atlas-native --lib replica): full and incremental increments keep the replica's tree equal to the source snapshot, with usage counters checked on every operation; partial increments are discarded by a different increment and by promotion; failback by demotion.

• Integration test (cargo test -p atlas-native-fuse --test replicate, two in-process clusters): full round, empty round, an increment that copies only the rewritten grid cell and the new file, truncate, rename, unlink, xattrs, symlinks, resuming a cut-short round, read-only replica (EROFS), failover, failback with the hint, and back to the original direction.

• Live on the lab k3s (two single-node native clusters in two namespaces on one shared host, local-path volumes; 2026-10-06):

  Step	Result
  Full round: 1 012 inodes, a 256 MiB file and 1 000 files of 4 KiB	260 MiB in 105 s (~2.5 MB/s); SHA-256 of the large file equal on both sides
  Increment: 4 MiB rewritten in the large file, 50 new files, 10 removed	54 inodes, 10 removals, exactly 4 399 104 bytes sent, 4.3 s
  Create on the replica	HTTP 409 read_only
  Promote the replica, write a file there	55 ms
  Reverse round against the writable old source	Refused, naming the common snapshot; after demote, 2 inodes and 22 bytes sent, the DR-side file readable on the old source
  Forward again after promoting the original side and demoting the DR side	Empty increment, 0.2 s
  Helm-deployed replicator (replication.jobs), pod deleted 8 s into a 256 MiB round	Next pod: resumed=true, the increment finished (256 MiB in 63 s), SHA-256 equal; the source kept only its base snapshot

  Throughput on the lab is bound by the shared host's disk: writing the same data straight into the source cluster ran at ~3 MB/s. It has not been measured on NVMe or across a WAN.