Skip to main content

Per-node enrollment

Until now every node held the fleet-admin token: one leaked node meant a compromised fleet, and a node's identity was the name it chose for itself. With enrollment a node joins with a join token and receives its own credential, which can only register that node.

operator ── POST /fleet/enroll/tokens ──► fvjt_<id>.<secret> (shown once; central keeps only a hash)
node ── POST /fleet/enroll (Bearer fvjt_…) {name, node_id} ──► fvnc_<node_id>.<generation>.<mac>
node ── POST /fleet/register (Bearer fvnc_…) ──► heartbeats, as before

Rules​

  • Join tokens can expire (ttl_secs), be limited to max_uses nodes and to a name_pattern (rack1-*), and be revoked (DELETE /fleet/enroll/tokens/{id}; nodes already enrolled keep their credentials). A use is counted the first time a node enrolls. 150 concurrent enrolls on a token with max_uses 100 yield exactly 100 credentials.
  • node_id is generated by the node once (<state-dir>/node-id, mode 0600). Enrolling the same name with the same node_id and token again returns the same credential and costs no use, so a lost response does not burn one. Another node_id for an enrolled name is 409 and leaves the original intact; so is one node_id under two names. To move a name to new hardware the operator runs POST /fleet/nodes/{name}/rebind, which kills the old credential.
  • Credentials are not stored: the mac is an HMAC-SHA256 of name, id and generation under a secret only central holds (fleet-enroll.json, mode 0600). A credential works for POST /fleet/register as that node (another name is 403), POST /fleet/enroll/renew, and GET /fleet/nodes/{self}/overlays. Everything else is 403. It is neither an operator nor a tenant token, and a join token is good for /fleet/enroll only.
  • Revoke (POST /fleet/nodes/{name}/revoke) stops a node that is still heartbeating, unlike deregister, which refuses a healthy node. Rotate (POST /fleet/nodes/{name}/rotate, or the node's own daily renew) issues the next generation; the previous one keeps working for an hour. Deregistering a node drops its credential.
  • Node side: fluxvm-agent node --token-file <file with fvjt_…> --state-dir … exchanges the token once, stores the credential 0600 and uses it from then on, also when restarted without the join token. It refuses a node-id or node-credential file that group or others can read. Pass the join token through a file or systemd LoadCredential=, never argv; never bake it into an image.
  • Enrollment is rate limited separately from heartbeats (500/s, burst 1000, 256 in flight; a refusal is 429 with Retry-After, which nodes and sim-fleet honour).

Migration: --node-auth​

ModeFleet token registers nodesNode credentialsUse
legacyyesrefused, /fleet/enroll is 404enrollment off
both (default)yesyeswhile nodes migrate
enrolledno (403 on /fleet/register)yesafter every node has enrolled

Check progress with GET /fleet/enroll/nodes against GET /fleet/nodes. The plan flips the default to enrolled one release after migration tooling ships; that is a separate, deliberate change.

What was tested​

cargo test -p fluxvm-agent covers the rules above (including the 150/100 race and an HTTP walk through every allowed and refused route, and a restart of central) and the node's file permission checks. scripts/test-enroll-invariants.sh runs a real central and sim-fleet: exactly 100 of 150 concurrent enrolls, a mixed enrolled/legacy fleet of 400 registering exactly once, the registry and the enrollments surviving kill -9, and enrolled mode refusing the fleet token. A real fluxvm-agent node enrolled with a join token and kept working when restarted without it.

Not tested / needs your review before relying on it: this is authentication code and has had no external security review; TLS between nodes and central is whatever you configure (credentials travel in a bearer header); there is no per-IP throttle on failed join-token guesses (the secrets are 244 random bits, and attempts are rate limited globally); mTLS node identity is deferred as decided; real-fleet behaviour (10,000 distinct clients) is untested.