User guide: CRD versioning and the conversion webhook scaffold

Every kairon.zyvor.dev CRD serves two versions with an identical schema:

Version	served	storage
`v1beta1`	yes	yes
`v1alpha1`	yes	no

Because the schemas are the same (one YAML anchor in each charts/kairon/crds/*.yaml, checked by scripts/validate.py), the API server converts between them with the default None strategy: it only rewrites apiVersion. Existing v1alpha1 manifests, GitOps repos and Kairon's own binaries (which still call v1alpha1) keep working; new objects are stored as v1beta1.

Upgrading an existing cluster

Helm installs crds/ only on first install and never upgrades them, so apply the CRDs yourself before upgrading the chart:

kubectl apply --server-side -f deploy/crd.yaml

Objects written before this stay stored as v1alpha1 until something writes them again; that is harmless while both versions are served. To rewrite them all now (needed only before a future release stops serving v1alpha1), re-store each one and then drop v1alpha1 from status.storedVersions:

for r in $(kubectl api-resources --api-group=kairon.zyvor.dev -o name); do
  kubectl get "$r" -A -o json | kubectl replace -f -
  kubectl patch crd "$r" --subresource=status --type=merge \
    -p '{"status":{"storedVersions":["v1beta1"]}}'
done

A later release moves Kairon's clients and examples to v1beta1 and marks v1alpha1 deprecated; removing it comes after that. A version with a different schema (say v1) needs the conversion webhook described below.

Why this needed real work ahead of time

A Kubernetes CRD can only ever have one storage version at a time, but can serve more than one at once -- kubectl get machinequota.v1beta1... and ...v1alpha1... both working against the same stored object, converted on the fly. That conversion has to happen somewhere: either a declarative field-rename-only strategy (None, limited to trivial renames the API server can do itself) or a real webhook the API server calls per request (Webhook, for anything more involved -- restructuring a field, splitting one into several, changing a type). This project had never built the latter, and the existing production-readiness review flagged that explicitly: "the first v1beta1 bump will need this built from scratch."

Building it speculatively, with no real second version to prove it against, would have meant either shipping untested plumbing or inventing a throwaway schema change with no real justification just to exercise it. Instead, this scaffold does the former with a real, tested worked example (MachineQuota) that never goes live -- the machinery is proven against real conversion logic and a real HTTP contract, without committing this project to an actual API version bump nobody's asked for yet.

What exists today

internal/conversion: hand-rolls the apiextensions.k8s.io/v1 ConversionReview wire format, the same kind of deliberate, documented exception internal/admission already makes for AdmissionReview -- see that package's doc comment for why this project doesn't pull in client-go/k8s.io/api for one small, stable JSON schema. Handler(log, kind, convert) returns an http.HandlerFunc implementing the webhook contract for one kind: decode the incoming Review, run every object in request.objects through a Converter, encode the outgoing Review. A decode or conversion failure always gets a well-formed Failure response, never a bare HTTP error -- the API server surfaces response.result.message straight back to whoever's kubectl apply/get triggered the conversion, which is far more useful than an opaque transport error.
ConvertMachineQuota (internal/conversion/machinequota.go): the worked example. It converts a MachineQuota between a hypothetical kairon.zyvor.dev/v1 and today's v1alpha1, renaming spec.maxTotalCpu/maxTotalMemory to spec.maxCpu/maxMemory and the matching status fields -- dropping the redundant "Total" qualifier, a plausible real API cleanup. spec.maxMachines/status.usedMachines pass through unrenamed on purpose: not every field needs to move just because the version does, and a converter that only touches what actually changed is the realistic shape a real one will take too. Operates on generic map[string]any JSON, not a typed Go struct -- there's no Go type for a version that isn't wired into the rest of Kairon's still-single-version reconcile paths, and inventing one only for this would be speculative in the same way the rest of this scaffold deliberately isn't.
kairon-controller's webhook server already exposes POST /convert/machinequotas (internal/controller/webhook.go, WebhookHandler), on the exact same TLS listener, certificate, and Service as the existing validating admission webhook (webhook.enabled, see SECURITY.md's "MachineQuota / MachineDisruptionBudget admission" section). No new listener, no new certificate to provision, no new trust boundary to reason about -- cutting a real version reuses infrastructure this project already operates.
Both are covered by real tests: internal/conversion's own unit tests (including a same-version identity case, a full round trip, an unsupported-version-pair error, and confirming the converter never mutates its input) and internal/controller's TestWebhookHandlerConvertMachineQuotaEndToEnd, which drives the route through the real HTTP ConversionReview envelope, not just the converter function in isolation.

What doesn't exist yet, on purpose

No CRD declares a spec.conversion webhook: v1alpha1 and v1beta1 share a schema, so None is enough. /convert/machinequotas (which converts between a hypothetical renamed-field v1 MachineQuota and v1alpha1) is live, tested code the API server never calls; it is the template for the first version whose schema actually differs.

What cutting a schema-changing version requires

When a version with a different schema is needed:

Add the new version to the CRD's spec.versions (served: true, storage: false initially -- flip storage to the new version only once every component that writes the CRD directly, if any, is updated to tolerate it; scripts/validate.py expects the storage version first in the array and today requires identical schemas, so update it too).

Add spec.conversion to that same CRD object:

spec:
  conversion:
    strategy: Webhook
    webhook:
      conversionReviewVersions: ["v1"]
      clientConfig:
        service:
          name: kairon-controller-webhook
          namespace: <release namespace>
          path: /convert/machinequotas
          port: 443
        caBundle: <base64 PEM, same value as webhook.caBundle>

Register or extend a Converter in internal/conversion for the real field changes (ConvertMachineQuota is the template to copy), and wire it into WebhookHandler (internal/controller/webhook.go) the same way /convert/machinequotas already is.
Solve the one real gap this scaffold deliberately leaves open: charts/kairon/crds/*.yaml lives in Helm's special crds/ directory, which Helm never templates (no .Values, no .Release access, by Helm's own design -- CRDs must be installable before any values are known) and never updates on helm upgrade. webhook.caBundle above needs a live value from values.yaml, exactly like the existing ValidatingWebhookConfiguration in templates/webhook.yaml already gets -- but that file lives in templates/, not crds/, which is how it gets away with templating .Values.webhook.caBundle at all. Cutting a real version means either moving that one CRD's management into templates/ (gaining real helm upgrade semantics for it, at the cost of diverging from how the other seven CRDs are managed) or applying spec.conversion via a separate, explicitly-documented step (a small patch script, following the same "we don't auto-mint your TLS material" posture this project already takes for webhook.tlsSecretName/console.tls.secretName/ migration.dataplaneTlsSecretName). Pick one deliberately when the time comes -- don't let it default to whichever seems less work in the moment.
Verify against a real cluster: register the version, kubectl apply an object using the new version's field names, read it back under the old version's, and vice versa -- confirming the real API server is actually invoking the webhook, not just that the converter function is correct in isolation (the same distinction this project's admission webhook work already draws between a Validator and its HTTP envelope).

Real limits today

The conversion webhook is not live. v1alpha1 ↔ v1beta1 uses the API server's None conversion; the webhook is infrastructure for a future schema change.
Only MachineQuota has a worked converter. The other CRDs have no Converter implementation yet -- ConvertMachineQuota is the template, not a generic field-rename engine every kind gets for free.
The crds/-directory templating gap (step 4 above) is unsolved. This is the one piece of "built from scratch" this scaffold doesn't remove -- it names the problem precisely instead.

Upgrading an existing cluster​

Why this needed real work ahead of time​

What exists today​

What doesn't exist yet, on purpose​

What cutting a schema-changing version requires​

Real limits today​