Distribute and revoke trust anchors
Goal
Get a private CA certificate (root or intermediate) trusted on every agent host it belongs on, using the same agents that already renew and deploy your leaf certificates, and revoke it cleanly when the CA is retired.
This is a different problem from certificate renewal: a trust anchor has no private key and nothing to renew. It is one CA certificate that a fleet of hosts must agree to trust (or stop trusting), tracked so a later revoke can prove it is safe to remove.
What a trust anchor is
A trust anchor is a root or intermediate CA certificate registered in your workspace. Registering it does not touch any host by itself; it only makes the anchor a valid target for distribute and revoke jobs dispatched to your agents, the same claim/lease/result lifecycle every other CertOps job uses.
| Term | Meaning |
|---|---|
| Distribute | Install the anchor into a host's OS trust store |
| Revoke | Remove the anchor from a host's OS trust store |
| Retire (anchor-level) | Mark the anchor itself inactive workspace-wide, distinct from revoking any one host's copy |
Requires the workspace admin role specifically, one level above the manager role that ordinary renewal/deploy jobs need: a trust anchor changes what an entire host trusts, not one certificate's lifecycle. Base CertOps, including trust-anchor operations, ships ungated: there is no separate license entitlement, only the CERTOPS_ENABLED rollout flag every other CertOps route already depends on.
Register the anchor
POST /api/v1/workspaces/{id}/certops/trust-anchors
{
"name": "Internal Root CA 2026",
"anchorType": "root",
"pem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
}
anchorType is root or intermediate, an explicit choice you make here, never inferred from the certificate's own fields: it decides which OS store a Windows agent targets (LocalMachine\Root vs LocalMachine\CA). Re-submitting the same PEM (same SHA-256 fingerprint) updates the existing row and reactivates it if it was retired, rather than creating a duplicate.
The response's id is the subjectId you reference below. List existing anchors with GET on the same path, optionally filtered by ?status=active.
Distribute it to an agent
Trust-anchor jobs go through the same manual-job endpoint as everything else, with operation set to distribute-trust or revoke-trust:
POST /api/v1/workspaces/{id}/certops/jobs
{
"operation": "distribute-trust",
"idempotencyKey": "distribute-trust-acme-load-balancer-fleet-1",
"subjectId": "<id from above>",
"agentId": "<target agent id>",
"owner": "acme-load-balancer-fleet"
}
owner is required (1-128 characters) and is a caller-supplied identifier for whoever is asking for this specific installation, not a TokenTimer concept: pick a stable string for the team, service, or automation that depends on this certificate being trusted on this host, since it is what a later revoke-trust is scoped against (see Ownership and multi-owner installs below).
This is always an explicit human or API-token decision: an automation-sourced request for a trust-anchor operation is rejected outright, independent of the role check above, because a job that changes what a host trusts must never have a weaker path than the one just described.
What the agent actually does
The agent resolves the target trust store from its own detected platform, never from anything the control plane sends:
| Platform | Target | Update mechanism |
|---|---|---|
| Windows | LocalMachine\Root (root) or LocalMachine\CA (intermediate) | certutil |
| Debian/Ubuntu | /usr/local/share/ca-certificates | update-ca-certificates |
| RHEL/Fedora/AlmaLinux | /etc/pki/ca-trust/source/anchors | update-ca-trust extract |
On RHEL/Fedora/AlmaLinux, a non-zero exit from update-ca-trust extract does not by itself mean the anchor failed to install: see Diagnose Linux OS trust-store failures for why, and how the agent tells the two cases apart.
A host that is none of the above reports the job blocked, never a silent no-op. On Debian/RHEL the update command is gated by the agent's own policy.allowedCommands allowlist under two dedicated profile names, trust-store:update-ca-certificates and trust-store:update-ca-trust; see Agent configuration. An agent whose policy omits the profile for its own platform rejects the job locally before touching anything.
On Linux the agent also needs write access to those directories, which the hardened systemd unit denies by default. install-agent.sh grants it automatically on Debian/Ubuntu and RHEL/AlmaLinux; an agent installed before that grant existed fails distribution with EROFS until the installer is re-run. See OS trust store for the grant, the --no-trust-store opt-out, and the fix for already-installed agents. Windows needs no grant.
The agent also proves ownership locally: before installing or removing the certificate, it writes a restart-safe receipt to its own state directory (<configDir>/trust-receipts/, i.e. /opt/tokentimer-agent/state/trust-receipts/ on the default install layout), keyed by store and fingerprint, and only removes material on a revoke-trust job when it holds a matching, confirmed-installed receipt for it. A missing or unreadable receipt fails closed: the agent reports the removal as impossible rather than guessing.
Revoking
Send the same job-creation call with "operation": "revoke-trust". If more than one thing on that host still depends on the same certificate (for example, two independent installs referencing the identical root), the platform detects the still-live reference and marks this one released without touching the OS store at all, so a shared root is never pulled out from under a dependent that is still relying on it.
For example: if you distribute the same trust anchor to the same agent twice, once with "owner": "team-a" and once with "owner": "team-b", the platform tracks these as two independent reference rows against the same host/store/fingerprint. Revoking team-a's reference marks that row released but does not touch the OS trust store, because team-b's reference is still live; the certificate stays installed on the host. Once team-b also revokes its reference, it is the last live reference, and what happens next depends on whether TokenTimer ever actually installed the material: if some prior distribute-trust on this host/store/fingerprint genuinely mutated the OS (an agent holds an ownership receipt), the platform dispatches a real revoke-trust job that removes the certificate. If every agent that ever saw this material only ever reported it as already present (provenance: preexisting, no ownership receipt anywhere in the tuple's history), the platform releases the bookkeeping row and records the release for audit and idempotency, but dispatches no job at all, since there is nothing TokenTimer installed to remove.
To stop distributing the anchor to new hosts without touching hosts it is already on, retire the anchor itself instead of revoking every installation:
POST /api/v1/workspaces/{id}/certops/trust-anchors/{anchorId}/retire
{ "reason": "CA replaced by Internal Root CA 2028" }
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
Job blocked, no trust store detected | Host is not Windows and not a detected Debian/RHEL-family trust store | Not a supported platform for this operation |
Job stuck at pending, never claimed | The target agent has not declared the trust-anchor-deploy-v1 capability, so it never becomes eligible to claim a distribute-trust/revoke-trust job | Check the agent's declared capabilities via CERTOPS_AGENT_REGISTERED/CERTOPS_AGENT_CAPABILITIES_CHANGED in the audit log; add trust-anchor-deploy-v1 to the agent's declared capabilities and let it heartbeat again, since capabilities are re-sent on every heartbeat |
Job rejected, command_not_allowlisted | The agent's policy.allowedCommands is missing the trust-store:update-ca-certificates / trust-store:update-ca-trust profile for its platform | Add the profile to the agent's config.json and restart |
Job fails with EROFS: read-only file system on Linux | The hardened unit leaves the OS trust store read-only and the agent user has no write ACL there. Typical on an agent installed before the trust-store grant existed | Re-run install-agent.sh with the original flag set, then systemctl daemon-reload && systemctl restart tokentimer-agent. See OS trust store |
revoke-trust reports it cannot remove anything, error mentions receipt_pending_install | The agent holds no matching, confirmed-installed local receipt for that store/fingerprint. This includes the settled receipt_finalize_conflict case (a prior distribute-trust actually installed the CA and the control plane recorded it as succeeded, but the agent's own local receipt-finalize write failed afterward) and an old receipt stranded pending_install from before 0.14.2 | Re-run revoke-trust against the same target: as of 0.14.2 it re-observes the real OS trust-store state first and unwinds (finalizes, then removes) a stranded receipt whenever the anchor is still observably present, which also covers the receipt_finalize_conflict case. If the anchor genuinely is not present, this is expected: check the original distribute-trust job's result instead. See Diagnose Linux OS trust-store failures |
403 INSUFFICIENT_ROLE | Caller's workspace role is below admin | Trust-anchor operations require admin, one level above the manager role ordinary renewal jobs need |
Related
- Agent configuration - the
policy.allowedCommandsprofile names trust-store operations are gated by. - Automation and executors - the shared claim/lease/result job lifecycle.
- Audit log - trust-anchor distribution and revocation events, with the anchor name, fingerprint, host, and outcome recorded on each one.