Distribute and revoke trust anchors
Goal
Get a private CA certificate (root or intermediate) trusted on every agent host it belongs on, using the same agents that already renew and deploy your leaf certificates, and revoke it cleanly when the CA is retired.
This is a different problem from certificate renewal: a trust anchor has no private key and nothing to renew. It is one CA certificate that a fleet of hosts must agree to trust (or stop trusting), tracked so a later revoke can prove it is safe to remove.
What a trust anchor is
A trust anchor is a root or intermediate CA certificate registered in your workspace. Registering it does not touch any host by itself; it only makes the anchor a valid target for distribute and revoke jobs dispatched to your agents, the same claim/lease/result lifecycle every other CertOps job uses.
| Term | Meaning |
|---|---|
| Distribute | Install the anchor into a host's OS trust store |
| Revoke | Remove the anchor from a host's OS trust store |
| Retire (anchor-level) | Mark the anchor itself inactive workspace-wide, distinct from revoking any one host's copy |
Requires the workspace admin role specifically, one level above the manager role that ordinary renewal/deploy jobs need: a trust anchor changes what an entire host trusts, not one certificate's lifecycle. Base CertOps, including trust-anchor operations, ships ungated: there is no separate license entitlement, only the CERTOPS_ENABLED rollout flag every other CertOps route already depends on.
Register the anchor
POST /api/v1/workspaces/{id}/certops/trust-anchors
{
"name": "Internal Root CA 2026",
"anchorType": "root",
"pem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
}
anchorType is root or intermediate, an explicit choice you make here, never inferred from the certificate's own fields: it decides which OS store a Windows agent targets (LocalMachine\Root vs LocalMachine\CA). Re-submitting the same PEM (same SHA-256 fingerprint) updates the existing row and reactivates it if it was retired, rather than creating a duplicate.
The response's id is the subjectId you reference below. List existing anchors with GET on the same path, optionally filtered by ?status=active.
Distribute it to an agent
Trust-anchor jobs go through the same manual-job endpoint as everything else, with operation set to distribute-trust or revoke-trust:
POST /api/v1/workspaces/{id}/certops/jobs
{
"operation": "distribute-trust",
"idempotencyKey": "distribute-trust-acme-load-balancer-fleet-1",
"subjectId": "<id from above>",
"agentId": "<target agent id>",
"owner": "acme-load-balancer-fleet"
}
owner is required (1-128 characters) and is a caller-supplied identifier for whoever is asking for this specific installation, not a TokenTimer concept: pick a stable string for the team, service, or automation that depends on this certificate being trusted on this host, since it is what a later revoke-trust is scoped against (see Ownership and multi-owner installs below).
This is always an explicit human or API-token decision: an automation-sourced request for a trust-anchor operation is rejected outright, independent of the role check above, because a job that changes what a host trusts must never have a weaker path than the one just described.
If Job approval on the CertOps Settings tab is Always required, a distribute or revoke job starts at pending_approval like every other CertOps job. The per-job checkbox cannot turn that off. See Approval gates.
Choosing an agent
Look at the agent before you pin a trust job to it. The Agents tab Execution column is Enabled when the agent declared any executable action on its last successful claim, or No capability declared (hover explains that this can mean observe-only, or an agent that has not polled yet). The Compatibility column is Compatible, Outdated, or Blocked.
The distribute and revoke dialogs annotate each agent option and show an inline warning when the selected agent has not declared trust-anchor-deploy-v1. Creating the job against a retired or Blocked agent is rejected immediately, instead of leaving a row stuck at Pending that can never be claimed. A live agent that simply has not advertised the capability yet is still accepted; that job waits at Pending until the agent heartbeats the capability, which is the usual case after an upgrade.
A trust-anchor installation stuck in pending_install or pending_remove with no error shows the same advisory reason inline, rather than a bare Pending badge.
What the agent actually does
The agent resolves the target trust store from its own detected platform, never from anything the control plane sends:
| Platform | Target | Update mechanism |
|---|---|---|
| Windows | LocalMachine\Root (root) or LocalMachine\CA (intermediate) | certutil |
| Debian/Ubuntu | /usr/local/share/ca-certificates | update-ca-certificates |
| RHEL/Fedora/AlmaLinux | /etc/pki/ca-trust/source/anchors | update-ca-trust extract |
On RHEL/Fedora/AlmaLinux, a non-zero exit from update-ca-trust extract does not by itself mean the anchor failed to install: see Diagnose Linux OS trust-store failures for why, and how the agent tells the two cases apart.
A host that is none of the above reports the job blocked, never a silent no-op. On Debian/RHEL the update command is gated by the agent's own policy.allowedCommands allowlist under two dedicated profile names, trust-store:update-ca-certificates and trust-store:update-ca-trust; see Agent configuration. An agent whose policy omits the profile for its own platform rejects the job locally before touching anything.
On Linux the agent also needs write access to those directories, which the hardened systemd unit denies by default. install-agent.sh grants it automatically on Debian/Ubuntu and RHEL/AlmaLinux; an agent installed before that grant existed fails distribution with EROFS until the installer is re-run. See OS trust store for the grant, the --no-trust-store opt-out, and the fix for already-installed agents. Windows needs no grant.
The agent also proves ownership locally: before installing or removing the certificate, it writes a restart-safe receipt to its own state directory (<configDir>/trust-receipts/, i.e. /opt/tokentimer-agent/state/trust-receipts/ on the default install layout), keyed by store and fingerprint, and only removes material on a revoke-trust job when it holds a matching, confirmed-installed receipt for it. A missing or unreadable receipt fails closed: the agent reports the removal as impossible rather than guessing.
Revoking
Send the same job-creation call with "operation": "revoke-trust". If more than one thing on that host still depends on the same certificate (for example, two independent installs referencing the identical root), the platform detects the still-live reference and marks this one released without touching the OS store at all, so a shared root is never pulled out from under a dependent that is still relying on it.
For example: if you distribute the same trust anchor to the same agent twice, once with "owner": "team-a" and once with "owner": "team-b", the platform tracks these as two independent reference rows against the same host/store/fingerprint. Revoking team-a's reference marks that row released but does not touch the OS trust store, because team-b's reference is still live; the certificate stays installed on the host. Once team-b also revokes its reference, it is the last live reference, and what happens next depends on whether TokenTimer ever actually installed the material: if some prior distribute-trust on this host/store/fingerprint genuinely mutated the OS (an agent holds an ownership receipt), the platform dispatches a real revoke-trust job that removes the certificate. If every agent that ever saw this material only ever reported it as already present (provenance: preexisting, no ownership receipt anywhere in the tuple's history), the platform releases the bookkeeping row and records the release for audit and idempotency, but dispatches no job at all, since there is nothing TokenTimer installed to remove.
To stop distributing the anchor to new hosts without touching hosts it is already on, retire the anchor itself instead of revoking every installation:
POST /api/v1/workspaces/{id}/certops/trust-anchors/{anchorId}/retire
{ "reason": "CA replaced by Internal Root CA 2028" }
Reading installation state
Expand an anchor on the Trust anchors panel, or call GET /api/v1/workspaces/{id}/certops/trust-anchors/{anchorId}/installations, to see every agent the anchor landed on (store, transition, provenance, last error). That read still works while certificate operations are paused, so you can inspect a halted workspace.
A pending_install or pending_remove row that automatic reconciliation is still retrying looks like an ordinary pending transition. Once the sweep gives up, the same row gets a Needs attention badge and a plain-language explanation, instead of looking identical to an active retry or showing the raw internal reason code.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
Job blocked, no trust store detected | Host is not Windows and not a detected Debian/RHEL-family trust store | Not a supported platform for this operation |
| Create is rejected (retired or compatibility-blocked agent) | The selected agent cannot claim any job | Pick a live, Compatible agent, or upgrade / un-retire the one you meant |
Job stuck at pending, never claimed | The target agent has not declared the trust-anchor-deploy-v1 capability, so it never becomes eligible to claim a distribute-trust/revoke-trust job | Check the agent's Execution column and the CERTOPS_AGENT_REGISTERED/CERTOPS_AGENT_CAPABILITIES_CHANGED events in the audit log; add trust-anchor-deploy-v1 to the agent's declared capabilities and let it heartbeat again, since capabilities are re-sent on every heartbeat |
| Installation row shows Needs attention | Automatic reconciliation stopped retrying a pending_install / pending_remove that made no progress | Read the plain-language reason on the row, then retry distribute or revoke once the host or agent is healthy again |
Job rejected, command_not_allowlisted | The agent's policy.allowedCommands is missing the trust-store:update-ca-certificates / trust-store:update-ca-trust profile for its platform | Add the profile to the agent's config.json and restart |
Job fails with EROFS: read-only file system on Linux | The hardened unit leaves the OS trust store read-only and the agent user has no write ACL there. Typical on an agent installed before the trust-store grant existed | Re-run install-agent.sh with the original flag set, then systemctl daemon-reload && systemctl restart tokentimer-agent. See OS trust store |
revoke-trust reports it cannot remove anything, error mentions receipt_pending_install | No matching confirmed-installed local receipt, or a stranded pending_install / receipt_finalize_conflict while the CA is still in the store | Re-run revoke-trust on the same target. It re-checks the OS trust store first and unwinds a stranded receipt when the anchor is still present. If the anchor is genuinely absent, check the original distribute-trust job. See Diagnose Linux OS trust-store failures |
403 INSUFFICIENT_ROLE | Caller's workspace role is below admin | Trust-anchor operations require admin, one level above the manager role ordinary renewal jobs need |
Related
- Agent configuration - the
policy.allowedCommandsprofile names trust-store operations are gated by. - Approval gates - the workspace setting that holds every new job, including distribute and revoke, at
pending_approval. - Automation and executors - the shared claim/lease/result job lifecycle.
- Audit log - trust-anchor distribution and revocation events, with the anchor name, fingerprint, host, and outcome recorded on each one.