Distribute one certificate to several destinations

Start with the Vault publication and distribution walkthrough. Use this page for the detailed API sequence and Vault Agent authentication for host credentials.

The first-run guide offers Distribute certificate files only and Distribute and reload a service before consumer assignment. The explicit installed policy installs and reads back the certificate/key pair (and configured chain), without reload commands or TLS probes. Its receipt has installed: true, with bound, servedVerified and trustValidated false; a completed rollout containing file-only consumers has state installed. Existing served and trust policies still require live listener evidence. Current counts expire with evidence freshness and do not reuse historical service verification after a policy or generation change. Use the matching updated API/worker and consumer-agent build for file-only delivery; older agents reject that policy before mutation.

What this workflow does

Use this workflow when several services must serve the same certificate and private key: two load-balancer listeners, a primary server and its fallback, or several reverse proxies. The certificate can cover a single name, several SANs, or a wildcard. A wildcard is one use case; it is not a requirement.

For a first distribution, the setup guide runs signed checks, creates the publication, assigns consumers, returns each host's configuration revisions and requests the selected version's rollout. It shows inline timelines, read-only Agent output and actual consumer verification; no terminal TokenTimer API calls are required. Inspect later status in Certificate Management → Certificates → certificate → Deployments. Renewals contains renewal profiles and AD CS issuers. The API sequence below remains available for automation, advanced binding changes, containment and verification-only requests. Approval review uses the existing Jobs surface when workspace policy requires it. Distribution changes and topology remain manager/admin-only; Cloud also enforces the workspace's Pro/Team entitlement and current workspace state.

One customer-side agent issues or renews the certificate and publishes the certificate/key bundle to your Vault KV v2. Explicitly assigned consumer agents retrieve that exact version, install it on their own hosts, reload the service, and verify the certificate each listener actually serves. TokenTimer coordinates jobs and records public evidence.

Supported scope

The three required agent capabilities (material-store-vault-kv2-v1, certificate-publication-v1, and deploy-from-store-v1) are declared by the 0.18.0 agent. Your own Vault authentication, DNS provider, host filesystem and TLS listeners remain customer-owned: test them before you rely on this path.

The initial execution path targets Linux file-based services. NGINX has been exercised as a consumer on Linux. Test any other web server yourself before relying on it. Firewall appliances, their HA members, Windows shared-key import, and Kubernetes bundle distribution are not covered. Existing IIS and cert-manager workflows are separate.

How the pieces fit together

Term	Meaning
Issuer	The agent responsible for issuance and renewal. Consumers do not create independent renewal orders.
Distribution group	The shared publication owner and its explicitly assigned destinations.
Material version	An immutable reference to one certificate/key bundle. It includes public fingerprint and expiry metadata and a pinned Vault provider version.
Consumer binding	An assignment to one agent and one locally configured deployment profile, with an authorization revision.
Rollout	An authorized, version-pinned attempt to bring a fixed set of consumers to a particular material version.
Generation	A monotonically increasing rollout number that prevents older work from overwriting newer intent.

Publishing successfully means the bundle is available. It does not mean every destination uses it. A disconnected fallback remains out of date until its own verification succeeds.

Prerequisites and local configuration

Cloud requires a Pro or Team workspace with CertOps enabled. Use a workspace manager/admin session for distribution changes; approval also requires an authorized session. Frozen or unpaid workspaces cannot start new distribution effects. Claim-bound results from work already executing remain reportable.

Prepare the customer environment before requesting work:

1. Install and enroll the agents. Use a 0.18.0 or later agent, which declares the distribution capabilities described above.
2. Configure a Vault KV v2 mount and separate Vault Agent auto-auth processes for the issuer and consumers. Each process writes its own protected token sink; the adapter rereads it on each request so token renewal/replacement works. Keep the sinks readable only by the agent account, with protected parent directories.
3. Give the issuer create/read permission for its exact bundle prefix and consumers read-only permission. Give the public inventory scanner a separate identity with no bundle access. Broader Vault policies must not defeat that separation.
4. Configure HTTPS trust for Vault locally, including an approved CA file when needed. Keep the address, mount, namespace, token-file path, DNS credentials and CA settings on the customer hosts.
5. Bind each local store entry to the exact workspace, group, SAN set and key algorithm. The issuer policy also identifies the issuance profile/revision, CA and DNS zone. Each consumer policy identifies its deployment profile/revision, authorization revision, file destinations, approved reload command and every dial address, port and SNI to verify.

The version-matched Core source includes the configuration templates docs/certops/examples/wildcard-vault/issuer.json, consumer.json and vault-policies.hcl. Their directory name is historical; the same mechanism supports ordinary and SAN certificates. Substitute customer IDs and paths, then merge the appropriate template into the existing local agent configuration. A binding uses the agent's database row ID; this differs from its local wire agentId.

Create each consumer deploy folder with the installer's --write-path or with sudo install -d -o tokentimer-agent -g tokentimer-agent -m 0750 <folder>. Any placeholder fullchain.pem and privkey.pem must also belong to the agent user. With root-owned files the first rollout fails with material_install_failed. A 0.18.0 agent restores the previous binding record when a deploy fails before anything was written (validation, path resolution, reading the existing files, taking backups), so the retry runs as soon as the ownership is fixed. A failure after writing started, with no clean rollback, still leaves the binding uncertain and later jobs report material_binding_uncertain; pre-release 0.18.0 agent builds also did this after a deploy that wrote nothing. Local reconciliation (see Reconcile a binding) only clears that marker when the installed pair matches the intended version; otherwise it reports material_reconciliation_mismatch. Create the folder and any placeholder files as the agent user before the first rollout. The agent also reads the issuer's private key from execution.keysDir, so list that folder (on the Linux installer layout /opt/tokentimer-agent/state/keys) in policy.allowedPaths next to the deploy folder. The templates ship dryRun: true; set it to false and restart the agent only after a signed dry run has passed.

The agent routes jobs by what it declares, not by what the policy allows. Set declaredTargetSelectors (issuer: the certificate SANs; consumer: the binding ID) and, for the issuer, declaredCommandProfileNames (for example ["certbot"]). Without them the agent registers but never claims work. The templates run certbot with --agree-tos and --register-unsafely-without-email; use --email <address> instead if you want expiry notices from the CA. For an nginx consumer, nginx -t fails for the unprivileged agent user, so the template validates with systemctl is-active and reloads with systemctl reload nginx. Add the one-time systemd drop-in that tests the configuration as root, described under nginx reload.

With verificationPolicy: "trust" the consumer checks the chain the listener serves against the host's trust store. When the certificate comes from a private CA, add trustCaFile (an absolute path to the CA PEM) to the probe entry, for example { "dialAddress": "127.0.0.1", "port": 443, "sni": "www.example.com", "trustCaFile": "/etc/ssl/tokentimer/ca.pem" }. A probe that cannot validate the chain fails the rollout with material_service_verification_failed.

Never put private keys, Vault tokens, DNS credentials, secret bundle contents or local executable commands in a TokenTimer API request. The control plane stores public metadata and references. Private-key payloads are rejected with HTTP 422. Protect Vault backups and agent staging as private-key material.

Publish once, then assign destinations

Run the signed no-op transport check before allocating the first publication. Then keep issuer polling disabled (execution.enabled: false), or hold the initial job for approval. The job response supplies job.payload.publication.groupId; use that real UUID in the local group policy, Vault prefix and AppRoles. Configure the separate Vault Agent roles and sinks before approving or starting the issuer. A randomly chosen group UUID will not authorize the allocated publication. With ordinary job approval off, do not resume polling with dryRun: true while the real publication is queued: it can be claimed and blocked. Resume with execution.enabled: true and execution.dryRun: false only after policy and Vault access are ready.

The examples below use the supplied templates' aliases: customer for the store, wildcard for the issuance profile, and nginx-public for the consumer profile. Keep each alias identical in local configuration and API payloads, or rename it consistently everywhere. wildcard is an alias, not a requirement to request a wildcard certificate.

The API paths below use this prefix:

/api/v1/workspaces/{workspaceId}/certops

1. Create an issuance job assigned to the issuer agent. Keep the normal issuance target, SAN, CA, DNS and key settings. Add payload.publicationDestination with these fields:

   {
     "materialStoreRef": "customer",
     "issuanceProfileRef": "wildcard",
     "profileRevision": 1
   }

   These are aliases for the issuer's local policy. Do not include filesystem, reload or listener destinations in this publication request. Include keyAlgorithm in the payload (for example ecdsa with keySize: 256); the agent refuses the job without it. Supply the normal issuance idempotency key.

2. Job creation reserves the managed source, distribution group and logical material version before agent execution. After local Vault policy is ready, approve the job if required and enable issuer execution. Initial issuance is stored as an issue operation and dispatched through the agent's signed renew action. The issuer validates the pair, stages it locally, writes the deterministic Vault object with create-only CAS, and reads back its pinned version. The API accepts a public publication receipt only for a valid live claim and management period.

3. List GET .../distribution-groups, then GET .../distribution-groups/{groupId}/versions to obtain the group and published version IDs.

4. Assign each destination using PUT .../distribution-groups/{groupId}/consumers/{bindingId}. Choose a distinct UUID for each binding. For example:

   {
     "agentId": "11111111-1111-4111-8111-111111111111",
     "deploymentProfileRef": "nginx-public",
     "profileRevision": 1,
     "required": true,
     "wave": 0,
     "verificationPolicy": "trust",
     "freshnessSeconds": 3600,
     "state": "active"
   }

   Replace the example agent UUID and alias with the enrolled agent and its local profile. Use the initial wave for a canary and later waves for the other destinations. Required consumers must succeed before later waves proceed. An optional failure stays visible and never counts as verified.

Changing a binding increments its authorization revision. Review and update the matching local policy before requesting new work; a server assignment cannot grant access to an arbitrary local path or command.

Request and verify a rollout

Finish configuring the consumer hosts before requesting a rollout. With Job approval off, the following request authorizes the exact rollout and queues deployment without another user. With Job approval on, it waits for a different authorized approver.

Request POST .../distribution-groups/{groupId}/rollouts with an Idempotency-Key header and this body, replacing the material version UUID with the published version:

{
  "materialVersionId": "22222222-2222-4222-8222-222222222222",
  "maxParallel": 1
}

Use the actual published version UUID. With workspace Job approval off, a rollout starts without a second user unless the request explicitly sets requiresApproval: true. With approval on, another authorized manager must approve the returned job through Certificate operations → Jobs or the job approval API. Consumer jobs also follow workspace policy. The first-run guide supplies assignment and rollout controls; the certificate inventory panel is a later status view.

The rollout pins membership, policy revisions, material and Vault versions, fingerprint, generation and concurrency. Changed intent must be authorized again before expansion; when approval is required, stale approval cannot authorize the changed request. Consumer work is dispatched as signed deploy-from-store jobs; consumers do not issue a replacement certificate. To hold a rollout for review even when Job approval is off, use this request body instead:

{
  "materialVersionId": "22222222-2222-4222-8222-222222222222",
  "maxParallel": 1,
  "requiresApproval": true
}

Use a fresh idempotency key for a new request; retry the same request with its original key after a lost response. A request cannot bypass workspace-forced approval by setting requiresApproval: false. If the returned job is pending_approval, open it in Certificate Management → Jobs and have a different authorized user approve it, or use the job approval API. When approval is off, the rollout authorization job can already be succeeded; follow the rollout and its consumer jobs for deployment progress.

In Certificate Management → Certificates, open Deployments in the certificate details, or call GET .../distribution-groups/{groupId}/consumers. Compare desired and observed versions, generation, fingerprint, deployed expiry, verification method/time and the last failure. Publication is separate from verified deployment: a published Vault version does not prove any consumer serves it. Partial, pending, outdated, failed and uncertain states identify which consumers need attention. Use the workspace-scoped job and evidence links for diagnostics. When several registrations share one token or fingerprint, choose the registration explicitly. Historical identities and ended management periods retain history without inheriting current verification.

The read-only Agent output log viewer complements the job's signed result and verification evidence. It is not an interactive terminal. Workspace viewers can follow stream status; managers can read the curated output. Claim identifiers remain restricted to workspace admins. Missing or interrupted console output does not prove whether a certificate was deployed; inspect the job result and destination proof before retrying.

• served checks that the configured listeners serve the requested fingerprint.
• trust additionally requires hostname/SNI and CA trust validation.
• Every locally configured listener must pass. Fresh proof for the current binding revision and an active, unexpired identity is required for convergence. Stale evidence remains history.

A completed rollout becomes degraded if an optional consumer's authorization has changed, even when that consumer has a successful historical receipt. Review its current binding before requesting another rollout.

Renewal, drift checks and rollback

The issuer remains the renewal owner. Its configured renewal publishes a new material version and a new fingerprint identity while retaining history. Distribution to destinations is a separate rollout under the workspace approval policy: do not interpret the issuer's successful renewal as a successful rollout everywhere. Keep the issuer's renewal profile and destination verification current.

After transferring an existing source to distribution, configure its distribution renewal workflow. A source with no local key reference cannot be repaired through the ordinary local-agent renewal-setup action. Importing a public PEM or discovering an endpoint also does not create a distribution owner or grant access to its key.

To check drift without installing or reloading, request another authorized rollout of the desired version with verificationOnly: true. It advances generation, compares the installed pair and probes listeners. Drift reports material_drift_detected; prior proof is historical and does not make the destination current. Periodic checks need an explicitly authorized workflow; this workflow does not silently request periodic checks.

Rollback is a new, version-pinned rollout under the workspace approval policy of an older retained material version. Generation still advances. Each deployment keeps the identity of its pinned material version, even when the issuer has renewed since then. Retiring that identity blocks unstarted deployments; results from overlapping execution remain historical and require operator reconciliation. Expired or locally recorded revoked identities cannot be deployed. Confirm CA revocation state and your policy before choosing an older certificate. Retain the Vault object and protected staging needed for recovery; this workflow introduces no automated bundle deletion. Publishing a renewed certificate does not automatically request a consumer rollout; explicitly select and deploy that new version.

Pause and recover safely

Use POST .../distribution-groups/{groupId}/rollouts/{rolloutId}/state with {"state":"paused"} to stop new child work and lease reauthorization. Resume explicitly with {"state":"deploying"} after remediation, or retire with {"state":"retired"}. Required-consumer failure pauses the rollout. An external effect that already occurred may still need reconciliation.

Authorized session managers and admins can pause or retire after workspace pause, freeze or downgrade; these containment actions enqueue no work. Resuming still requires current Pro/Team, freeze and pause admission. Version-list responses never include publishing claim IDs, including for administrators.

Symptom	Action
Lost publication response	Retry the original distribution job. Recovery reuses staged material or the matching deterministic Vault object and the same version. It does not authorize another ACME order.
material_issuance_uncertain	Investigate the CA order, protected staging and remote object. An unresolved version blocks fresh group issuance. Do not delete evidence to force a replacement.
orphaned_unknown_effect	Inspect the real installed pair and all listeners; the service may already have changed. Fix the local problem, then run the reconcile script from Reconcile a binding as the agent account.
Local reconciliation succeeds	Request the original signed retry with a fresh live claim. The local command only clears the local uncertain marker; it does not promote server state.
Agent crashed with a lock	Stop that agent and prove the recorded PID, including possible PID reuse, is no longer executing before removing only the confirmed abandoned lock. Preserve public checkpoint/receipt evidence.

POST .../distribution-jobs/{jobId}/retry requeues the same failed, blocked or uncertain job, preserves its authorized intent/version/generation, and is bounded to three claims. A changed intent needs a new rollout and approval when policy requires it. Rejected or cancelled publication releases an allocation only when the database proves no execution attempt or claim evidence; a generic failure is insufficient.

Reconcile a binding

The installer does not put certops-reconcile-binding on PATH. Run the script from the installed app as the agent account with its normal configuration directory:

sudo -u tokentimer-agent env TOKENTIMER_AGENT_CONFIG_DIR=/opt/tokentimer-agent/state \
  node /opt/tokentimer-agent/app/bin/reconcile-material-binding.js <bindingId>

It takes the binding UUID, reads the protected record under material-bindings in the state directory, and prints a one-line JSON result, or only a short material_* error code with exit status 1.

After local reconciliation, retry the original consumer job and resume its rollout. Only an accepted, fresh receipt for the current authorized deployment clears that job's server reconciliation flag. The local command alone, historical evidence and invalid receipts leave the flag in place.

Protect material-artifacts (including key-bearing bundle.json), material-bindings and material-locks in the agent state directory. Never attach bundles to support tickets. Do not delete generation state or restore an old state directory as a fencing reset. A cleared lock alone does not prove the external effect.

For compromise, pause affected work, restrict credentials, revoke the exact leaf through the issuing CA from customer-controlled infrastructure, and distribute a new key/certificate under the workspace approval policy. Recording inventory revocation or deleting a Vault object does not revoke a certificate at the CA. Verify every concrete listener and HA member before closing the incident.

Vault inventory scans and cleanup

The Vault inventory integration reads public inventory; it is separate from certificate material distribution. Reserved bundles/<UUID> paths are excluded before data reads or nested listing, including direct-prefix probes.

A scan that excludes protected paths is incomplete (protected_material_paths). A denied or failed exact-leaf probe, except a genuine 404, also cannot prove absence. Such scans must not authorize deletion of public inventory in manual cleanup or auto-sync. A legitimate sibling such as bundles/<UUID>-public must be retained.

After upgrading from a build without this correction, discard unconsumed old scan IDs and perform a fresh scan before requesting destructive cleanup. A fresh pin does not retroactively change old scan evidence.