Certificate Lifecycle Management (CLM)

Audience: operators deploying TokenTimer Enterprise who need to enable, license, and network-plan the CLM surface. This is the deployment- and licensing-aware operator view of CLM for Enterprise's Helm/Compose topology. Inventory, issue/renew, the executor API, the agent protocol, and approvals are the same as self-hosted TokenTimer: start with issuing a certificate and the executor API. Env vars are in the self-hosted CLM reference. The rollout flag is in the Enable CLM runbook.

1. What ships, and how it is licensed

CLM is one feature set split into two independently gate-able layers:

Layer	Gate	Covers
Base CLM	Ungated - no license entitlement required, same as self-hosted	Managed certificate inventory, public-PEM import, endpoint-monitor bridge, retire-first lifecycle, machine-token executor API, cert-manager Kubernetes controller, TokenTimer agent protocol, renewal profiles and scheduled auto-renewal, approval workflows, trust-anchor registration/distribution/revocation
Compliance reporting	feature:certops-compliance entitlement	Evidence classification, key-custody / key-locality fields, job change-ticket references, the aggregate compliance report page (/certops/compliance)

This is a deliberate decision, not an oversight: the base CLM surface has never required an explicit license grant, and Enterprise does not introduce a breaking change to that. The only switch on the base surface is the operational CERTOPS_ENABLED environment variable / Helm value, which pauses the operational surface - it is not a licensing control. The workspace settings read/write endpoints stay available while disabled so an operator can inspect or stage the kill switch.

Trust-anchor management is part of this same ungated base surface, inherited unmodified from Core. It requires the workspace admin role rather than workspace_manager - see Authentication, role-based access control.

Zero certificate-private-key custody is a permanent structural invariant across both layers: the TokenTimer control plane never persists, uses, or generates certificate private keys. Key-bearing operations run locally in the customer-controlled agent or Kubernetes execution plane. API payloads containing detected private-key material are rejected with 422 PRIVATE_KEY_MATERIAL_REJECTED after any applicable authentication or workspace-authorization check, without reflecting or persisting that material.

2. Deployment topology

Public CSR workflows

From Enterprise's Core 0.17.0 composition, Certificates → CSR workflows accepts an operator-supplied public CSR, tracks external CA signing, and verifies that the imported signed certificate uses the same public key. This is part of base CLM and does not require the compliance-reporting entitlement.

Follow the public CSR workflow guide for upload/export, issued-name review, and installation confirmation. Both reads and writes require a human session and a workspace manager or administrator. Private keys remain on your host. Manual confirmation records an attestation for an unmonitored target; monitored targets require a matching observation. A rotation preserves the selected certificate identity and deployment history.

CLM adds the following moving parts on top of the base API/dashboard/ worker/PostgreSQL stack:

Component	Compose service	Helm resource	Purpose
API routes	api (existing service)	existing API Deployment	Inventory, executor, agent-protocol, and approval endpoints
Maintenance worker	worker-certops	CronJob (tokentimer.worker.cronjobs.certops, schedule via WORKER_CERTOPS_CRON / tokentimer.worker.cronjobs.certops.schedule, both default */1 * * * *)	Renewal scheduling, agent lease reaping, stale-agent cleanup, outbox draining
cert-manager controller (optional)	n/a (Kubernetes-only)	Deployment + ServiceAccount + RBAC + NetworkPolicy under tokentimer.certops.controller.*	Observes cluster Certificate/CertificateRequest resources into the managed inventory
TokenTimer agent (optional)	n/a (installed on customer hosts, outside the stack)	n/a	Outbound-only discovery and ACME renewal execution on customer infrastructure

Enterprise-specific work on top of the shared mechanics above is limited to: wiring CERTOPS_ENABLED consistently across the api, worker-endpoint-check, and worker-certops services/values, and the standalone compliance-reporting layer described in section 4.

Enabling and disabling

• Compose: set CERTOPS_ENABLED=true (default) or false in .env. It must be set the same way for the api and worker-certops services, or the API will report CLM as available while the worker silently does not sweep (or vice versa).
• Helm: tokentimer.config.certopsEnabled (default true), rendered into CERTOPS_ENABLED on every relevant workload. The cert-manager controller is independently controlled by tokentimer.certops.controller.enabled (default false - it is Kubernetes-specific and not every customer runs a controller workload in the same cluster as TokenTimer).
• Disabling CERTOPS_ENABLED makes operational CLM routes return 404 and stops the maintenance worker's sweep. GET and PUT /api/v1/workspaces/{id}/certops/settings deliberately remain available so authorized operators can inspect and stage workspace settings while the kill switch is active. Disabling the flag never deletes existing inventory, job, evidence, agent, or settings data.

Signing and registration keys (agent protocol)

The agent bootstrap/registration protocol requires two deployment-wide secrets on the API process, both fail-closed if missing or malformed:

Env var	Format	Purpose
CERTOPS_SIGNING_ENCRYPTION_KEY	64 hex chars (32 bytes)	Encrypts the Ed25519 job-signing private key at rest
CERTOPS_REGISTRATION_ENCRYPTION_KEY	64 hex chars (32 bytes)	Encrypts replay-recovery envelopes for interrupted agent registrations

Generate with openssl rand -hex 32 for each. In Helm these are passed through as the tokentimer.config.certopsSigningEncryptionKey / tokentimer.config.certopsRegistrationEncryptionKey values; in Compose set them directly as environment variables on the api service. Preserve the matching signing wrapping key with database backups: replacing it alone makes stored signing private keys unreadable. Operational signing-key rotation does not rotate this wrapping key; replacement requires a separate encrypted-record migration. The registration wrapping key protects only short-lived registration-replay credentials for lost-response recovery. Changing it can invalidate outstanding replay recovery, but existing enrolled-agent credentials are stored as hashes and do not inherently require fleet re-registration.

3. Operator workflow

1. Inventory: import public certificate PEMs (paste or upload) under Certificate operations. Private key material is rejected at upload time. Certificates observed on monitored HTTPS endpoints (the existing endpoint-monitor bridge) feed the inventory automatically.
2. Machine reporting (no agent required): issue a scoped, hashed machine API token under Certificate operations -> Settings, and have your existing renewal scripts, CI jobs, or cert-manager hooks call the executor event API to report lifecycle events and evidence.
3. Agent-based discovery and renewal (optional, deeper automation):
   • Create a time-boxed bootstrap token in the dashboard's agent fleet panel (shown exactly once). Scripting that call against the API with a session cookie needs an X-CSRF-Token header; see CSRF token for scripted API calls. Machine-token executor routes do not.
   • Install the TokenTimer agent on the target host using the generated install command. The agent is outbound-only: it never accepts an inbound connection, and it registers, heartbeats, claims work, and reports results by polling the control plane.
   • Once registered, the agent discovers certificates on its host and reports them into the inventory with agent-provenance metadata (agent id, declared capabilities).
   • Use Set up renewal on an active, unprofiled certificate to adopt it: this immediately queues a renew job and, once that job succeeds, derives a renewal profile (agent, key algorithm, renewal window) from the job's own result rather than accepting an arbitrary hand-authored profile. Detach removes a certificate's link to its renewal profile - the profile itself is left alone since other certificates may still use it - and cancels any outstanding setup intent for that certificate.
4. Trust-anchor management (optional; requires the admin role, see section 1): approve a root/intermediate CA as a workspace trust anchor, then distribute or revoke it on a specific agent's OS trust store (Windows certutil, Debian update-ca-certificates, RHEL update-ca-trust, auto-detected per agent).
   • The distribute/revoke dialogs warn when the selected agent has not declared trust-anchor-deploy-v1. Creating the job against a retired or compatibility-blocked agent is rejected immediately; a live agent that has not advertised the capability yet still waits at Pending until it heartbeats it.
   • A pending_install / pending_remove row that reconciliation gave up on shows Needs attention with a plain-language reason, not the same badge as an active retry.
   • Multiple owners can reference the same anchor on the same agent; revoking one owner's reference only removes the certificate once no other owner references it.
   • TokenTimer never removes a certificate it didn't install itself.
   • Retiring an anchor stops new distributions but doesn't remove it from hosts it's already on.
   • On AlmaLinux/RHEL-family agent hosts, a chmod-related non-zero exit from the OS trust-update command does not always mean the anchor failed to install; see the self-hosted Linux trust-store troubleshooting runbook for the failure categories and recovery path.
5. Approvals (optional): mark a job as requiring approval, or turn on Job approval on the Certificate Settings tab (Require approval before every new job can run, regardless of the per-job setting.; API field certOpsRequireApprovalAlways) so every new job, including scheduled renewals and trust-anchor distribute/revoke, starts at pending_approval. A workspace manager other than the requester must then approve or reject it. Every decision is audit-logged with the actor and reason; the jobs list shows the approver's display name. A job whose side effects could not be confirmed shows a red Needs reconciliation badge.
6. Fleet health: the agent fleet panel shows registration status, last heartbeat, clock-drift state, an Execution column (Enabled or No capability declared; hover explains observe-only or not polled yet), and a Compatibility column (Compatible / Outdated / Blocked) per agent. Blocked means the agent is outside the accepted version window and cannot claim work until upgraded. An agent that stops heartbeating is marked offline by the worker-certops sweep, not instantly - allow a few sweep cycles before treating a status change as unexpected.
7. Retirement: retire an agent from the fleet panel when decommissioning a host. A retire is refused while the agent holds actively leased jobs unless you force it with a reason; a forced retire cancels queued work and flags in-flight work for manual reconciliation, since the host's actual state after a forced retire is not guaranteed.
8. Certificate identity and lifecycle: inventory groups all locations and management sources by workspace SHA-256 fingerprint. Rotation never transfers retirement from A to B. Recording revocation does not contact the CA; decommissioning is blocked by fresh service use and requires acknowledgment for uncertain visibility. Retired certificates still observed remain visible. Stop managing closes one source period without changing lifecycle or evidence; quota is released only after the final period closes. Details retain linked public fields and Notes through rotation.

4. Compliance reporting layer (requires feature:certops-compliance)

On top of the always-available base surface above, licensed workspaces get:

• Evidence classification (internal / confidential / restricted / regulated) and key-custody / key-locality fields on certificate evidence.
• A change-ticket reference field on certificate jobs.
• An aggregate compliance report (the Compliance tab under Certificate operations, or /certops/compliance): evidence breakdown by classification and key custody, plus job change-ticket coverage, per workspace.

This is a read-only aggregate view today; classification, key-custody / key-locality, and change-ticket values are written through the REST API (PATCH /api/v1/workspaces/:id/certops/evidence/:evidenceId/compliance and PATCH /api/v1/workspaces/:id/certops/jobs/:jobId/compliance, workspace manager role required). Every compliance field change is audit-logged in the same transaction as the write. Without the entitlement, the dashboard shows a clear upgrade notice rather than an empty state that could be mistaken for "no certificates yet."

keyLocality accepts only short, non-secret location references: host:, filesystem:, hsm:, vault:, k8s:, or os-store:, followed by an identifier/path made from letters, digits, ., _, /, and -. Examples include host:web-01, hsm:slot-2, vault:secret/pki/example, and k8s:secret/prod/tls-example. Credentials, tokens, URLs containing user-info, credential query strings, PEM data, and free-form text are rejected and never copied into audit metadata.

The rule "Enterprise may store key references and locality evidence, never keys" applies identically to compliance evidence sourced from the agent path and the manual-import path: TokenTimer records where a key lives and who has access, never the key material itself.

5. Network requirements

See Architecture, network requirements for the full list. The rows relevant specifically to CLM:

From	To	Protocol	Purpose
CertOps maintenance worker	PostgreSQL, API	TCP 5432, HTTP/S	Renewal scheduling, lease reaping, stale-agent cleanup, outbox draining
TokenTimer agent (customer host)	API	HTTPS (443), outbound only	Bootstrap/heartbeat/claim/lease/result protocol
TokenTimer agent (customer host)	Configured ACME/CA endpoint	HTTPS	Certificate issuance/renewal
TokenTimer agent (customer host)	Configured DNS-01 provider API	HTTPS	DNS-01 challenge automation (only for allowlisted zones/providers)

No inbound connection to an agent host is ever initiated by TokenTimer. Plan firewall rules for outbound-only egress from each agent host to the API origin, the CA/ACME endpoint you configure, and whichever DNS-01 provider API your zones use - nothing else is required for the agent to function.

6. Roadmap and support boundary

The supported surface is the functionality described in this guide. TokenTimer can prove orchestration and evidence, but cannot recover certificate private keys, debug CA signing policy, or override agent-local policy - those remain customer-controlled by design.