Certificate operations (CertOps)

Audience: operators deploying TokenTimer Enterprise who need to enable, license, and network-plan the CertOps surface. This is the deployment- and licensing-aware operator view of CertOps for Enterprise's Helm/Compose topology; the underlying feature mechanics (inventory, agent protocol, renewal profiles, sweeps) are shared with the self-hosted product and documented in full in the self-hosted CertOps reference and the Enable CertOps runbook.

1. What ships, and how it is licensed

CertOps is one feature set split into two independently gate-able layers:

Layer	Gate	Covers
Base CertOps	Ungated - no license entitlement required, same as self-hosted/Cloud	Managed certificate inventory, public-PEM import, endpoint-monitor bridge, retire-first lifecycle, machine-token executor API, cert-manager Kubernetes controller, TokenTimer agent protocol, renewal profiles and scheduled auto-renewal, approval workflows, trust-anchor registration/distribution/revocation
Compliance reporting	feature:certops-compliance entitlement	Evidence classification, key-custody / key-locality fields, job change-ticket references, the aggregate compliance report page (/certops/compliance)

This is a deliberate decision, not an oversight: the base CertOps surface has never required an explicit license grant, and Enterprise does not introduce a breaking change to that. The only switch on the base surface is the operational CERTOPS_ENABLED environment variable / Helm value, which pauses the operational surface - it is not a licensing control. The workspace settings read/write endpoints stay available while disabled so an operator can inspect or stage the kill switch.

Trust-anchor management is part of this same ungated base surface, inherited unmodified from Core. It requires the workspace admin role rather than workspace_manager - see Authentication, role-based access control.

Zero certificate-private-key custody is a permanent structural invariant across both layers: the TokenTimer control plane never persists, uses, or generates certificate private keys. Key-bearing operations run locally in the customer-controlled agent or Kubernetes execution plane. API payloads containing detected private-key material are rejected with 422 PRIVATE_KEY_MATERIAL_REJECTED after any applicable authentication or workspace-authorization check, without reflecting or persisting that material.

2. Deployment topology

CertOps adds the following moving parts on top of the base API/dashboard/ worker/PostgreSQL stack:

Component	Compose service	Helm resource	Purpose
API routes	api (existing service)	existing API Deployment	Inventory, executor, agent-protocol, and approval endpoints
Maintenance worker	worker-certops	CronJob (tokentimer.worker.cronjobs.certops, schedule via WORKER_CERTOPS_CRON / tokentimer.worker.cronjobs.certops.schedule, both default */1 * * * *)	Renewal scheduling, agent lease reaping, stale-agent cleanup, outbox draining
cert-manager controller (optional)	n/a (Kubernetes-only)	Deployment + ServiceAccount + RBAC + NetworkPolicy under tokentimer.certops.controller.*	Observes cluster Certificate/CertificateRequest resources into the managed inventory
TokenTimer agent (optional)	n/a (installed on customer hosts, outside the stack)	n/a	Outbound-only discovery and ACME renewal execution on customer infrastructure

Enterprise-specific work on top of the shared mechanics above is limited to: wiring CERTOPS_ENABLED consistently across the api, worker-endpoint-check, and worker-certops services/values, and the standalone compliance-reporting layer described in section 4.

Enabling and disabling

• Compose: set CERTOPS_ENABLED=true (default) or false in .env. It must be set the same way for the api and worker-certops services, or the API will report CertOps as available while the worker silently does not sweep (or vice versa).
• Helm: tokentimer.config.certopsEnabled (default true), rendered into CERTOPS_ENABLED on every relevant workload. The cert-manager controller is independently controlled by tokentimer.certops.controller.enabled (default false - it is Kubernetes-specific and not every customer runs a controller workload in the same cluster as TokenTimer).
• Disabling CERTOPS_ENABLED makes operational CertOps routes return 404 and stops the maintenance worker's sweep. GET and PUT /api/v1/workspaces/{id}/certops/settings deliberately remain available so authorized operators can inspect and stage workspace settings while the kill switch is active. Disabling the flag never deletes existing inventory, job, evidence, agent, or settings data.

Signing and registration keys (agent protocol)

The agent bootstrap/registration protocol requires two deployment-wide secrets on the API process, both fail-closed if missing or malformed:

Env var	Format	Purpose
CERTOPS_SIGNING_ENCRYPTION_KEY	64 hex chars (32 bytes)	Encrypts the Ed25519 job-signing private key at rest
CERTOPS_REGISTRATION_ENCRYPTION_KEY	64 hex chars (32 bytes)	Encrypts replay-recovery envelopes for interrupted agent registrations

Generate with openssl rand -hex 32 for each. In Helm these are passed through as the tokentimer.config.certopsSigningEncryptionKey / tokentimer.config.certopsRegistrationEncryptionKey values; in Compose set them directly as environment variables on the api service. Losing either key after agents are registered requires re-registering the fleet; there is no recovery path for a lost encryption key by design (it protects the corresponding secret material, so a recoverable key would defeat the point).

3. Operator workflow

1. Inventory: import public certificate PEMs (paste or upload) under Certificate operations. Private key material is rejected at upload time. Certificates observed on monitored HTTPS endpoints (the existing endpoint-monitor bridge) feed the inventory automatically.
2. Machine reporting (no agent required): issue a scoped, hashed machine API token under Certificate operations -> Settings, and have your existing renewal scripts, CI jobs, or cert-manager hooks call the executor event API to report lifecycle events and evidence.
3. Agent-based discovery and renewal (optional, deeper automation):
   • Create a time-boxed bootstrap token in the dashboard's agent fleet panel (shown exactly once).
   • Install the TokenTimer agent on the target host using the generated install command. The agent is outbound-only: it never accepts an inbound connection, and it registers, heartbeats, claims work, and reports results by polling the control plane.
   • Once registered, the agent discovers certificates on its host and reports them into the inventory with agent-provenance metadata (agent id, declared capabilities).
   • Use Set up automatic renewal on an active, unprofiled certificate to adopt it: this immediately queues a renew job and, once that job succeeds, derives a renewal profile (agent, key algorithm, renewal window) from the job's own result rather than accepting an arbitrary hand-authored profile. Detach removes a certificate's link to its renewal profile - the profile itself is left alone since other certificates may still use it - and cancels any outstanding setup intent for that certificate.
4. Trust-anchor management (optional; requires the admin role, see section 1): approve a root/intermediate CA as a workspace trust anchor, then distribute or revoke it on a specific agent's OS trust store (Windows certutil, Debian update-ca-certificates, RHEL update-ca-trust, auto-detected per agent).
   • The distribute/revoke dialogs warn when the selected agent has not declared trust-anchor-deploy-v1. Creating the job against a retired or compatibility-blocked agent is rejected immediately; a live agent that has not advertised the capability yet still waits at Pending until it heartbeats it.
   • A pending_install / pending_remove row that reconciliation gave up on shows Needs attention with a plain-language reason, not the same badge as an active retry.
   • Multiple owners can reference the same anchor on the same agent; revoking one owner's reference only removes the certificate once no other owner references it.
   • TokenTimer never removes a certificate it didn't install itself.
   • Retiring an anchor stops new distributions but doesn't remove it from hosts it's already on.
   • On AlmaLinux/RHEL-family agent hosts, a chmod-related non-zero exit from the OS trust-update command does not always mean the anchor failed to install; see the self-hosted Linux trust-store troubleshooting runbook for the failure categories and recovery path.
5. Approvals (optional): mark a job as requiring approval, or turn on Job approval on the CertOps Settings tab (Require approval before every new job can run, regardless of the per-job setting.; API field certOpsRequireApprovalAlways) so every new job, including scheduled renewals and trust-anchor distribute/revoke, starts at pending_approval. A workspace manager other than the requester must then approve or reject it. Every decision is audit-logged with the actor and reason; the jobs list shows the approver's display name. A job whose side effects could not be confirmed shows a red Needs reconciliation badge.
6. Fleet health: the agent fleet panel shows registration status, last heartbeat, clock-drift state, an Execution column (Enabled or No capability declared; hover explains observe-only or not polled yet), and a Compatibility column (Compatible / Outdated / Blocked) per agent. Blocked means the agent is outside the accepted version window and cannot claim work until upgraded. An agent that stops heartbeating is marked offline by the worker-certops sweep, not instantly - allow a few sweep cycles before treating a status change as unexpected.
7. Retirement: retire an agent from the fleet panel when decommissioning a host. A retire is refused while the agent holds actively leased jobs unless you force it with a reason; a forced retire cancels queued work and flags in-flight work for manual reconciliation, since the host's actual state after a forced retire is not guaranteed.

4. Compliance reporting layer (requires feature:certops-compliance)

On top of the always-available base surface above, licensed workspaces get:

• Evidence classification (internal / confidential / restricted / regulated) and key-custody / key-locality fields on certificate evidence.
• A change-ticket reference field on certificate jobs.
• An aggregate compliance report (Workspace preferences -> Certificate operations -> Compliance report, or /certops/compliance): evidence breakdown by classification and key custody, plus job change-ticket coverage, per workspace.

This is a read-only aggregate view today; classification, key-custody / key-locality, and change-ticket values are written through the REST API (PATCH /api/v1/workspaces/:id/certops/evidence/:evidenceId/compliance and PATCH /api/v1/workspaces/:id/certops/jobs/:jobId/compliance, workspace manager role required). Every compliance field change is audit-logged in the same transaction as the write. Without the entitlement, the dashboard shows a clear upgrade notice rather than an empty state that could be mistaken for "no certificates yet."

keyLocality accepts only short, non-secret location references: host:, filesystem:, hsm:, vault:, k8s:, or os-store:, followed by an identifier/path made from letters, digits, ., _, /, and -. Examples include host:web-01, hsm:slot-2, vault:secret/pki/example, and k8s:secret/prod/tls-example. Credentials, tokens, URLs containing user-info, credential query strings, PEM data, and free-form text are rejected and never copied into audit metadata.

The rule "Enterprise may store key references and locality evidence, never keys" applies identically to compliance evidence sourced from the agent path and the manual-import path: TokenTimer records where a key lives and who has access, never the key material itself.

5. Network requirements

See Architecture, network requirements for the full list. The rows relevant specifically to CertOps:

From	To	Protocol	Purpose
CertOps maintenance worker	PostgreSQL, API	TCP 5432, HTTP/S	Renewal scheduling, lease reaping, stale-agent cleanup, outbox draining
TokenTimer agent (customer host)	API	HTTPS (443), outbound only	Bootstrap/heartbeat/claim/lease/result protocol
TokenTimer agent (customer host)	Configured ACME/CA endpoint	HTTPS	Certificate issuance/renewal
TokenTimer agent (customer host)	Configured DNS-01 provider API	HTTPS	DNS-01 challenge automation (only for allowlisted zones/providers)

No inbound connection to an agent host is ever initiated by TokenTimer. Plan firewall rules for outbound-only egress from each agent host to the API origin, the CA/ACME endpoint you configure, and whichever DNS-01 provider API your zones use - nothing else is required for the agent to function.

6. Roadmap and support boundary

The supported surface is the functionality described in this guide. TokenTimer can prove orchestration and evidence, but cannot recover certificate private keys, debug CA signing policy, or override agent-local policy - those remain customer-controlled by design.