CertOps with ACME and cert-manager

How TokenTimer fits into an ACME flow

TokenTimer splits certificate work across two planes:

• Control plane (your TokenTimer instance): observes, plans, records evidence, and keeps the timeline and inventory. It holds no private keys.
• Execution plane (your clusters and hosts): cert-manager, certbot, acme.sh, or your own agent generate keys locally, build CSRs, run the ACME protocol against the CA (for example Let's Encrypt), deploy, and reload. Keys stay on the hosts they were generated on; TokenTimer never backs them up.

The execution plane is outbound-only: executors call your TokenTimer API over HTTPS. The control plane never opens connections into the execution plane. This holds even when both run in the same cluster.

In practice, with cert-manager driving ACME issuance and renewal:

1. cert-manager creates a key pair inside your cluster, builds a CSR, and completes the ACME challenge with the CA.
2. cert-manager stores the result as a Kubernetes Secret containing tls.crt (public) and tls.key (private).
3. Your reporting integration (renewal hook, controller, or script) tells TokenTimer what happened: events on the job timeline plus evidence such as the observed public certificate fingerprint.
4. TokenTimer's endpoint-check worker independently observes what certificate is actually served at your monitored HTTPS endpoints, so rotations show up in inventory even without executor reports. For this bridge to run, CERTOPS_ENABLED must be set on the worker as well as the API (see Enable CertOps).

note

TokenTimer is an observer and system of record here. It does not run renewals, does not connect to your hosts, and does not push or patch certificates to your endpoints.

How cert-manager Secrets are handled

cert-manager Secrets contain both the public certificate and the private key, so any component reading them is a potential custody path. TokenTimer's design rules for Kubernetes observation:

• Observation prefers Certificate / CertificateRequest status, then annotations, then tls.crt only. tls.key is never read, copied, or transmitted.
• Only public material and status are reported upstream as evidence; key material stays in the cluster.
• Kubernetes RBAC is scoped to the minimum verbs needed. Note the platform limitation: RBAC cannot restrict access to individual data keys inside a Secret, so per-key non-access is enforced by the strict read hierarchy above, by code that never deserializes tls.key, by tests asserting that, and by content scanning of everything reported upstream.

The same content scanning protects your instance on the reporting path: if an integration ever accidentally includes key material in an event or evidence payload, the request is rejected with HTTP 422 PRIVATE_KEY_MATERIAL_REJECTED before any other validation runs and before anything is persisted.

danger

Never write scripts that read a cert-manager Secret wholesale and forward it to TokenTimer. Extract tls.crt (or better, just the fingerprint) explicitly. Whole-Secret payloads contain tls.key and will be rejected.

note

This applies to TokenTimer's own ingress too: if you used the Helm chart's ingress with a cert-manager.io/cluster-issuer annotation to serve TokenTimer itself over TLS, that certificate's Secret is subject to the same rules as any other and its key never enters the TokenTimer database.

Job signing and replay protection, practically

When TokenTimer dispatches work to a TokenTimer agent (as opposed to your own hook scripts reporting inbound), every job is protected so that a captured or forged job cannot be executed:

• Jobs are signed with an Ed25519 platform operational signing key held by the control plane. Agents pin the corresponding public key (signingKeyId) and verify the signature before executing anything. Symmetric HMAC signing is rejected by design, because a shared secret would let any agent forge jobs for others.
• This signing key is a platform operational key only: it is never used for certificate issuance and never holds customer key material. It is the single private key the platform holds.
• Each job envelope carries replay protection: jobId, nonce, and an issuedAt/expiresAt validity window. Agents keep a replay cache keyed by nonce and job ID, and reject jobs outside their time window (clock-drift aware).
• Agent-local policy wins: agents execute only preconfigured, allowlisted command profiles, paths, CA endpoints, and DNS zones/providers. If control-plane intent exceeds local policy, the agent refuses and reports the refusal as evidence. DNS provider API tokens for DNS-01 challenges are provisioned agent-side only; the control plane stores references and allowlists, never the credentials.

What this means for you as an operator: even a fully compromised control plane cannot make your execution plane run arbitrary commands, replay old jobs, or exfiltrate DNS credentials.

Step 1 - Keep issuance in cert-manager

Run ACME issuance exactly as you do today. Nothing about your Issuer/ClusterIssuer or Certificate resources changes for TokenTimer. For ingress TLS, the standard annotation pattern applies:

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  tls:
    - secretName: my-app-tls
      hosts:
        - app.example.com

You should see: cert-manager issuing and renewing as usual, with the certificate served at your endpoint.

Step 2 - Get renewals into TokenTimer

Two complementary paths, use either or both:

Path A - Endpoint monitor (no integration code)

Add your HTTPS endpoint as a monitored endpoint in TokenTimer. The endpoint-check worker observes the public certificate served at the URL and records it into the managed-certificate inventory. When cert-manager rotates the certificate, the next observation sees the new fingerprint: the managed certificate row is updated in place (new fingerprint, serial, expiry) and the previous instance is kept as rotation history.

You should see: the certificate on your Tokens list with observed subject, issuer, serial, fingerprint, and expiry; after a renewal, a new instance row appears under the same managed certificate.

warning

The monitor bridge only runs when the endpoint-check worker also has CERTOPS_ENABLED=true. If observations update the token but never reach CertOps inventory, check the worker's flag first.

Path B - Executor events and evidence (explicit reporting)

Report renewals from a hook with a machine API token (see Connect a CertOps agent for token creation). After a successful renewal, post a completion event and attach evidence:

API_BASE="https://tokentimer.example.com/api/v1"   # your instance's API_URL
TOKEN="ttx_<id>_<secret>"          # certops:events:write + certops:evidence:write
WORKSPACE_ID="<your workspace id>"
JOB_ID="<job id issued for this certificate's renewal>"

# Report completion
curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/events" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"succeeded\",\"eventType\":\"job.completed\",\"occurredAt\":\"$(date -u +%FT%TZ)\",\"message\":\"certificate renewed and reloaded\"}"

# Attach evidence: what was observed after the renewal
curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/evidence" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"accepted\",\"eventType\":\"evidence.attached\",\"occurredAt\":\"$(date -u +%FT%TZ)\",\"evidence\":[{\"schemaVersion\":1,\"evidenceId\":\"$(uuidgen)\",\"eventType\":\"certificate.observed\",\"source\":\"executor\",\"observedAt\":\"$(date -u +%FT%TZ)\",\"summary\":\"Observed public certificate fingerprint after renewal\"}]}"

Evidence item eventType values: certificate.observed, deployment.checked, deployment.updated, validation.passed, validation.failed, policy.checked. The evidence route requires the certops:evidence:write scope; an events-only token is rejected, and a request that includes evidence with an events-only token is rejected too.

You should see: both requests answered with HTTP 202; the evidence response includes populated evidenceIds and a redactionApplied flag.

Evidence redaction, in practice

Redaction and rejection are two different code paths:

• Private keys are rejected, never redacted. Any payload matching known private-key patterns (PEM key blocks, PKCS#8/PKCS#1 headers, base64-wrapped variants) fails with HTTP 422 PRIVATE_KEY_MATERIAL_REJECTED. Nothing is stored.
• Incidental generic secrets (tokens, passwords, API keys) accidentally captured in evidence output are redacted in place to [REDACTED] and the evidence is still stored, with redactionApplied: true in the response. Treat this as a safety net, not a license to pipe raw logs.
• Evidence output is size-limited to 64 KB (checked before redaction). Oversized output is rejected with HTTP 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than silently truncated.

Step 3 - Monitor renewals

1. Job timeline: open /certops/operations in the dashboard. Executor jobs are listed with status badges; selecting a job shows every reported event in order plus attached evidence with redaction markers.
2. Inventory status: managed certificates move through lifecycle statuses (active, renewing, expiring, expired). A certificate that reaches expiring without a subsequent rotation observation is your signal that cert-manager renewal is not landing.
3. Alerts: cert-category tokens participate in normal TokenTimer expiry alerting, so your configured thresholds and channels fire on certificates like on any other tracked asset.
4. Instance history: rotation history per certificate is available in the token detail CertOps panel and via GET /api/v1/workspaces/:id/certops/certificates/:certId/instances (public observation fields only, such as observedAt, observedFingerprintSha256, observedIssuer, observedSerialNumber).

You should see: after each successful renewal, either a fresh observation (Path A), a completed job with evidence (Path B), or both, and the certificate's expiry date moving out.

Failure modes and remediation

Symptom	Likely cause	Fix
All CertOps routes return 404 NOT_FOUND	CERTOPS_ENABLED is unset or false (fail-closed)	Set the flag on backend and worker, restart both (see Enable CertOps)
Observations update tokens but never reach CertOps inventory	Endpoint-check worker's CERTOPS_ENABLED is out of sync with the API's	Set the flag identically on both, restart the worker
Certificate hits expiring in TokenTimer but cert-manager shows no activity	cert-manager renewal is failing cluster-side (ACME challenge failure, issuer misconfiguration, rate limits at the CA)	Debug in the cluster: check Certificate and CertificateRequest status and cert-manager logs; TokenTimer is telling you the renewal has not landed, the root cause is in the execution plane
Renewal succeeded in cluster but TokenTimer still shows the old certificate	Endpoint monitor has not re-observed yet, or the renewed cert is not actually being served (missed reload)	Verify the endpoint serves the new fingerprint (openssl s_client); if it does, wait for the next endpoint-check pass; if not, reload the serving workload
Executor report rejected with 422 PRIVATE_KEY_MATERIAL_REJECTED	Hook script captured key material (for example whole-Secret dump or certbot lineage files) into the payload	Send only fingerprints, paths, or tls.crt-derived public fields; never raw Secret contents
Evidence rejected with 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE	Raw ACME client output exceeds the 64 KB cap	Trim to the relevant tail of the log or a summary before attaching
Evidence stored but marked redactionApplied: true	Command output contained secret-shaped values (bearer tokens, passwords)	Harmless for storage, but filter your logs at the source; redaction is best effort
404 CERTOPS_JOB_NOT_FOUND when reporting	The jobId is unknown; executors cannot create jobs	Create or look up a job on the Certificate operations page (Create manual job) or via GET/POST /api/v1/workspaces/{id}/certops/jobs; then report against that job ID
409 CERTOPS_EXECUTOR_EVENT_CONFLICT on retries	Retry reused an eventId with a modified payload	Retries must be byte-identical; new information needs a new eventId
429 CERTOPS_MACHINE_RATE_LIMITED during renewal bursts	Many hooks sharing one machine token exceeded the shared 120 req/60s bucket across all executor routes	Honor Retry-After and retry (idempotent by eventId), or split integrations across separate tokens

Related

• Connect a CertOps agent - token creation, executor loop, verification.
• Enable CertOps - the CERTOPS_ENABLED flag and fail-closed gating.