CertOps with ACME and cert-manager

How TokenTimer fits into an ACME flow

TokenTimer splits certificate work across two planes:

• Control plane (TokenTimer Cloud): observes, plans, records evidence, and keeps the timeline and inventory. It holds no private keys.
• Execution plane (your cluster and hosts): cert-manager, certbot, acme.sh, or your own agent generate keys locally, build CSRs, run the ACME protocol against the CA (for example Let's Encrypt), deploy, and reload. Keys stay on the hosts they were generated on; TokenTimer never backs them up.

The execution plane is outbound-only: your side calls TokenTimer over HTTPS. The control plane never opens connections into your cluster.

In practice, with cert-manager driving ACME issuance and renewal:

1. cert-manager creates a key pair inside your cluster, builds a CSR, and completes the ACME challenge with the CA.
2. cert-manager stores the result as a Kubernetes Secret containing tls.crt (public) and tls.key (private).
3. Your reporting integration (renewal hook, controller, or script) tells TokenTimer what happened: events on the job timeline plus evidence such as the observed public certificate fingerprint.
4. TokenTimer's endpoint monitors independently observe what certificate is actually served at your HTTPS endpoints, so rotations show up in inventory even without executor reports.

note

In this flow, cert-manager keeps driving issuance and renewal; TokenTimer observes, records, and alerts. The control plane never connects into your cluster and never pushes or patches certificates itself. If you want TokenTimer to schedule and execute renewals too, deploy the TokenTimer agent on your hosts or enable the controller's opt-in provisioning mode.

How cert-manager Secrets are handled

cert-manager Secrets contain both the public certificate and the private key, so any component reading them is a potential custody path. TokenTimer's design rules for Kubernetes observation:

• Observation prefers Certificate / CertificateRequest status, then annotations, then tls.crt only. tls.key is never read, copied, or transmitted.
• Only public material and status are reported upstream as evidence; key material stays in the cluster.
• Kubernetes RBAC is scoped to the minimum verbs needed. Note the platform limitation: RBAC cannot restrict access to individual data keys inside a Secret, so per-key non-access is enforced by the strict read hierarchy above, by code that never deserializes tls.key, by tests asserting that, and by content scanning of everything reported upstream.

The same content scanning protects you on the reporting path: if your integration ever accidentally includes key material in an event or evidence payload, the request is rejected with HTTP 422 PRIVATE_KEY_MATERIAL_REJECTED before any other validation runs and before anything is persisted.

danger

Never write scripts that read a cert-manager Secret wholesale and forward it to TokenTimer. Extract tls.crt (or better, just the fingerprint) explicitly. Whole-Secret payloads contain tls.key and will be rejected.

Job signing and replay protection, practically

When TokenTimer dispatches work to a TokenTimer agent (as opposed to your own hook scripts reporting inbound), every job is protected so that a captured or forged job cannot be executed:

• Jobs are signed with an Ed25519 platform operational signing key held by the control plane. Agents pin the corresponding public key (signingKeyId) and verify the signature before executing anything. Symmetric HMAC signing is rejected by design, because a shared secret would let any agent forge jobs for others.
• This signing key is a platform operational key only: it is never used for certificate issuance and never holds customer key material.
• Each job envelope carries replay protection: jobId, nonce, and an issuedAt/expiresAt validity window. Agents keep a replay cache keyed by nonce and job ID, and reject jobs outside their time window (clock-drift aware).
• Agent-local policy wins: agents execute only preconfigured, allowlisted command profiles, paths, CA endpoints, and DNS zones/providers. If control-plane intent exceeds local policy, the agent refuses and reports the refusal as evidence. DNS provider API tokens for DNS-01 challenges are provisioned agent-side only; the control plane stores references and allowlists, never the credentials.

What this means for you as an operator: even a fully compromised control plane cannot make your execution plane run arbitrary commands, replay old jobs, or exfiltrate DNS credentials.

Step 1 - Keep issuance in cert-manager

Run ACME issuance exactly as you do today. Nothing about your Issuer/ClusterIssuer or Certificate resources changes for TokenTimer. For ingress TLS, the standard annotation pattern applies:

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  tls:
    - secretName: my-app-tls
      hosts:
        - app.example.com

You should see: cert-manager issuing and renewing as usual, with the certificate served at your endpoint.

Step 2 - Get renewals into TokenTimer

Two complementary paths, use either or both:

Path A - Endpoint monitor (no integration code)

Add your HTTPS endpoint as a monitored endpoint in TokenTimer. The worker observes the public certificate served at the URL and records it into the managed-certificate inventory. When cert-manager rotates the certificate, the next observation sees the new fingerprint: the managed certificate row is updated in place (new fingerprint, serial, expiry) and the previous instance is kept as rotation history.

You should see: the certificate on your Tokens list with observed subject, issuer, serial, fingerprint, and expiry; after a renewal, a new instance row appears under the same managed certificate.

Path B - Executor events and evidence (explicit reporting)

Report renewals from a hook with a machine API token (see Connect an external executor for token creation). After a successful renewal, post a completion event and attach evidence:

API_BASE="https://tokentimer.ch/api/v1"
TOKEN="ttx_<id>_<secret>"          # certops:events:write + certops:evidence:write
WORKSPACE_ID="<your workspace id>"
JOB_ID="<job id issued for this certificate's renewal>"

# Report completion
curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/events" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"succeeded\",\"eventType\":\"job.completed\",\"occurredAt\":\"$(date -u +%FT%TZ)\",\"message\":\"certificate renewed and reloaded\"}"

# Attach evidence: what was observed after the renewal
curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/evidence" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"accepted\",\"eventType\":\"evidence.attached\",\"occurredAt\":\"$(date -u +%FT%TZ)\",\"evidence\":[{\"schemaVersion\":1,\"evidenceId\":\"$(uuidgen)\",\"eventType\":\"certificate.observed\",\"source\":\"executor\",\"observedAt\":\"$(date -u +%FT%TZ)\",\"summary\":\"Observed public certificate fingerprint after renewal\"}]}"

Evidence item eventType values: certificate.observed, deployment.checked, deployment.updated, validation.passed, validation.failed, policy.checked. The evidence route requires the certops:evidence:write scope; an events-only token is rejected, and a request that includes evidence with an events-only token is rejected too.

You should see: both requests answered with HTTP 202; the evidence response includes populated evidenceIds and a redactionApplied flag.

Evidence redaction, in practice

Redaction and rejection are two different code paths:

• Private keys are rejected, never redacted. Any payload matching known private-key patterns (PEM key blocks, PKCS#8/PKCS#1 headers, base64-wrapped variants) fails with HTTP 422 PRIVATE_KEY_MATERIAL_REJECTED. Nothing is stored.
• Incidental generic secrets (tokens, passwords, API keys) accidentally captured in evidence output are redacted in place to [REDACTED] and the evidence is still stored, with redactionApplied: true in the response. Treat this as a safety net, not a license to pipe raw logs.
• Evidence output is size-limited to 64 KB (checked before redaction). Oversized output is rejected with HTTP 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than silently truncated.

Step 3 - Monitor renewals

1. Job timeline: open /certops/operations in the dashboard. Executor jobs are listed with status badges; selecting a job shows every reported event in order plus attached evidence with redaction markers.
2. Inventory status: managed certificates move through lifecycle statuses (active, renewing, expiring, expired). A certificate that reaches expiring without a subsequent rotation observation is your signal that cert-manager renewal is not landing.
3. Alerts: cert-category tokens participate in normal TokenTimer expiry alerting, so your configured thresholds and channels fire on certificates like on any other tracked asset.
4. Instance history: rotation history per certificate is available in the token detail CertOps panel and via GET /api/v1/workspaces/:id/certops/certificates/:certId/instances (public observation fields only, such as observedAt, observedFingerprintSha256, observedIssuer, observedSerialNumber).

You should see: after each successful renewal, either a fresh observation (Path A), a completed job with evidence (Path B), or both, and the certificate's expiry date moving out.

Failure modes and remediation

Symptom	Likely cause	Fix
Certificate hits expiring in TokenTimer but cert-manager shows no activity	cert-manager renewal is failing cluster-side (ACME challenge failure, issuer misconfiguration, rate limits at the CA)	Debug in the cluster: check Certificate and CertificateRequest status and cert-manager logs; TokenTimer is telling you the renewal has not landed, the root cause is in the execution plane
Renewal succeeded in cluster but TokenTimer still shows the old certificate	Endpoint monitor has not re-observed yet, or the renewed cert is not actually being served (missed reload)	Verify the endpoint serves the new fingerprint (openssl s_client); if it does, wait for the next monitor pass; if not, reload the serving workload
Executor report rejected with 422 PRIVATE_KEY_MATERIAL_REJECTED	Hook script captured key material (for example whole-Secret dump or certbot lineage files) into the payload	Send only fingerprints, paths, or tls.crt-derived public fields; never raw Secret contents
Evidence rejected with 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE	Raw ACME client output exceeds the 64 KB cap	Trim to the relevant tail of the log or a summary before attaching
Evidence stored but marked redactionApplied: true	Command output contained secret-shaped values (bearer tokens, passwords)	Harmless for storage, but filter your logs at the source; redaction is best effort
404 CERTOPS_JOB_NOT_FOUND when reporting	The jobId is unknown; executors cannot create jobs	Create or look up a job on the Certificate operations page (Create manual job) or via GET/POST /api/v1/workspaces/{id}/certops/jobs; then report against that job ID
409 CERTOPS_EXECUTOR_EVENT_CONFLICT on retries	Retry reused an eventId with a modified payload	Retries must be byte-identical; new information needs a new eventId
429 CERTOPS_MACHINE_RATE_LIMITED during renewal bursts	Many hooks sharing one machine token exceeded the shared 120 req/60s bucket across all executor routes	Honor Retry-After and retry (idempotent by eventId), or split integrations across separate tokens
All CertOps calls return 404 NOT_FOUND	CERTOPS_ENABLED is off platform-wide (fail-closed), not a per-workspace state	The disabled state is deliberately indistinguishable from a missing route; contact support to check platform availability
Executor calls return 402 PLAN_FEATURE_REQUIRED	Workspace is on the Free plan	Upgrade to Pro or Team; endpoint SSL monitoring itself stays free

Related

• Connect an external CertOps executor to TokenTimer - token creation, executor loop, verification.