Skip to main content
Version: next

Certificate Automation

Overview​

CertOps automation renews certificates end to end on your own instance. The TokenTimer control plane schedules renewal work before certificates expire, routes each job to an eligible executor in your environment, and records issuance, deployment, reload, verification, and evidence on an audited timeline. Approval gates and a workspace kill switch keep humans in control of what runs.

The control plane plans, signs, and audits; execution happens where your certificates live. TokenTimer never connects into your hosts and never touches private keys: keys are generated and used on the agent host (or inside your cluster with cert-manager), and any payload containing key material is rejected outright.

Three execution modes are available and can be mixed freely:

  • TokenTimer agent: an outbound-only daemon on your hosts that verifies signed jobs, runs ACME renewals with DNS-01, deploys atomically, reloads services, and verifies the served certificate. See The TokenTimer agent.
  • cert-manager controller: a Kubernetes controller that observes cert-manager certificates and, in opt-in provisioning mode, creates authorized Certificate resources. See cert-manager mode.
  • External executors: your own renewal scripts, certbot/acme.sh hooks, and CI pipelines authenticate with a machine API token and report against executor jobs. See Connecting external executors.

You manage agents, tokens, and jobs on the Certificate operations page (/certops/operations in the dashboard), reached from the Control Center footer link or from Workspace Preferences when CertOps is enabled. See Certificates for the managed inventory these jobs attach to.

Enabling automation​

Agents, machine tokens, scheduling, and the executor API ship with CertOps. Enable CertOps on your install, then restart the API and worker processes:

CERTOPS_ENABLED=true

When CertOps is disabled, token management, agent routes, and executor endpoints return 404 and the Certificate operations page is not reachable. There is no plan gating on self-hosted installs; all workspaces on your instance can use automation once CertOps is enabled.

For the full list of environment variables, see the Configuration Reference. See also Enable CertOps for the rollout runbook.

Scheduled renewals and approvals​

Renewal work is planned by the control plane, not by cron jobs you have to remember:

  • Renewal scheduler: a maintenance worker sweep creates renewal jobs when a managed certificate approaches expiry. A job is only scheduled when the certificate has a complete, validated renewal profile (see below), so automation never creates unexecutable work.
  • Renewal profiles: a profile binds the operational inputs a renewal needs: the CA endpoint, the command profile (certbot or acme.sh), and the target. The profile snapshot is resolved at schedule time and bound into the signed job payload before approval, so what was approved is exactly what runs.
  • Eligible-agent routing: jobs are routed to agents whose declared capabilities and target selectors match. Agents claim work over an outbound-only protocol with lease renewal. Lease expiry is deliberately conservative: a job whose lease never renewed has proven no side effects and is safely requeued, but a job that was already running is never silently retried. It is held while the agent may still report in, then flagged for manual reconciliation. See Reconciling interrupted jobs.
  • Signed dispatch: every dispatched job is signed with a control-plane Ed25519 key. Agents verify the signature against a pinned key, enforce an issuedAt/expiresAt validity window, and keep a persisted replay cache, so a forged or replayed job never executes.
  • Approval gates: jobs can require explicit approval before dispatch. A requester cannot approve their own job, the approval is bound to the exact job payload hash, and every decision is written to the audit log inside the approval transaction. See Approval gates for the full non-requester and invalidation rules.
  • Per-CA caps and bulk renew: CERTOPS_RENEWAL_PER_CA_CAP bounds in-flight renewals per CA endpoint so the scheduler cannot flood one CA. POST .../certops/jobs/bulk-renew pushes many certificates through the same validated creation path with per-item results.
  • Kill switch: a workspace-level pause blocks new CertOps jobs and dispatch instantly, without deleting queued work or history. Inventory, status, audit, and evidence reporting keep working while paused. The deployment-wide CERTOPS_ENABLED flag remains a separate gate.
  • Renewal failure alerting: failed renewals raise a cert_renewal_failed alert through your normal alerting channels. See Certificate renewal failures for when it fires and how it routes.

The TokenTimer agent​

The agent is a zero-dependency Node.js daemon (Node 22 or later) that runs on the hosts where certificates are deployed. It is outbound-only: it opens HTTPS connections to your TokenTimer instance and never accepts inbound connections. No listening port, no firewall holes.

Installation​

  1. On Certificate operations, open the Deploy an agent panel and create a bootstrap token. It is shown exactly once and consumed at first start.
  2. Copy the generated install command and run it on the target host. The installer (install-agent.sh) verifies the runtime, creates a dedicated system user and install directory, stores the bootstrap token in a 0600-mode file, and installs a hardened systemd unit (ProtectSystem=strict). --dry-run prints every action without touching the system.
  3. Watch the Agent fleet panel flip the agent to registered on its first heartbeat. The fleet panel shows status, heartbeat, clock drift, and compatibility, and lets you retire an agent (a retired agent exits cleanly on its next heartbeat).

If your instance uses a private CA for its own HTTPS endpoint, point the agent at a PEM bundle with the caBundlePath config field (or TOKENTIMER_AGENT_CA_BUNDLE).

For install options, write-path and reload permissions, upgrades, and troubleshooting, see Install and upgrade an agent.

Agent-local policy and permissions​

The agent enforces its own default-deny policy, configured in its local config.json: allowlists for command profiles, filesystem paths, CA endpoints, and DNS zones/providers. Local policy always wins over control-plane intent; a job that exceeds it is rejected and the rejection is reported as evidence. Key-export requests are rejected unconditionally, with no config knob to permit them.

A dry-run mode (execution.dryRun, the default) exercises the full trust chain and reports the steps a job would run without any filesystem or exec side effects, so you can validate a rollout safely before enabling real execution.

What a renewal job does​

Entirely on your host, the agent:

  1. generates (or reuses) the private key and builds the CSR locally; keys are written 0600 and never leave the host,
  2. drives certbot or acme.sh in CSR mode against the allowlisted CA endpoint,
  3. solves DNS-01 challenges with native solvers (see below),
  4. deploys atomically with a timestamped backup and automatic rollback on failure,
  5. validate-then-reloads nginx, Apache, or HAProxy (a failing validation means the reload never runs),
  6. verifies the deployment by fingerprinting the installed certificate, with an optional live TLS probe against the serving endpoint,
  7. reports each step as evidence on the job timeline.

Deployment targets are allowlisted absolute paths on the agent host; the deploy module re-checks the realpath-resolved destination immediately before writing so a symlink cannot escape the allowlisted roots.

DNS-01 providers​

Native DNS-01 solvers ship for 11 providers: Cloudflare, Route 53, Azure DNS, Google Cloud DNS, RFC 2136, acme-dns, OVHcloud, Hetzner, Infomaniak, Exoscale, and PowerDNS. certbot/acme.sh call back into the agent's DNS hook, which polls for propagation before validation and locks records across concurrent challenges.

Provider credentials live in 0600-mode files on the agent host, referenced from the agent's config.json; they are never passed on argv, never sent to the control plane, and a zoneProviderMap routes zones to providers on multi-provider hosts. The control plane stores only references and allowlists.

Per-provider credential fields, propagation tuning, zone routing, and provider-specific caveats are documented in DNS-01 providers.

PKI and private CA integrations​

The ACME adapters work against any ACME-compatible CA endpoint on the agent's allowlist, including internal PKI such as step-ca or vendor ACME gateways. Because the CSR is generated locally and only public material moves, internal PKI workflows keep full key custody. Private CA bundles are supported both for the CA connection (via the ACME tool's own trust configuration) and for the agent-to-control-plane channel (caBundlePath).

cert-manager mode​

For Kubernetes estates, the CertOps cert-manager controller runs in your cluster and is deployed via the TokenTimer Helm chart. It is disabled by default and defaults to observe mode:

  • Observe (read-only RBAC): watches cert-manager Certificate/CertificateRequest resources and reports public-only observations (status first; tls.key is never read). Renewals stay entirely with cert-manager; TokenTimer gets inventory, rotation history, and renewal status.
  • Provision (explicit opt-in, additive RBAC of Certificate create/patch only): the controller creates narrowly authorized, cluster-bound Certificate resources on TokenTimer's instruction, re-authorized before every mutation. cert-manager then issues and manages the key inside the cluster. The controller never writes Secrets and never deletes Kubernetes resources.

The controller is outbound-only, like the agent: your cluster calls TokenTimer; TokenTimer never connects to your Kubernetes API and does not accept uploaded kubeconfigs. See CertOps with ACME and cert-manager.

Connecting external executors​

External tooling remains a fully supported execution mode: anything that can send an HTTPS request can report certificate work against a job. Scheduled renewal jobs are created by TokenTimer automation; you can also create a job manually for break-glass work, hook testing, or work performed entirely by your own tooling:

  1. Have a job to report against: use a scheduled job from the jobs list, or create one as a workspace manager: on Certificate operations, click Create manual job (or call POST /api/v1/workspaces/{id}/certops/jobs). Choose an operation (renew, deploy, reload, revoke, or noop). Optionally set subjectType + subjectId together (for example a managed certificate id). Copy the job ID (shown truncated and copyable on each job row).
  2. Create a machine API token with at least certops:events:write (add certops:evidence:write if the executor uploads evidence). Store the ttx_ value once; it is shown only at creation.
  3. Hand the executor the token, your workspace ID, and the job ID (environment variables or a secret store).
  4. Run the work locally (renew/deploy/reload on your hosts). The executor posts lifecycle events against that jobId.
  5. Review status, timeline, and evidence on Certificate operations. Search the audit log with the job ID if you need the create/report trail.

Machine API tokens​

Machine API tokens are scoped API keys for non-human callers. They authenticate scripts and executors against the CertOps API without a browser session. Each token is bound to one workspace and carries an explicit set of scopes.

  • Format: tokens look like ttx_<id>_<secret>. The ttx_<id> prefix identifies the token in lists; the secret part is never shown again.
  • Hashed at rest: only a SHA-256 hash of the token is stored. The raw value cannot be recovered.
  • Show once: the full token is displayed a single time at creation. Copy it immediately into your secret store.
  • Optional expiry: set an expiry in days at creation. Expired tokens are refused automatically.
  • Last used: the token list shows when each token last authenticated, so you can spot stale or unused credentials.

Create and revoke tokens from the API tokens panel on the Certificate operations page. Revocation is immediate and cannot be undone. Send the token as a bearer credential:

Authorization: Bearer ttx_abc123_...

Token scopes​

Scopes limit what a token can do. Grant only what the executor needs.

  • certops:read read certificate inventory data
  • certops:events:write report executor events for existing jobs (status updates only; creating jobs requires a session as workspace manager, not a machine token)
  • certops:jobs:read read executor jobs and their timelines
  • certops:evidence:write attach evidence to jobs

A typical renewal hook needs certops:events:write and, if it uploads command output, certops:evidence:write. Read scopes are only needed by tools that query state.

Token security​

Treat machine API tokens like passwords.

  • Never share a token between executors or teams. Create one token per integration so you can revoke them independently.
  • Never commit tokens to source control or bake them into images. Inject them from a secret manager or environment variable at runtime.
  • Prefer expiry: set an expiry and rotate tokens on a schedule instead of keeping long-lived credentials.
  • Revoke on suspicion: if a token may have leaked, revoke it immediately and create a replacement. Check the last-used timestamp for unexpected activity.

The full token value is shown only once at creation. If you lose it, revoke the token and create a new one; the raw value cannot be recovered from the stored hash.

Executor jobs​

An executor job is a recorded unit of certificate work: a renewal, a deployment, a service reload, a revocation, or a no-op check. Jobs are created by TokenTimer's renewal scheduler, by a workspace manager, or through the bulk-renew endpoint; executors (agents or your own tooling) perform the work and report against the job. TokenTimer builds a timeline of log entries and evidence for each job.

Where jobs come from​

Jobs enter the system in three ways, and each job records its origin in a source field:

  • Scheduled automation (automation): the renewal scheduler creates renewal jobs when a managed certificate with a validated renewal profile approaches expiry, subject to approval gates and per-CA caps.
  • Manual creation (api): a workspace manager creates a job for break-glass work, hook testing, or work performed by external tooling.
  • System (system): jobs created by TokenTimer's control plane outside the renewal sweep.

Machine-token executors cannot create jobs; they only report against an existing jobId. To create a job manually:

  • Dashboard: Certificate operations → Create manual job. Job IDs are listed there (truncated, copyable).
  • API: POST /api/v1/workspaces/{id}/certops/jobs (session authentication; workspace manager role; source forced to api).

To look up existing jobs for executor hooks, use that page or GET /api/v1/workspaces/{id}/certops/jobs. Posting executor events for an unknown jobId returns 404 CERTOPS_JOB_NOT_FOUND rather than creating it.

The Certificate operations page lists executor jobs for the workspace with status, operation, and timing. Workspace managers can open Create manual job from the jobs panel, including when the list is empty. Selecting a job opens its timeline: every reported event in order, plus attached evidence with redaction markers where secrets were removed.

Jobs move through statuses such as pending, claimed, running, and terminal states succeeded, failed, rejected, blocked, or cancelled. Jobs that require approval start at pending_approval and move to approved before becoming claimable. Transitions are monotonic: a job never moves backward, and a terminal status is final. Events that would regress a job are recorded in the log without changing the job status.

Reporting events​

Executors report work by posting executor events with a machine API token. Each event carries a caller-chosen eventId, a jobId, the workspace ID, a status, an event type, and a timestamp.

  • Event types: job.accepted, job.started, job.progress, job.completed, job.failed, job.rejected, and evidence.attached.
  • Idempotent by eventId: sending the same eventId with the same payload again is a safe no-op, so executors can retry freely. Reusing an eventId with a different payload is rejected as a conflict.
  • Events target an existing job: every event must reference a jobId that already exists (created by a workspace manager or by TokenTimer automation). Machine tokens cannot create jobs. Posting to an unknown jobId is rejected with CERTOPS_JOB_NOT_FOUND.
  • Unknown fields are rejected: extra top-level fields in the request body are rejected rather than silently dropped, so a typo in a field name surfaces immediately instead of being ignored.
  • Inline evidence: an event may include up to 16 evidence items, and public metadata name/value pairs (secret-looking names are refused).

Requests need the certops:events:write scope. Evidence-bearing requests also need certops:evidence:write.

Evidence and redaction​

Evidence entries attach proof of work to a job: command output, ACME logs, deployment confirmations, or certificate fingerprints. Evidence appears on the job timeline in the dashboard.

Evidence output is scanned before storage. Values that look like generic secrets (passwords, API keys, bearer tokens) are replaced with a redaction placeholder, and the entry is marked with redactionApplied so the timeline shows that redaction occurred. Redaction is best effort; do not rely on it as your only safeguard.

Evidence output is capped at 64 KiB before redaction runs; oversized output is rejected outright with CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than truncated. The stored outputSizeBytes and outputSha256 both describe the redacted output that was actually persisted, not your original submission, since redaction can shrink or grow the byte length.

Private keys are different: they are never redacted and stored, they are rejected outright. Any event, evidence, or metadata payload containing private key material (PEM key blocks, key-material field names, PFX or JKS blobs) fails with HTTP 422 and PRIVATE_KEY_MATERIAL_REJECTED. Nothing from the request is stored.

Never pipe raw certbot or openssl output that may include key files into evidence. Filter your logs so only public material and status output is uploaded.

Executor example​

A minimal executor posts one event when work completes. This example reports a successful renewal from a certbot deploy-hook. The jobId must already exist in TokenTimer (create one with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs, or pick one from the jobs list). Look it up beforehand and pass it to the hook. Do not invent a UUID locally; unknown job ids are rejected with 404 CERTOPS_JOB_NOT_FOUND:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/tokentimer-report.sh
# TOKENTIMER_TOKEN comes from your secret store, never hardcode it.
# TOKENTIMER_JOB_ID is an existing job id from Create manual job /
# POST .../certops/jobs, or from the jobs list (GET .../jobs, scope certops:jobs:read).

JOB_ID="${TOKENTIMER_JOB_ID:?set TOKENTIMER_JOB_ID to an existing job id}"
EVENT_ID="$JOB_ID-completed"

curl -s -X POST \
  'https://your-tokentimer.example/api/v1/certops/executor/events' \
  -H "Authorization: Bearer $TOKENTIMER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{
    \"schemaVersion\": 1,
    \"eventId\": \"$EVENT_ID\",
    \"jobId\": \"$JOB_ID\",
    \"workspaceId\": \"WORKSPACE_ID\",
    \"executorId\": \"certbot-$(hostname)\",
    \"status\": \"succeeded\",
    \"eventType\": \"job.completed\",
    \"occurredAt\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"message\": \"certbot renewed $RENEWED_DOMAINS\"
  }"

For a longer-running job, post job.started when work begins and job.completed or job.failed when it ends, reusing the same jobId with distinct eventId values. Retries can resend the exact same request safely. The same pattern works for acme.sh hooks, Ansible handlers, or any script that can send an HTTPS request.

API reference​

Executor endpoints authenticate with Authorization: Bearer ttx_... and are rate limited per token:

  • POST /api/v1/certops/executor/events ingest an executor event for an existing job
  • POST /api/v1/certops/jobs/:jobId/events append an event to an existing job
  • POST /api/v1/certops/jobs/:jobId/evidence attach evidence to a job

Each machine token shares a single rate-limit bucket (120 requests / 60s by default) across all three routes above, not a separate budget per route; exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED. Common error codes: 401 CERTOPS_API_TOKEN_UNAUTHORIZED (bad/revoked/expired token), 403 CERTOPS_API_TOKEN_SCOPE_DENIED (missing scope), 404 CERTOPS_JOB_NOT_FOUND (unknown jobId), 409 CERTOPS_EXECUTOR_EVENT_CONFLICT (same eventId replayed with a different payload), 422 PRIVATE_KEY_MATERIAL_REJECTED, and 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE.

Workspace-scoped endpoints under /api/v1/workspaces/:id/certops serve the dashboard and session-authenticated callers:

  • GET .../jobs list executor jobs
  • POST .../jobs create an executor job manually (workspace manager; body: { operation, subjectType?, subjectId?, payload? }; source is forced to api)
  • GET .../jobs/:jobId job detail
  • GET .../jobs/:jobId/log job timeline entries
  • GET .../jobs/:jobId/evidence evidence attached to the job
  • GET .../tokens list machine API tokens
  • POST .../tokens create a token (raw ttx_ value returned once)
  • POST .../tokens/:tokenId/revoke revoke a token

See Create a manual CertOps job for the create-job request schema, the API reference for authentication, request bodies, and response shapes, and Certificates for the inventory endpoints.