Connect an external CertOps executor

Goal

Connect an external executor (a renewal script, certbot/acme.sh hook, or CI job) to your own TokenTimer instance, so certificate renewals, deployments, and reloads are reported to your workspace and appear on the job timeline.

tip

Looking for the native TokenTimer agent that runs renewals for you (ACME, DNS-01, deployment, verification)? See The TokenTimer agent. This runbook covers the machine-token integration for your own tooling.

Throughout this runbook, replace https://tokentimer.example.com with the public URL of your own TokenTimer API (your API_URL).

Security model in one paragraph

TokenTimer is built on zero private-key custody: the control plane never stores, generates, exports, transmits, or processes private key material. All key-bearing work happens in the execution plane, on hosts you control. Executors are outbound-only: they call your TokenTimer API over HTTPS; the control plane never opens connections to executors. Any payload that contains private key material is rejected outright with HTTP 422 PRIVATE_KEY_MATERIAL_REJECTED before anything is stored.

danger

Never send private keys, key files, or anything derived from key material to TokenTimer. It is always rejected, never redacted and stored. Keys stay on the hosts where they were generated; TokenTimer does not back them up.

Prerequisites

1. A running TokenTimer instance with CertOps enabled (CERTOPS_ENABLED=true on the backend and worker; see Enable CertOps). While the flag is off, every CertOps route returns 404 NOT_FOUND.
2. A workspace manager or admin role, to create machine API tokens.
3. HTTPS in front of your API (secure cookies and bearer credentials should never travel over plain HTTP).
4. Network access from the executor host to your TokenTimer API URL.
5. Your workspace ID.
6. An existing job ID to report against (created in Step 1). Executors cannot create jobs. See also Where jobs come from. Posting to an unknown jobId returns 404 CERTOPS_JOB_NOT_FOUND.

Step 1 - Create a manual job

Executors report against an existing jobId. Create one before you wire the token into a hook.

1. Open Certificate operations at /certops/operations (Control Center footer link, or Workspace Preferences when CertOps is enabled).
2. In the Machine executor jobs panel, click Create manual job (workspace managers and above).
3. Choose an operation (renew, deploy, reload, revoke, or noop). Optionally set subject type and subject ID together (for example managed_certificate and a certificate id from the inventory).
4. Submit. Copy the job ID from the success toast or from the new job row (truncated, click-to-copy).

To create the same job programmatically (session-authenticated workspace API), use Create a manual CertOps job (POST /api/v1/workspaces/{id}/certops/jobs). Example body:

{
  "operation": "renew",
  "subjectType": "managed_certificate",
  "subjectId": "CERT_ID"
}

The response includes job.id and always records source as "api". Prefer the dashboard when you are setting up a single hook; use the API reference page when you need request/response details from an authenticated session client.

You should see: a new row on the jobs list with the operation you chose and a copyable job ID.

Step 2 - Create a machine API token

Machine tokens are scoped API keys for non-human callers (scripts, CI jobs, certbot/ACME hooks).

1. In the dashboard, open the Certificate operations page at /certops/operations. There is no nav entry; reach it from the Control Center certificate-operations panel footer link, or from Workspace Preferences (last section, shown when CertOps is enabled).
2. In the Machine API tokens panel, create a token with a descriptive name and only the scopes the executor needs:

Scope	Grants
certops:events:write	Report job lifecycle events (started, completed, failed)
certops:evidence:write	Attach evidence records to jobs (separate scope; an events-only token cannot attach evidence)
certops:read	Read certificates and jobs (accepted and stored, enforced by a future read API)
certops:jobs:read	Poll job status (same forward-compatible note as above)

3. Copy the token immediately. It has the format ttx_<id>_<secret> and the plaintext is shown exactly once in the create-token response. TokenTimer stores only a SHA-256 hash; it cannot be retrieved or re-displayed later.

You should see: the new token listed in the panel with its ttx_<id> prefix, scopes, and optional expiry.

warning

Store the token in a secret manager immediately. If you lose it, revoke it and create a new one. Create one token per integration so you can revoke them independently. Never commit tokens to source control. Machine-token auth has no session dependency and is not subject to CSRF checks; treat the token itself as the entire credential.

Step 3 - Run the executor

A plain HTTPS loop is enough to be a valid executor; there is no SDK requirement. The executor authenticates with the machine token as a bearer credential:

Authorization: Bearer ttx_<id>_<secret>

A minimal executor reports a start event, does the certificate work out of band, then reports completion:

#!/usr/bin/env bash
# minimal-executor.sh - illustrative only, not a shipped tool
set -euo pipefail

API_BASE="https://tokentimer.example.com/api/v1"   # your instance's API_URL
TOKEN="ttx_<id>_<secret>"          # certops:events:write + certops:evidence:write
WORKSPACE_ID="<your workspace id>"
JOB_ID="$1"                        # from Step 1 (Create manual job); executors do not create jobs

curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/events" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"running\",\"eventType\":\"job.started\",\"occurredAt\":\"$(date -u +%FT%TZ)\"}"

# ... do the certificate work (renew/deploy/reload) out of band ...

curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/events" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"succeeded\",\"eventType\":\"job.completed\",\"occurredAt\":\"$(date -u +%FT%TZ)\"}"

Key rules for event payloads:

• Both status and workspaceId are required top-level fields. workspaceId must match the token's own workspace, or the request is rejected with 403 CERTOPS_EXECUTOR_WORKSPACE_MISMATCH.
• eventType is one of job.accepted, job.started, job.progress, job.completed, job.failed, job.rejected, evidence.attached.
• eventId makes ingestion idempotent per workspace/job/event: replaying the same eventId with the same payload is a safe no-op (HTTP 202, idempotent: true), so retries after network errors are free. Replaying it with a different payload returns 409 CERTOPS_EXECUTOR_EVENT_CONFLICT.
• metadata, when present, is an array of {"name": "...", "value": ...} entries, not a free-form object, and entry names must not look like key/credential fields (secret-looking names are rejected outright).
• Unknown top-level fields are rejected, not silently dropped, so field-name typos surface immediately.
• Job status transitions are monotonic: once a job reaches a terminal status (succeeded, failed, rejected, blocked, cancelled), later events are still accepted and logged but do not reopen the job.

There are two event routes; both work with the same token:

POST /api/v1/certops/executor/events        (jobId in the body)
POST /api/v1/certops/jobs/:jobId/events     (jobId in the URL)

You should see: each request answered with HTTP 202 (Accepted) and a body like:

{
  "ok": true,
  "eventId": "...",
  "logId": "...",
  "jobId": "5f3a1c9e-7d21-4e11-8b2a-9c7d6e5f4a3b",
  "status": "succeeded",
  "evidenceIds": [],
  "duplicate": false,
  "idempotent": false
}

Step 4 - Hook into your renewal tooling

Both certbot's --deploy-hook and acme.sh's --renew-hook run a shell command after a successful renewal. Report success from that hook; report failure from your own error handling around the renewal command (certbot/acme.sh do not invoke deploy hooks on failure).

#!/usr/bin/env bash
# certbot-deploy-hook.sh
# certbot renew --deploy-hook /path/to/certbot-deploy-hook.sh
set -euo pipefail

API_BASE="https://tokentimer.example.com/api/v1"
TOKEN="ttx_<id>_<secret>"
WORKSPACE_ID="<your workspace id>"
JOB_ID="<job id from Create manual job / POST .../certops/jobs>"

curl -sS -X POST "$API_BASE/certops/jobs/$JOB_ID/events" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"schemaVersion\":1,\"eventId\":\"$(uuidgen)\",\"workspaceId\":\"$WORKSPACE_ID\",\"jobId\":\"$JOB_ID\",\"status\":\"succeeded\",\"eventType\":\"job.completed\",\"occurredAt\":\"$(date -u +%FT%TZ)\",\"message\":\"renewed via certbot for $RENEWED_DOMAINS\"}"

danger

Do not include $RENEWED_LINEAGE file contents (certbot's live cert/key directory) in any event or evidence payload. Deploy hooks run with access to the private key on disk, but nothing derived from that key should ever be sent to TokenTimer. Only reference paths or fingerprints, never key material.

Step 5 - Verify connectivity

1. Post a job.started event for a known job (Step 2).
2. Confirm the response is HTTP 202 with "ok": true.
3. In the dashboard, open /certops/operations and select the job. You should see the reported event on the job timeline, with status badges and, where evidence was attached, redaction markers and audit links.
4. Check the Machine API tokens panel: the token's last used timestamp should update, which also confirms which credential the executor actually used.

Rate limits

Every machine token is limited to a default of 120 requests per 60-second window, shared as a single bucket across all three executor routes (/certops/executor/events, /certops/jobs/:jobId/events, /certops/jobs/:jobId/evidence), not a separate budget per route. Exceeding the limit returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED with a Retry-After header (seconds). Because event ingestion is idempotent by eventId, backing off and resending the same request is safe.

Troubleshooting

Symptom	Likely cause	Fix
404 NOT_FOUND on every CertOps route	CERTOPS_ENABLED is unset or false on your instance (fail-closed), or the route path is wrong	Enable the flag on backend and worker and restart (see Enable CertOps); the disabled state is deliberately indistinguishable from a missing route
401 CERTOPS_API_TOKEN_UNAUTHORIZED	Token missing, malformed, unknown, revoked, or expired	Check the Authorization: Bearer ttx_... header; if the token was revoked or expired, create a new one and rotate it in
403 CERTOPS_API_TOKEN_SCOPE_DENIED	Token valid but lacks a required scope	Recreate the token with certops:events:write (and certops:evidence:write if attaching evidence)
403 CERTOPS_EXECUTOR_WORKSPACE_MISMATCH	Body workspaceId/jobId conflicts with the token's bound workspace or the URL path parameter	Use the workspace ID the token was created in; keep body and URL jobId consistent
404 CERTOPS_JOB_NOT_FOUND	The jobId does not reference a job TokenTimer already knows about	Executors cannot create jobs; look up or create a job on the Certificate operations page (job IDs are listed there) or via GET/POST /api/v1/workspaces/{id}/certops/jobs (manager role for create)
409 CERTOPS_EXECUTOR_EVENT_CONFLICT	Same eventId replayed with a different payload	Generate a fresh eventId per distinct event; only exact retries may reuse one
422 PRIVATE_KEY_MATERIAL_REJECTED	Payload matched a private-key pattern (PEM key block, PKCS#8/PKCS#1, base64-wrapped)	Remove all key material from the payload; send fingerprints or paths instead
413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE	Evidence output exceeds the 64 KB pre-redaction cap	Trim or summarize command output before attaching it
429 CERTOPS_MACHINE_RATE_LIMITED	Token exceeded its shared 120 req/60s bucket	Honor the Retry-After header and retry; idempotent eventId makes the retry safe
Executor cannot reach the API at all	Network path or TLS termination issue between executor host and your API_URL	Test with curl https://tokentimer.example.com/health from the executor host

All error bodies use the flat envelope { "error": "<human-readable message>", "code": "<ERROR_CODE>" } (an optional details field may carry validation specifics), and never echo tokens, key material, or raw secrets back.

Next steps

• CertOps with ACME and cert-manager - drive issuance with cert-manager and report renewals as evidence.