Skip to main content
Version: 0.11

Troubleshoot local policy and verification failures

Goal​

Diagnose a job that fails or is rejected on the agent host even though everything upstream looked fine: it was created, matched to an online agent, signed, and dispatched. This covers the two places that failure most often comes from - the agent's own local policy boundary, and the verify step it runs after a reload.

Why an agent can reject a job the control plane already accepted​

Routing a job to an agent and authorizing what that agent may do are two separate decisions. The control plane matches a job to an agent by declared target selectors, DNS providers, and command profiles, then signs and dispatches it. What the agent actually does with that job is governed entirely by its own policy block in config.json, on the host, which the control plane cannot see into and cannot override. See Policy (default-deny) for the full field reference; this page is about what happens when a job hits that boundary.

This is deliberate, not a bug: it is the agent's authorization boundary, and it is meant to win even when the control plane's own matching was permissive. A job that exceeds it is rejected locally and reported back as evidence, never silently dropped and never executed anyway.

rejectionReasonWhat it meansWhere to fix it
command_not_allowlistedThe job's commandRef does not match any profile in policy.allowedCommandsAdd the profile to policy.allowedCommands
path_not_allowlistedcertPath or chainPath is outside policy.allowedPathsAdd the path to policy.allowedPaths, and confirm the systemd sandbox and filesystem permissions also allow it - see the --write-path warnings in the install runbook, since this is a separate gate from policy
ca_endpoint_not_allowlistedcaEndpoint is not in policy.allowedCaEndpointsAdd the endpoint to policy.allowedCaEndpoints
dns_zone_not_allowlisteddnsZone is not covered by policy.allowedDnsZonesAdd the zone to policy.allowedDnsZones
dns_provider_not_allowlisteddnsProvider is not in policy.allowedDnsProvidersAdd the provider id to policy.allowedDnsProviders; see DNS-01 providers for valid ids
target_out_of_scopeThe job's target, or a verifyHost, is outside what this agent is authorized to touchSee The verify step below if it is a verify destination; otherwise check declaredTargetSelectors
key_export_requestedThe job asked the agent to export private key materialNot fixable. Key export is rejected unconditionally - there is no policy setting that permits it
Two similarly-named things, two different lifecycles

policy.allowedCommands (and the other policy.* allowlists) are read fresh from config.json on every job, so editing the file and restarting the agent is enough to widen them. declaredCommandProfileNames and declaredTargetSelectors are different: they are sent to the control plane only once, at registration, and a config edit does not update what the control plane has on file. If a job never routes to an agent at all (rather than routing and then being rejected), you likely need to fix the declared side, not the policy side - see Capability declaration.

Step 1 - Find the exact rejection reason​

Open the job on the Jobs tab of Certificate operations. A locally-rejected job shows status rejected, and its timeline carries a policy-evidence entry with the rejectionReason from the table above plus a free-text detail string naming exactly what did not match (for example, the specific commandRef or path the agent compared against its allowlist).

If you have host access, the agent's own log names the same reason at the moment of rejection:

journalctl -u tokentimer-agent -n 100 --no-pager | grep -i rejected

Step 2 - Widen the policy, then re-run​

  1. Edit /opt/tokentimer-agent/state/config.json and add the missing value to the relevant policy.* list.
  2. Restart the agent: sudo systemctl restart tokentimer-agent.
  3. Re-run the job (or wait for the next scheduled attempt, if it still has attempts left).

No new bootstrap token or re-registration is needed for a policy.* change - only declaredCommandProfileNames/declaredTargetSelectors require that, per the tip above.

The verify step​

After a successful deploy and reload, the agent can optionally open a TLS connection to the live endpoint and confirm it is actually serving the certificate it just deployed, by comparing the SHA-256 fingerprint. This only runs when the job payload sets verifyHost (optionally with verifyPort); a job with no verifyHost skips it entirely and relies on the deploy-time fingerprint read-back alone.

Configuring allowedVerifyHosts​

Verifying the domain the job is actually renewing needs no extra configuration - the agent always allows a verify destination that matches the job's own certificate target. You only need policy.allowedVerifyHosts when verifyHost points somewhere else: a load balancer VIP in front of several backends, an internal alias, or localhost/127.0.0.1 while you are validating a new agent by hand.

Two destination classes behave specially, and no configuration can change either:

  • Loopback (127.0.0.1, ::1, localhost) is allowed only with an explicit entry in policy.allowedVerifyHosts, even when it happens to match the job's target.
  • Link-local and metadata-endpoint-class addresses (including 169.254.169.254) are hard-denied. They can never be probed, allowlisted or not.

A hostname entry in policy.allowedVerifyHosts also covers its subdomains at a dot boundary; an IP entry only ever matches exactly.

{
  "policy": {
    "allowedVerifyHosts": ["localhost", "127.0.0.1", "lb.internal.example.com"]
  }
}

"Verify step failed: live endpoint does not serve the deployed certificate"​

Two different causes produce this message, and they need different fixes:

  1. Policy rejection. If the job's evidence shows rejectionReason: target_out_of_scope for the verify step, this is local policy - verifyHost is not covered by policy.allowedVerifyHosts and does not match the job's own target. Fix it as in Configuring allowedVerifyHosts above.
  2. A genuine mismatch was read. The agent actually connected and the certificate it saw did not match. This can be a real deployment problem, but immediately after a reload it can also be transient: a reload can be accepted by the service manager before every worker process has finished switching to the new certificate. If you see this only in the first moment after a reload, and a manual check a few seconds later shows the correct certificate, treat it as transient and re-run the job. If it persists or recurs on every renewal for the same host, treat it as a real deployment problem:
# Compare what the agent should have deployed against what the endpoint actually serves
openssl x509 -in /path/to/deployed/fullchain.pem -noout -fingerprint -sha256
openssl s_client -connect <verifyHost>:<verifyPort> -servername <verifyHost> </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

If the two fingerprints genuinely differ and stay different on a retry, check that the reload target you configured (reloadService, or the reloadCommandRefs on a manual job) is the same process actually serving verifyHost - a reverse proxy or a second listener not covered by the reload is the most common real cause.