Monitoring & metrics

Self-hosted TokenTimer exposes Prometheus metrics from the API and pushes worker metrics to an optional Pushgateway. Metrics are disabled by default.

Enabling metrics

Variable	Description	Default	Scope
ENABLE_METRICS	Enable /metrics and metric push attempts when true	false	API, worker
PUSHGATEWAY_URL	Prometheus Pushgateway endpoint for worker metrics	unset	Worker
ENVIRONMENT_SUFFIX	Metrics environment label (env) when running multiple instances	unset	Worker

With ENABLE_METRICS=true, the API serves Prometheus text format at GET /metrics on the API port (default 4000). The endpoint has no authentication, so restrict access at the network layer (NetworkPolicy, firewall, or scrape from inside the cluster only).

Worker runs are short-lived (one job per scheduled run, whether launched by a Kubernetes CronJob or by the long-running Compose runner), so workers push their metrics to PUSHGATEWAY_URL at the end of each run instead of exposing a scrape endpoint. Schedules are described in Configuration → Worker scheduling.

API metrics

Metric	Type	Labels	Meaning
tokentimer_api_http_request_duration_seconds	Histogram	method, route, status	HTTP request duration
tokentimer_api_http_requests_total	Counter	method, route, status	Total HTTP requests
tokentimer_api_login_attempts_total	Counter	outcome	Login attempts
tokentimer_api_2fa_verifications_total	Counter	outcome	2FA verification attempts
tokentimer_api_rate_limit_hits_total	Counter		HTTP 429 responses
tokentimer_api_csrf_rejections_total	Counter		Requests rejected by CSRF protection
tokentimer_api_rbac_denied_total	Counter		RBAC authorization denials
tokentimer_api_workspace_created_total	Counter		Workspaces created
tokentimer_api_invite_sent_total	Counter		Workspace invites created
tokentimer_invite_cancelled_total	Counter		Pending invites cancelled

Route labels use the matched Express route pattern (for example /api/v1/workspaces/:id), never raw request paths, to keep label cardinality bounded.

Auto-sync metrics

Core 0.18.2 adds these installation-wide API metrics when ENABLE_METRICS=true; they do not require a Pushgateway:

Metric	Type	Labels	Meaning
auto_sync_items_created_total	Counter	provider	Committed inventory creations since instrumentation was enabled
auto_sync_run_outcomes_total	Counter	provider, status	Committed terminal runs: success, partial, failed, or superseded
auto_sync_last_attempt_timestamp_seconds	Gauge	provider	Latest run start, as Unix seconds
auto_sync_last_success_timestamp_seconds	Gauge	provider	Latest fully successful completion, as Unix seconds
auto_sync_enabled_configurations	Gauge	provider	Currently enabled connections

Apply migrations 79–80 before deploying this version. Counters and timestamps survive restarts and deletion; historical activity is not backfilled. A timestamp is zero until observed, and partial or failed runs do not update the last success. A failed scrape is not evidence of zero activity.

Scrape one stable API target per database, establish a baseline, and do not sum replicas: each exposes the same installation totals. For freshness alerts, require a timestamp above zero and scope alerts to enabled connections. Provider aggregates can hide a failed individual connection, so inspect its run history. The optional deploy/helm/examples/values-auto-sync-alerts.yaml and deploy/prometheus/auto-sync.rules.yaml provide examples; configure a unique installation label per database, verify target and rule discovery, and tune their windows for your schedule. Alertmanager controls delivery.

Worker metrics

Pushed to the Pushgateway per run, labeled with env (from ENVIRONMENT_SUFFIX or NODE_ENV):

Metric	Type	Labels	Meaning
alerts_queue_depth	Gauge	status	Alert queue depth by status
alerts_queue_due_now	Gauge		Alerts due now (pending/failed, not under cooldown)
alerts_cooldown_in_effect	Gauge		Alerts with active retry cooldown
runner_up	Gauge	component	Worker heartbeat by component
alerts_delivery_total	Counter	channel, provider, status	Alert deliveries
alerts_retry_total	Counter	channel, kind	Retries (auto/manual)
alerts_delivery_latency_seconds	Histogram	channel, provider	Delivery latency
alerts_webhook_denied_host_total	Counter	provider, host	Webhook hosts denied by the provider allowlist
weekly_digest_sent_total	Counter	channel, status	Weekly digests sent
weekly_digest_processed	Gauge		Groups processed in the last digest run
auto_sync_runs_total	Counter	provider, status	Auto-sync runs by outcome (success/failure)
auto_sync_items_imported_total	Counter	provider	Items imported (created or updated) by auto-sync
auto_sync_last_run_timestamp	Gauge	provider, status	Unix timestamp of the last auto-sync attempt
app_log_errors_total	Counter	service	Error-level log events

Worker metrics are pushed with job name tokentimer-alerts and a worker grouping label per worker type.

Kubernetes (Helm)

The Helm chart ships optional monitoring resources, all disabled by default:

monitoring:
  metrics:
    enabled: true          # sets ENABLE_METRICS=true
    pushgatewayUrl: "http://pushgateway.monitoring:9091"

  serviceMonitor:
    enabled: true          # requires the Prometheus Operator CRDs
    interval: "30s"
    scrapeTimeout: "10s"
    labels: {}             # see the note below: usually required for discovery
    namespace: ""          # defaults to the release namespace
    metricRelabelings: []

  prometheusRules:
    enabled: true
    groups: []             # your PrometheusRule groups

serviceMonitor.enabled alone is often not enough

Prometheus only picks up ServiceMonitors that match its serviceMonitorSelector. With kube-prometheus-stack that selector is label-based, so an enabled ServiceMonitor with no labels is created and then silently never scraped. Set the label your Prometheus expects:

monitoring:
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack

Confirm with kubectl get servicemonitor -n tokentimer --show-labels, then check the target appears under Status > Targets in the Prometheus UI.

The Ingress template routes /metrics to the API service; if you enable the Ingress and do not want metrics exposed publicly, keep monitoring.metrics.enabled off or block the path at your ingress controller.

Suggested alerts

• alerts_queue_due_now above a threshold for more than 15 minutes: the delivery worker is stuck or SMTP/webhooks are failing.
• Rate of alerts_delivery_total{status="failure"} above zero: inspect the alert queue and the audit log for ALERT_SEND_FAILED events.
• runner_up stale for a component: the corresponding worker CronJob is not running.
• Increase in tokentimer_api_rate_limit_hits_total or tokentimer_api_csrf_rejections_total: possible abusive traffic or a misconfigured client.

Health endpoint

Independent of metrics, GET /health on the API returns a JSON status suitable for container healthchecks and load balancer probes:

{
  "status": "healthy",
  "timestamp": "2026-03-19T12:00:00.000Z",
  "uptime": 123.456,
  "environment": "production"
}