Monitoring & metrics

Self-hosted TokenTimer exposes Prometheus metrics from the API and pushes worker metrics to an optional Pushgateway. Metrics are disabled by default.

Enabling metrics

Variable	Description	Default	Scope
ENABLE_METRICS	Enable /metrics and metric push attempts when true	false	API, worker
PUSHGATEWAY_URL	Prometheus Pushgateway endpoint for worker metrics	unset	Worker
ENVIRONMENT_SUFFIX	Metrics environment label (env) when running multiple instances	unset	Worker

With ENABLE_METRICS=true, the API serves Prometheus text format at GET /metrics on the API port (default 4000). The endpoint has no authentication, so restrict access at the network layer (NetworkPolicy, firewall, or scrape from inside the cluster only).

Worker runs are short-lived (one job per scheduled run, whether launched by a Kubernetes CronJob or by the long-running Compose runner), so workers push their metrics to PUSHGATEWAY_URL at the end of each run instead of exposing a scrape endpoint. Schedules are described in Configuration → Worker scheduling.

API metrics

Metric	Type	Labels	Meaning
tokentimer_api_http_request_duration_seconds	Histogram	method, route, status	HTTP request duration
tokentimer_api_http_requests_total	Counter	method, route, status	Total HTTP requests
tokentimer_api_login_attempts_total	Counter	outcome	Login attempts
tokentimer_api_2fa_verifications_total	Counter	outcome	2FA verification attempts
tokentimer_api_rate_limit_hits_total	Counter		HTTP 429 responses
tokentimer_api_csrf_rejections_total	Counter		Requests rejected by CSRF protection
tokentimer_api_rbac_denied_total	Counter		RBAC authorization denials
tokentimer_api_workspace_created_total	Counter		Workspaces created
tokentimer_api_invite_sent_total	Counter		Workspace invites created
tokentimer_invite_cancelled_total	Counter		Pending invites cancelled

Route labels use the matched Express route pattern (for example /api/v1/workspaces/:id), never raw request paths, to keep label cardinality bounded.

Worker metrics

Pushed to the Pushgateway per run, labeled with env (from ENVIRONMENT_SUFFIX or NODE_ENV):

Metric	Type	Labels	Meaning
alerts_queue_depth	Gauge	status	Alert queue depth by status
alerts_queue_due_now	Gauge		Alerts due now (pending/failed, not under cooldown)
alerts_cooldown_in_effect	Gauge		Alerts with active retry cooldown
runner_up	Gauge	component	Worker heartbeat by component
alerts_delivery_total	Counter	channel, provider, status	Alert deliveries
alerts_retry_total	Counter	channel, kind	Retries (auto/manual)
alerts_delivery_latency_seconds	Histogram	channel, provider	Delivery latency
alerts_webhook_denied_host_total	Counter	provider, host	Webhook hosts denied by the provider allowlist
weekly_digest_sent_total	Counter	channel, status	Weekly digests sent
weekly_digest_processed	Gauge		Groups processed in the last digest run
auto_sync_runs_total	Counter	provider, status	Auto-sync runs by outcome (success/failure)
auto_sync_items_imported_total	Counter	provider	Items imported (created or updated) by auto-sync
auto_sync_last_run_timestamp	Gauge	provider, status	Unix timestamp of the last auto-sync attempt
app_log_errors_total	Counter	service	Error-level log events

Worker metrics are pushed with job name tokentimer-alerts and a worker grouping label per worker type.

Kubernetes (Helm)

The Helm chart ships optional monitoring resources, all disabled by default:

monitoring:
  metrics:
    enabled: true          # sets ENABLE_METRICS=true
    pushgatewayUrl: "http://pushgateway.monitoring:9091"

  serviceMonitor:
    enabled: true          # requires the Prometheus Operator CRDs
    interval: "30s"
    scrapeTimeout: "10s"

  prometheusRules:
    enabled: true
    groups: []             # your PrometheusRule groups

The Ingress template routes /metrics to the API service; if you enable the Ingress and do not want metrics exposed publicly, keep monitoring.metrics.enabled off or block the path at your ingress controller.

Suggested alerts

• alerts_queue_due_now above a threshold for more than 15 minutes: the delivery worker is stuck or SMTP/webhooks are failing.
• Rate of alerts_delivery_total{status="failure"} above zero: inspect the alert queue and the audit log for ALERT_SEND_FAILED events.
• runner_up stale for a component: the corresponding worker CronJob is not running.
• Increase in tokentimer_api_rate_limit_hits_total or tokentimer_api_csrf_rejections_total: possible abusive traffic or a misconfigured client.

Health endpoint

Independent of metrics, GET /health on the API returns a JSON status suitable for container healthchecks and load balancer probes:

{
  "status": "healthy",
  "timestamp": "2026-03-19T12:00:00.000Z",
  "uptime": 123.456,
  "memory": { "rss": 52428800 },
  "environment": "production"
}