Prometheus Alerting Rules: Writing Tests and CI Validation

A Prometheus alerting rule that never fires wastes weeks of debugging when an outage happens, while one that fires constantly trains your team to ignore pages. If you maintain Prometheus in production, you have pushed a bad threshold, a typo in a label selector, or a for duration that made an alert useless. Testing prometheus alerting rules before they reach your cluster catches these mistakes in CI, not at 3am.

promtool test rules runs your alert definitions against synthetic time series data and verifies which alerts fire, when they fire, and what labels they carry. You write test cases that inject metric values over time, then assert that specific alerts enter pending or firing states at the right moments. This article walks through the test file syntax, realistic examples for error rate and availability alerts, and CI pipeline configs that block merges when tests fail.

Prometheus Alert Rule Anatomy and Common Mistakes

A typical alert rule defines an expr PromQL query, a for duration before the alert fires, and labels and annotations that route the notification and provide context:

groups:
  - name: example
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status="500"}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on {{ $labels.instance }}"

Incorrect PromQL is the first mistake. The syntax passes promtool check rules, but the query selects no series because the label name is wrong or the metric does not exist. You deploy the rule, it never fires, and you discover the problem during an incident. Before: http_request_total (missing the s). After: http_requests_total. A test with realistic input series catches this immediately.

Wrong threshold means the alert fires constantly or never. An SLO-based alert might use > 0.001 (0.1%) when you meant > 0.01 (1%). The alert pages every evaluation interval. Before: rate(...) > 0.001. After: rate(...) > 0.01. Tests let you inject a 0.5% error rate and verify the alert stays silent, then inject 2% and confirm it fires.

Missing or incorrect labels break Alertmanager routing. You write severity: warning but your routing tree expects severity: page. The alert fires but lands in a black hole. Before: labels: {severity: warning}. After: labels: {severity: page}. Test assertions check the exact label set on exp_alerts, so a mismatch fails the build.

promtool test rules: File Structure and Syntax

A test file points to the rules you want to validate, defines synthetic time series as input, and asserts which alerts should fire at specific times. The Prometheus documentation on unit testing rules covers the full schema. Here is a minimal example that tests a simple alert:

rule_files:
  - rules.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'up{job="api", instance="server1"}'
        values: '1 1 1 0 0 0'

    alert_rule_test:
      - eval_time: 5m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels:
              severity: critical
              instance: server1
              job: api

Run promtool test rules test.yml. If the InstanceDown alert is defined in rules.yml and fires when up == 0 for the expected duration, the test passes. Output shows each test case and whether expectations matched.

The rule_files and evaluation_interval fields

rule_files is a list of paths to your alert and recording rule YAML files, relative to the test file. Prometheus loads these definitions and evaluates them against your test data. evaluation_interval sets how often Prometheus evaluates rules during the test; it defaults to 1m but you can use 30s or 15s for finer-grained tests. This interval applies to all tests in the file unless overridden in a specific test block with interval.

For syntax-only checks without running tests, use promtool check rules rules.yml. This catches YAML errors and invalid PromQL but does not verify alert behavior.

Writing an input_series block

input_series defines the metrics that exist during the test. Each series has a metric name, labels in the selector syntax, and a values string. The values are space-separated samples at each interval. The notation 1m+0 means "repeat the value 0 every 1m for the test duration," while 1 1 0 0 gives explicit samples at 0m, 1m, 2m, 3m. Use _ for a stale marker (the series disappears).

input_series:
  - series: 'http_requests_total{status="200", instance="web1"}'
    values: '0+10x10'  # starts at 0, increases by 10 each minute for 10 intervals
  - series: 'http_requests_total{status="500", instance="web1"}'
    values: '0+1x10'   # starts at 0, increases by 1 each minute

The 0+10x10 syntax means start at 0, add 10 each step, repeat 10 times. This creates a counter that grows steadily, which rate() can compute over.

Defining alert_rule_test expectations

alert_rule_test is a list of assertions. Each has an eval_time (when to check, like 5m), an alertname, and exp_alerts with the expected labels. If the alert should not be firing at that time, omit exp_alerts or set it to an empty list. Labels in exp_labels must match exactly, including any labels added by the rule itself.

alert_rule_test:
  - eval_time: 3m
    alertname: HighErrorRate
    exp_alerts: []  # alert should not be firing yet
  - eval_time: 6m
    alertname: HighErrorRate
    exp_alerts:
      - exp_labels:
          severity: critical
          instance: web1
        exp_annotations:
          summary: "Error rate above 5% on web1"

Annotations are optional but useful to verify templating works. If the test fails, promtool prints the diff between expected and actual alerts.

Testing a High Error Rate Alert

Consider an alert that fires when the HTTP 500 error rate exceeds 5% over a 5-minute window, and the condition persists for 5 minutes. The rule looks like this:

groups:
  - name: http_errors
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status="500"}[5m])
          / rate(http_requests_total[5m])
          > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on {{ $labels.instance }}"

You need input series for both status="500" and status="200" requests. The total request rate is the sum of both. If you inject 100 requests/min with 6 errors/min, the error rate is 6%, which should trigger the alert after the for duration.

Setting up realistic input series

Start both counters at 0 and increment them at each interval. For a 6% error rate, use 0+6x15 for 500s and 0+94x15 for 200s (total 100 req/min, 6 errors). Run the test for 15 minutes to cover the 5m window plus the 5m for duration:

rule_files:
  - http_errors.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{status="500", instance="web1"}'
        values: '0+6x15'
      - series: 'http_requests_total{status="200", instance="web1"}'
        values: '0+94x15'

At 5m, Prometheus has enough data to compute rate(...[5m]). The error rate is 6/100 = 0.06, which exceeds 0.05. The alert enters pending state. At 10m, the condition has been true for 5m, so the alert fires.

Testing the firing threshold and for duration

Write two assertions: one at 5m (alert pending but not firing), one at 10m (alert firing). The pending state is not directly testable in promtool, so you verify the alert is absent at 4m and present at 10m:

    alert_rule_test:
      - eval_time: 4m
        alertname: HighErrorRate
        exp_alerts: []
      - eval_time: 10m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              instance: web1
              status: "500"

Wait, the status label is wrong here. The rate() aggregation removes the status label unless you use by (instance) or similar. The actual alert labels depend on your expr. If the query groups by instance, only instance and severity appear. Adjust exp_labels to match what the rule produces. A common mistake is expecting labels that the aggregation discarded.

Verifying alert labels and annotations

Add a second test where the error rate spikes to 10% for 2 minutes, then drops to 1%. The alert should enter pending at 5m but resolve before firing because the condition does not hold for the full 5m for duration:

  - interval: 1m
    input_series:
      - series: 'http_requests_total{status="500", instance="web2"}'
        values: '0 0 0 0 0 10 10 1 1 1 1 1 1 1 1'
      - series: 'http_requests_total{status="200", instance="web2"}'
        values: '0+90x15'

    alert_rule_test:
      - eval_time: 7m
        alertname: HighErrorRate
        exp_alerts: []
      - eval_time: 12m
        alertname: HighErrorRate
        exp_alerts: []

Run promtool test rules http_errors.test.yml. If the test fails, the output shows which alerts were present but not expected, or expected but missing, along with their labels. This catches threshold bugs and for duration mistakes before you deploy.

Testing Absent Metric and Downtime Alerts

An alert that fires when a metric disappears protects you from silent failures: a scrape target goes down, the exporter crashes, or network issues block collection. The absent() function returns 1 when a time series does not exist. A typical downtime alert looks like this:

groups:
  - name: availability
    rules:
      - alert: ServiceDown
        expr: absent(up{job="api"}) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API service has no metrics for 2 minutes"

Testing this requires simulating metric absence. Prometheus marks a series stale 5 minutes after the last sample, but your alert should fire after 2 minutes of absence. You represent absence in input_series by ending the values string: the series stops at a specific time and never reappears.

Simulating metric absence with empty input_series

Start the up series with normal values, then stop providing samples. For example, values: '1 1 1' means the metric exists at 0m, 1m, 2m, then disappears. At 3m there is no new sample. Prometheus considers the series stale after 5m, but absent() returns 1 as soon as no sample matches the selector at evaluation time:

rule_files:
  - availability.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'up{job="api", instance="server1"}'
        values: '1 1 1'  # exists at 0m, 1m, 2m; absent from 3m onward

At 3m, the last sample is stale (older than the 5m lookback), so absent(up{job="api"}) evaluates to 1. The alert enters pending. At 5m, the condition has been true for 2m, so the alert fires.

Testing the absent() function

Write assertions at 2m (alert should not fire yet because the metric only just disappeared) and 5m (alert should be firing). The for: 2m means the absence must persist for 2 full minutes:

    alert_rule_test:
      - eval_time: 2m
        alertname: ServiceDown
        exp_alerts: []
      - eval_time: 5m
        alertname: ServiceDown
        exp_alerts:
          - exp_labels:
              severity: critical
              job: api

Now test the case where the metric returns within the for window. Extend the values to include a gap, then resume samples. Use _ to explicitly mark a stale sample if needed, though simply ending and restarting the series works:

  - interval: 1m
    input_series:
      - series: 'up{job="api", instance="server2"}'
        values: '1 1 1 _ 1 1 1 1 1 1'  # absent at 3m, returns at 4m

    alert_rule_test:
      - eval_time: 3m
        alertname: ServiceDown
        exp_alerts: []  # only 0m of absence, for duration not met
      - eval_time: 6m
        alertname: ServiceDown
        exp_alerts: []  # metric returned, alert resolved

Run the test. If it fails at 6m, the alert is still firing even though the metric returned. Check your expr: you might have a typo in the label selector or the for duration is longer than expected.

Handling stale series and lookback windows

Prometheus uses a 5-minute lookback window for instant queries. When you query up{job="api"} at 8m and the last sample was at 2m, Prometheus returns that sample if it is within 5m. At 7m and 1 second, the 2m sample is still valid. At 7m and 1 second, absent() returns 0 because the series is not absent yet. This interacts with your for duration: if you test at exactly 5m after the last sample, the series just became stale.

In practice, set eval_time to a safe margin past the for duration. For for: 2m, test at 5m or 6m, not 2m exactly, because the alert transitions from pending to firing between evaluations. The official Prometheus unit testing documentation explains staleness markers and lookback behavior in detail. If your alert uses a range selector like up{job="api"}[5m], the range must have at least one sample within the window, or the query returns no data and absent() fires.

CI Integration: GitHub Actions and GitLab CI

Running promtool test rules in CI blocks bad alert definitions before they merge. Use the official prom/prometheus container image, which includes promtool, and fail the pipeline if any test or syntax check fails. This setup catches typos, wrong thresholds, and missing labels every time someone opens a pull request.

GitHub Actions workflow with promtool container

Create .github/workflows/prometheus-tests.yml in your repository. Use the prom/prometheus image with docker run, mount your rules and test files, and run both check and test commands. Here is a complete workflow that runs on every PR:

name: Prometheus Alert Tests

on:
  pull_request:
    paths:
      - 'prometheus/rules/**'
      - 'prometheus/tests/**'

jobs:
  test-alerts:
    runs-on: ubuntu-latest
    container:
      image: prom/prometheus:latest

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Check rule syntax
        run: promtool check rules prometheus/rules/*.yml

      - name: Run alert tests
        run: promtool test rules prometheus/tests/*.test.yml

If promtool exits with a non-zero code, the job fails and the PR cannot merge. You can cache the Docker image with actions/cache, but the prom/prometheus image is small and pulls quickly. For repositories with many rule files, split tests into parallel jobs by directory or rule group; GitHub Actions supports matrix builds that run multiple jobs concurrently.

GitLab CI job using Prometheus image

In .gitlab-ci.yml, define a job that uses image: prom/prometheus:latest and runs promtool in the script block. GitLab pulls the image, mounts your repository at /builds/..., and executes the commands:

stages:
  - test

prometheus-tests:
  stage: test
  image: prom/prometheus:latest
  script:
    - promtool check rules rules/*.yml
    - promtool test rules tests/*.test.yml
  only:
    changes:
      - rules/**/*
      - tests/**/*

The only: changes: block runs the job only when rule or test files change, saving CI minutes. If you have dozens of test files, split them across multiple jobs with parallel: and pass different file globs to each. GitLab caches the Docker image across pipeline runs by default, so subsequent runs are faster.

Failing the pipeline on test or check failures

Both promtool check rules and promtool test rules exit with code 1 on failure. CI systems treat any non-zero exit as a failed job, so you do not need extra logic. If you want to continue running tests even after one file fails (to see all errors at once), use set +e in a shell script and track failures manually, but the default fail-fast behavior is simpler and catches the first broken rule immediately.

For additional validation, run promtool check config prometheus.yml to verify your main Prometheus config file. This catches mistakes in scrape_configs, alerting, and rule_files paths. Combine all three checks in one job to validate the entire monitoring stack before deployment. If you manage alert rules alongside infrastructure as code, integrate this with your broader CI/CD pipeline described in the CI/CD Pipeline Quick Reference.

Advanced Test Patterns: Grouped Alerts and Aggregations

Alerts that aggregate metrics across multiple instances require test cases with multiple input series and careful label matching. An alert that sums request rates across all instances in a job fires when the total exceeds a threshold, not when a single instance crosses it. The rule removes per-instance labels during aggregation, so your test assertions must expect only the grouped labels.

Testing alerts with sum, avg, or count aggregations

Consider an alert that fires when total requests across all api job instances exceed 100 req/s:

groups:
  - name: capacity
    rules:
      - alert: HighTotalLoad
        expr: sum(rate(http_requests_total[5m])) by (job) > 100
        for: 3m
        labels:
          severity: warning

The sum(...) by (job) aggregates all instances into a single value per job. Create input series for two instances with different request rates. If instance A does 60 req/s and instance B does 50 req/s, the sum is 110, which exceeds the threshold:

rule_files:
  - capacity.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", instance="server1"}'
        values: '0+3600x10'  # 60 req/s * 60s = 3600 per minute
      - series: 'http_requests_total{job="api", instance="server2"}'
        values: '0+3000x10'  # 50 req/s * 60s = 3000 per minute

    alert_rule_test:
      - eval_time: 3m
        alertname: HighTotalLoad
        exp_alerts: []
      - eval_time: 8m
        alertname: HighTotalLoad
        exp_alerts:
          - exp_labels:
              severity: warning
              job: api

Notice exp_labels includes job: api but not instance. The aggregation removed the instance label. If you expect instance: server1 in the alert, the test fails with a label mismatch. At 8m, the condition has been true for 5 minutes (since 3m when rate() had enough data), so the for: 3m duration is satisfied and the alert fires.

Multiple instances and label matching

Test the case where individual instances stay below the threshold but the sum exceeds it. Then test the inverse: one instance spikes to 120 req/s, but after aggregation by job, the alert should still fire because the expr sums all instances. Write a second test with server3 at 120 req/s and server4 at 10 req/s; the sum is 130:

  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", instance="server3"}'
        values: '0+7200x10'  # 120 req/s
      - series: 'http_requests_total{job="api", instance="server4"}'
        values: '0+600x10'   # 10 req/s

    alert_rule_test:
      - eval_time: 8m
        alertname: HighTotalLoad
        exp_alerts:
          - exp_labels:
              severity: warning
              job: api

For avg() aggregations, the math changes but the test structure is identical. If you alert on avg(rate(...)) by (job) > 50, inject two instances with 40 and 60 req/s; the average is 50, which does not exceed the threshold. Inject 70 and 80; the average is 75, and the alert fires. The exp_labels still omits instance because by (job) groups only on job.

Testing alert groups and routing labels

Alerts often include extra labels for Alertmanager routing. If your rule adds team: backend or component: api, verify these labels appear in exp_labels. For annotations that use {{ $labels.job }} templating, add exp_annotations to the test:

      - alert: HighTotalLoad
        expr: sum(rate(http_requests_total[5m])) by (job) > 100
        for: 3m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "Total load for {{ $labels.job }} exceeds 100 req/s"

In the test, check both labels and the annotation text:

        exp_alerts:
          - exp_labels:
              severity: warning
              team: backend
              job: api
            exp_annotations:
              summary: "Total load for api exceeds 100 req/s"

promtool does not deeply validate Go templating logic, but it confirms the annotation key exists and matches the expected string. If you use {{ $value }} or {{ printf }}, the test compares the rendered output, so you must compute what the final string should be at the eval_time you chose. For complex templating, consider testing the alert in a live Prometheus instance alongside unit tests.

Running and Debugging Test Failures

When a test fails, promtool prints a diff showing which alerts were expected but missing, or present but not expected, along with the actual labels. Reading this output line by line tells you whether the problem is a wrong eval_time, a threshold off by one, or a label mismatch from aggregation.

Reading promtool test output line by line

Run promtool test rules tests/http_errors.test.yml. A passing test prints:

Unit Testing: tests/http_errors.test.yml
  SUCCESS

A failing test shows the test name, the eval_time, and the diff. Example output for an expected alert that did not fire:

Unit Testing: tests/http_errors.test.yml
  FAILED:
    alertname: HighErrorRate, time: 10m,
        exp: labels:{instance="web1", severity="critical"}
        got: (none)

This means at 10m, you expected the HighErrorRate alert with instance="web1" and severity="critical", but no alert was active. The rule either did not evaluate to true, or the for duration was not met. Check your input_series: did you provide enough samples for the rate() window? Did the error rate actually exceed the threshold?

If the output shows an unexpected alert, you get:

    alertname: HighErrorRate, time: 4m,
        exp: (none)
        got: labels:{instance="web1", severity="critical"}

The alert fired at 4m when you expected it to still be silent. This usually means the for duration is shorter than you thought, or the threshold was crossed earlier than your input series intended.

Common errors: eval_time off by one interval, wrong labels

The most frequent mistake is eval_time that does not account for the for duration. If your rule has for: 5m and the condition becomes true at 5m (once rate(...[5m]) has enough data), the alert fires at 10m, not 5m. Testing at eval_time: 5m finds the alert pending or absent, not firing. Correct: test at eval_time: 10m or later.

Label mismatches happen when aggregation removes labels you expected. If your expr uses sum(...) by (job), the instance label disappears. Your test expects instance: server1, but the alert only has job: api. The diff shows:

    exp: labels:{instance="server1", job="api", severity="critical"}
    got: labels:{job="api", severity="critical"}

Remove instance from exp_labels. Another variant: you added a label in the rule definition (labels: {team: backend}) but forgot to include it in exp_labels. The test fails because the actual alert has an extra label. Add team: backend to your expectations.

Using promtool query instant for manual checks

If a test fails and you cannot see why the PromQL should evaluate differently, use promtool query instant against a running Prometheus instance to manually evaluate the expression at a specific time. This requires a live server with the same metrics, but it shows the exact output of your query:

promtool query instant http://localhost:9090 \
  --time 2024-01-15T10:00:00Z \
  'rate(http_requests_total{status="500"}[5m]) / rate(http_requests_total[5m])'

The output shows the value and labels at that timestamp. Compare this to what your test expects. If the value is 0.04 and your threshold is > 0.05, the alert correctly does not fire. If the value is 0.06 but the alert is absent, check the for duration: the condition must be true for the full duration before the alert transitions to firing. promtool query instant does not help with unit tests directly (it needs a running Prometheus), but it is useful for debugging complex PromQL during development. For pure test debugging, add more eval_time checkpoints at 1m intervals and print which alerts are active at each step by running the test verbosely.

Checklist: Adding Tests to Your Prometheus Repo

Start by creating a tests/ directory alongside your rules/ directory. Write one .test.yml file per rule file, or group related tests together. For each alert, write at least two test cases: one where the condition is met and the alert fires, one where it stays silent. Include edge cases like metrics returning after absence, thresholds at exact boundaries, and aggregations with varying numbers of instances.

Add a CI job that runs promtool check rules and promtool test rules on every pull request. Use the prom/prometheus:latest Docker image for consistency. Fail the pipeline if any test fails. Document your test coverage in the repository README: which alerts have tests, which edge cases are covered, and how to run tests locally. For teams new to testing infrastructure as code, the CI/CD Pipeline Quick Reference provides a broader checklist for integrating validation steps across your deployment workflow.

Review test results in every PR. If a test fails, do not merge until the failure is understood and fixed. Over time, expand tests to cover recording rules, multi-cluster scenarios, and integration with Alertmanager routing. Keep test files next to the rules they validate so changes to a rule prompt a review of its tests. For larger organizations, consider a shared test library with reusable input series for common metrics like up, node_cpu_seconds_total, or http_requests_total.

Frequently Asked Questions