Prometheus Interview Questions and Answers

Answer

Prometheus is an open-source time-series database and monitoring system, originally built at SoundCloud in 2012 and now a graduated CNCF project. It solves three problems that traditional monitoring tools (Nagios, Zabbix) handled poorly in cloud-native environments: (1) static configuration breaks in environments where servers come and go every minute, Prometheus uses service discovery instead, (2) push-based agents create operational fragility and need separate config for every host, Prometheus pulls metrics from a single config file, and (3) dimensional data (per-endpoint, per-status-code, per-tenant) requires a real query language, Prometheus introduced PromQL, which lets you slice metrics by labels. Prometheus stores metrics in its own TSDB optimized for high-cardinality time series, and integrates with AlertManager for routing alerts and Grafana for visualization.

Architecturally it ships as one statically-linked Go binary with a local TSDB and no external dependencies, and that is deliberate: your monitoring should survive the outage it is reporting on. The same choice is the limitation a senior interviewer probes, because a single server does not shard, does not replicate, and its data lives on one disk, so HA means running two identical replicas scraping the same targets and deduplicating at query time in Thanos or Mimir. Name what Prometheus deliberately does not do: no log storage, no distributed tracing, float64 samples only, and no per-scrape delivery guarantee, which makes it the wrong system for billing-grade accounting. The usual follow-up is 'what happens when a scrape fails?', and the answer is that the series simply has a gap, `rate()` interpolates across it, and `up{job="api"} == 0` is the thing you actually alert on.

Answer

Every Prometheus time series is uniquely identified by a metric name plus a set of key-value labels. The metric name describes WHAT is being measured (e.g. `http_requests_total`), and labels describe the DIMENSIONS along which the measurement varies (e.g. `method="GET"`, `status="200"`, `endpoint="/api/users"`). Together they form a unique series, and each series has a stream of (timestamp, float64) samples.

The same metric name can appear with different label combinations, each unique combination is its own series. This dimensional model is what makes Prometheus powerful: you can query `http_requests_total` aggregated any way you want, instead of pre-deciding which dimensions to track. Two details separate a memorised answer from a working one.

First, label ordering is irrelevant but label presence is not: `http_requests_total{method="GET"}` and `http_requests_total{method="GET",pod="api-1"}` are different series, so adding a label to an existing metric silently forks every series and breaks `rate()` continuity across the deploy. Second, the metric name is itself just a label, `__name__`, which is why `{__name__=~"http_.+"}` is a legal selector and why you can drop or rewrite names in `metric_relabel_configs`. Labels beginning with `__` are internal and stripped before storage unless you copy them onto a real label first. Sample values are always float64, there is no integer or string type, so string data has to be modelled as an info metric whose value is always 1 and whose labels carry the payload, then joined at query time with `group_left`.

# Metric exposition format (/metrics endpoint)
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200",endpoint="/api/users"} 1042
http_requests_total{method="GET",status="500",endpoint="/api/users"} 3
http_requests_total{method="POST",status="201",endpoint="/api/users"} 87

# Each unique label combination = a separate time series

Answer

Prometheus has four metric types, and picking the right one is the most common beginner mistake. (1) **Counter**, monotonically increasing value (only goes up, or resets to zero on process restart). Use for: request counts, errors, bytes processed. Always query with `rate()` or `increase()`. (2) **Gauge**, single value that goes up and down.

Use for: temperatures, memory usage, queue depth, in-flight requests. (3) **Histogram**, samples observations into configurable buckets (e.g. request latency in 10ms, 50ms, 100ms buckets). Cumulative on the client side, aggregatable on the server side. (4) **Summary**, like histogram but calculates quantiles on the client side. Cannot be aggregated across instances.

Histograms are preferred over summaries in 2026 because they aggregate cleanly across replicas, summaries are mathematically broken when you sum them. Two follow-ups come up constantly. First, how does Prometheus survive a counter reset? `rate()` and `increase()` detect that the value went down between two samples, treat the drop as a process restart, and add the pre-reset value back, which is why you must never do plain arithmetic on a raw counter.

Second, what actually goes over the wire? A histogram is not one series: it exposes `foo_bucket` (one series per `le`, cumulative), plus `foo_sum` and `foo_count`. A summary exposes `foo{quantile="0.99"}` series plus `foo_sum` and `foo_count`. Knowing that exposition shape is what lets you compute a mean for either type as `rate(foo_sum[5m]) / rate(foo_count[5m])`, and it explains why one histogram with ten buckets costs twelve series per label combination.

# Counter: only increases, resets to 0 on process restart
http_requests_total{status="200"} 4821

# Gauge: moves in both directions
node_memory_MemAvailable_bytes 3.221225472e+09

# Histogram: cumulative _bucket series + _sum + _count
http_request_duration_seconds_bucket{le="0.1"} 4102
http_request_duration_seconds_bucket{le="0.5"} 4790
http_request_duration_seconds_bucket{le="+Inf"} 4821
http_request_duration_seconds_sum 512.4
http_request_duration_seconds_count 4821

# Summary: client-computed quantiles + _sum + _count
gc_pause_seconds{quantile="0.99"} 0.031
gc_pause_seconds_sum 12.7
gc_pause_seconds_count 9310

# Mean latency works for either shape:
#   rate(http_request_duration_seconds_sum[5m])
# / rate(http_request_duration_seconds_count[5m])

Answer

In Prometheus's pull model, the server periodically fetches (scrapes) metrics from HTTP endpoints exposed by each target. Targets expose `/metrics` in a simple text format and don't know or care about Prometheus, they just expose stats. The Prometheus server has the full list of targets (via service discovery or a static config), and scrapes them at a configurable interval (typically 15s or 30s).

Why pull? (1) Easy to detect down targets, if the scrape fails, Prometheus knows immediately and can alert. (2) Easier to debug, you can manually `curl /metrics` to see the same data Prometheus sees. (3) No need to configure each target with the server's address, the server has the addresses. (4) Better control over scrape rate, you can't accidentally DDoS Prometheus by deploying 10,000 new instances. The drawback is short-lived jobs (batch jobs, lambdas) that finish before the next scrape, for those, Prometheus offers the Pushgateway as a workaround. The scrape itself has a budget: `scrape_timeout` defaults to 10s and is rejected at config load if it exceeds `scrape_interval`, and a target that blows the timeout is recorded as `up == 0` with no partial data kept, which is why a slow `/metrics` handler on a fat exporter shows up as a flapping target rather than as slow graphs.

Prometheus also writes four synthetic series per target on every scrape: `up`, `scrape_duration_seconds`, `scrape_samples_scraped` and `scrape_samples_post_metric_relabeling`. Those four are the first things to graph when a job misbehaves, because together they tell you whether the target is reachable, how long it took, how much it emitted, and how much survived your drop rules.

Answer

PromQL has two fundamental vector types. An **instant vector** is one sample per series at a single point in time, that's what you get when you query a metric name directly. A **range vector** is a series of samples over a time window, you get this by appending `[duration]` to a selector.

Most aggregation operators (sum, avg) work on instant vectors. Most rate functions (rate, increase, irate) require range vectors. The difference matters because Grafana panels almost always want an instant vector (one value per timestamp), so you typically wrap a range vector in `rate()` to collapse it.

Three rules follow from this and interviewers test all three. You cannot graph a range vector directly: `http_requests_total[5m]` errors out in a Grafana time-series panel because there is no single value per timestamp. You cannot attach `[5m]` to an expression, only to a selector, so `rate(sum(x)[5m])` is a parse error and you need a subquery, `rate(sum(x)[5m:30s])`, instead.

And the window must contain at least two samples for any rate function to return anything, so a `[1m]` range on a 60s scrape interval frequently returns empty; the practical rule is to make the range at least four times the scrape interval, which is exactly why `[5m]` is the default choice on a 15s scrape. There is also a third type, the scalar, produced by numeric literals and `scalar()`, and comparison operators between a vector and a scalar filter the vector rather than returning a boolean unless you append the `bool` modifier.

# Instant vector: latest value of each series
http_requests_total

# Range vector: all samples in last 5 minutes
http_requests_total[5m]

# rate() converts range vector -> instant vector
# (per-second rate of increase over the window)
rate(http_requests_total[5m])

# Aggregation works on instant vectors
sum by (status) (rate(http_requests_total[5m]))

Answer

All three operate on counter range vectors and handle counter resets automatically. (1) `rate(metric[5m])` returns the per-SECOND average rate of increase over the window, this is what you want 95% of the time. It uses linear regression-like smoothing across all samples in the window. (2) `increase(metric[5m])` returns the TOTAL increase over the window (rate × window seconds). Useful for 'how many errors in the last hour?'. (3) `irate(metric[5m])` returns the rate between the LAST TWO samples in the window, much spikier than `rate()`, useful for short-term anomaly detection but bad for alerting.

The rule: use `rate()` for graphs and alerts, `increase()` for human-readable totals, `irate()` only for high-resolution debugging. Two subtleties come up in senior interviews. Both `rate()` and `increase()` extrapolate out to the window edges rather than using only the observed endpoints, so `increase()` on a slow counter can legitimately return a non-integer like 3.7 requests; that is correct behaviour, not a bug, and Prometheus 3.x only rounds toward integers for the special case of a counter that clearly increments by whole numbers.

Neither function can see resets it never sampled either, so if a pod restarts twice between two scrapes the intermediate counts are lost forever. Because `increase()` is literally `rate()` multiplied by the window in seconds, alert on `rate()` and reserve `increase()` for human-facing panels where 'errors in the last hour' reads better than 0.0031 per second.

# Per-second request rate (smooth)
rate(http_requests_total[5m])

# Total requests in the last hour
increase(http_requests_total[1h])

# Instantaneous rate (spiky, for short windows)
irate(http_requests_total[1m])

Answer

In 2026, almost nobody installs Prometheus from a tarball anymore, the standard pattern is the kube-prometheus-stack Helm chart on Kubernetes (which bundles Prometheus + AlertManager + Grafana + node-exporter + kube-state-metrics), or the official Docker image for non-K8s setups. The core config file is `prometheus.yml` and contains four sections: `global` (scrape interval, evaluation interval), `scrape_configs` (what to scrape), `alerting` (where AlertManager lives), and `rule_files` (paths to alerting and recording rules). For Kubernetes, the Prometheus Operator introduces CRDs like `ServiceMonitor` and `PrometheusRule` so you can define scrape targets declaratively per service instead of editing one giant config.

Know the operational flags too, because that is where the follow-ups go: `--config.file`, `--storage.tsdb.path`, `--storage.tsdb.retention.time`, `--web.enable-lifecycle` (required before `curl -XPOST http://localhost:9090/-/reload` works, and off by default) and `--web.enable-admin-api` (required for the delete-series endpoint, also off by default and a genuine security hole if the port is exposed). Validate before you reload: `promtool check config prometheus.yml` and `promtool check rules rules/*.yml` catch YAML and PromQL errors in CI. The behaviour on a bad config differs by path, which interviewers like: a bad config on reload is rejected and the previous one keeps serving, while a bad config at startup makes the process exit non-zero. On Kubernetes the reload equivalent is the config-reloader sidecar the Operator injects, which watches the mounted Secret and hits the lifecycle endpoint for you.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node1:9100', 'node2:9100']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'rules/*.yml'

Answer

An exporter is a sidecar process that translates metrics from a non-Prometheus system into the Prometheus text format. Standard exporters in every production stack: (1) **node_exporter**, host-level metrics: CPU, memory, disk, network. Runs on every VM. (2) **cAdvisor**, container metrics.

Built into kubelet, so on Kubernetes you get this free. (3) **kube-state-metrics**, Kubernetes object state (pod count, deployment replicas, etc.). (4) **blackbox_exporter**, probe HTTP/HTTPS/TCP/ICMP/DNS endpoints from outside. Used for uptime checks. (5) **mysql_exporter / postgres_exporter / redis_exporter**, database metrics. The pattern: if your dependency doesn't natively expose Prometheus metrics, there's almost certainly an exporter for it on the Prometheus GitHub org or by a third party.

Two design rules interviewers look for. An exporter should be co-located with the thing it exports, one node_exporter per host and one postgres_exporter per database instance, so that a failed scrape unambiguously means that instance is unreachable rather than that a shared exporter died. And exporters must not aggregate or add unbounded labels: `mysqld_exporter`'s per-table `perf_schema` collectors and `node_exporter`'s `--collector.systemd` are the classic cardinality landmines, and you leave them off unless you actually query them. Multi-target exporters (blackbox_exporter, snmp_exporter) break the co-location rule on purpose: one exporter instance probes many endpoints, you pass the real target as a `?target=` query parameter, and relabeling rewrites `__param_target`, `instance` and `__address__` so the resulting series look like they came from the probed host.

# blackbox_exporter multi-target pattern
scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://api.example.com/healthz
          - https://app.example.com/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

# The exporter returns synthetic result metrics you alert on:
#   probe_success == 0
#   probe_ssl_earliest_cert_expiry - time() < 14 * 86400

Answer

The Pushgateway lets short-lived jobs push metrics to a stable endpoint that Prometheus then scrapes. Pure pull doesn't work for batch jobs that finish in 30 seconds, by the time Prometheus's next scrape happens, the process is gone. The Pushgateway sits in the middle: the batch job pushes its final metrics (success, duration, rows processed) on completion, and Prometheus scrapes the Pushgateway like any other target.

Important gotcha: Pushgateway is NOT for service-level metrics from long-running processes, that's an anti-pattern that defeats the pull model's failure detection. Only use it for cron jobs, batch pipelines, and CI pipelines. The gotchas are what actually get asked.

Pushed metrics are persistent: the Pushgateway keeps serving the last value forever until you issue `DELETE /metrics/job/<job>`, so a decommissioned cron job leaves a permanently green `backup_success 1` quietly lying to your alerts. It is a single point of failure, it does not aggregate, and two jobs pushing to the same grouping key overwrite each other rather than summing. Prometheus scrapes it with `honor_labels: true`, which is why the pushed `job` and `instance` labels survive instead of being overwritten by the scrape config. The modern alternative that avoids all of this is to have the batch job write `job_last_success_timestamp_seconds` and alert with `time() - job_last_success_timestamp_seconds > 86400`, because that catches a job that never ran at all, which a missing push cannot.

# Push metrics from a batch job (bash)
cat <<EOF | curl --data-binary @- http://pushgateway:9091/metrics/job/backup_job
# TYPE backup_duration_seconds gauge
backup_duration_seconds 142
# TYPE backup_success gauge
backup_success 1
EOF

Answer

Every major language has an official Prometheus client library. The pattern is the same: import the client, create metric objects at startup (Counter, Gauge, Histogram, Summary), increment/observe them in your application code, and expose a `/metrics` HTTP endpoint that the library writes the text format to. In Python, you wire it as a Flask/FastAPI route; in Go, you use `promhttp.Handler()`; in Node.js, you use `prom-client`.

The library handles the cumulative bookkeeping (counters, histogram buckets) so you only think in terms of 'I want to count this' or 'I want to record this latency'. The interview follow-ups are about discipline, not syntax. Declare metric objects once at module scope, never inside a request handler: re-registering the same name raises a duplicate-registration error, and creating them per request leaks memory.

Use base units and the conventional suffixes, seconds not milliseconds, bytes not megabytes, `_total` on counters, `_seconds` on durations, and never encode the unit in a label. Keep label values bounded and pre-declare the combinations you expect, because `.labels()` lazily creates a new child series and a typo'd or user-supplied value permanently adds cardinality that only a restart clears. In multi-process Python (gunicorn with several workers) the default in-process registry is wrong, since each worker keeps its own counters and Prometheus scrapes whichever one answers, so you set `PROMETHEUS_MULTIPROC_DIR` and use `MultiProcessCollector`. In Go, `promauto` against an explicit `prometheus.NewRegistry()` avoids the default registry's global state, which otherwise makes tests order-dependent.

# Python with prometheus_client
from prometheus_client import Counter, Histogram, start_http_server
import time

requests = Counter("http_requests_total", "Total HTTP requests", ["method", "status"])
latency = Histogram("http_request_duration_seconds", "Request latency", ["endpoint"])

@latency.labels(endpoint="/api").time()
def handle_request():
    requests.labels(method="GET", status="200").inc()
    # do work

start_http_server(8000)  # exposes /metrics on :8000

Answer

Grafana is the standard visualization tool for Prometheus (and most other observability data sources). It connects to Prometheus via its HTTP API, lets you write PromQL queries in dashboard panels, and renders them as graphs, gauges, tables, or heatmaps. Grafana doesn't store data itself, it's a query and visualization layer.

The standard pattern in 2026: Prometheus stores metrics, Grafana queries them, and both are exposed at internal URLs behind your VPN or SSO. Grafana also handles alerting in newer versions (Grafana 8+), but most teams still use AlertManager for alert routing because it integrates better with the broader Prometheus ecosystem and supports inhibition rules natively. Importing community dashboards from grafana.com (USE method, RED method, kube-prometheus dashboards) saves weeks of work, search for dashboards by the exporter name (e.g. 'node_exporter full', 'kube-state-metrics overview') to find well-maintained ones with thousands of downloads.

Provisioning dashboards as code (via the `grafana-operator` on Kubernetes or Terraform provider) is best practice so dashboards survive Grafana reinstalls. Two integration details separate people who have run this from people who have read about it. The Prometheus data source distinguishes instant from range queries per panel, and a stat or table panel accidentally left on 'range' is the most common reason a dashboard is slow for no visible reason, because it fetches every step and throws away all but the last. And inside `rate()` you use the `$__rate_interval` variable, not `$__interval`: `$__rate_interval` is computed to be at least four scrape intervals wide, which is what stops graphs from going blank when a user zooms into a five-minute window and the range no longer contains two samples.

Answer

Prometheus is open-source, self-hosted, and primarily focused on metrics. DataDog and New Relic are SaaS products that bundle metrics, logs, APM tracing, and synthetics into one paid platform. Trade-offs: (1) **Cost**, Prometheus scales to thousands of metrics for the cost of a couple of VMs; DataDog can hit ₹50L+/month at any meaningful scale due to per-host and per-custom-metric pricing. (2) **Ops burden**, you have to operate Prometheus yourself (storage, HA, alerting); DataDog is zero-ops. (3) **Cardinality**, DataDog quietly drops or charges extra for high cardinality; Prometheus surfaces the problem so you can fix it. (4) **Ecosystem**, Prometheus is the standard for Kubernetes-native shops; DataDog is more common at enterprises without K8s expertise. (5) **Integration depth**, DataDog ships hundreds of pre-built integrations with one-click setup; Prometheus has exporters but you wire each one yourself. (6) **Query language**, PromQL is more expressive for time-series math; DataDog's query language is more approachable for non-engineers.

In India, every cost-conscious startup runs Prometheus; large enterprises and regulated finance often pay for DataDog. Hybrid setups (Prometheus for infra metrics, DataDog for APM) are also common at series-C-and-later companies.

Answer

Cardinality = the number of unique time series. Every unique label-value combination creates a new series, and Prometheus's storage and query cost scales linearly with cardinality. The classic disaster: adding `user_id` as a label on a per-request metric.

With 1M users and 5 endpoints × 3 status codes, that's 15M series, which can OOM a Prometheus instance and slow queries to a crawl. The rule: NEVER put unbounded high-cardinality values (user_id, request_id, email, URL with query strings) in labels. Bound values like HTTP method (5-7 options), status code (~10 used), endpoint (tens to hundreds) are fine.

To diagnose, query `topk(20, count by (__name__) ({__name__=~".+"}))` to find your worst offenders, and use `prometheus_tsdb_symbol_table_size_bytes` to track series growth. In 2026, native histograms reduce cardinality further by replacing per-bucket series with a single sparse-encoded series. Two hard guardrails belong in the answer. `sample_limit` on a scrape job makes Prometheus discard the entire scrape and set `up == 0` once a target exposes more than N series, which converts a slow cardinality poisoning into a loud, obvious target failure. `label_limit`, `label_name_length_limit` and `label_value_length_limit` reject pathological label sets before they are stored. Also know the memory shape, because interviewers want a number: roughly 8KB of head-block memory per active series, so five million series is tens of gigabytes of resident memory before you have run a single query.

Answer

`histogram_quantile()` calculates a quantile (0-1) from a Prometheus histogram. The histogram metric exposes a series of `_bucket{le="X"}` time series, cumulative counts of observations less than or equal to each bucket boundary. To get p99 latency, you typically: (1) `rate()` the bucket series over a window, (2) `sum by (le)` them across instances, (3) feed the result to `histogram_quantile(0.99, ...)`.

Common bug: forgetting to keep `le` in the `sum by` clause, without it, the quantile calculation breaks because it can't figure out the bucket boundaries. Three accuracy caveats a senior interviewer fishes for. The result is linearly interpolated inside whichever bucket the quantile lands in, so if 99% of traffic falls between `le="0.5"` and `le="+Inf"` your p99 is a guess with no upper bound and the function returns the highest finite boundary.

Buckets must therefore straddle the number you care about: if the SLO is 300ms you need boundaries near 250ms and 500ms or the figure is meaningless. Second, quantiles neither average nor add, so `avg(histogram_quantile(...))` across services is arithmetically wrong; aggregate the buckets first and take the quantile last. Third, the function returns NaN when every bucket rate is zero (no traffic in the window), which renders as a gap in Grafana rather than an error, and that gap is often mistaken for an outage. In Prometheus 3.x the same function also accepts a native histogram directly, with no `_bucket` suffix and no `by (le)` clause.

# p99 latency across all instances of a service, last 5 minutes
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{service="api"}[5m])
  )
)

# p50, p95, p99 per endpoint
histogram_quantile(
  0.95,
  sum by (le, endpoint) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Answer

Use **Histogram** in 2026. The key difference: histograms compute quantiles on the SERVER side via `histogram_quantile()`, summaries compute them on the CLIENT side. This matters because summaries can't be aggregated across instances, averaging a p99 from instance A and a p99 from instance B is mathematically meaningless (you cannot average percentiles).

Histograms can, because the buckets sum cleanly. Trade-offs: histograms require you to pick bucket boundaries up-front (typically 10 buckets covering your expected range, e.g. `[5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]`), and they have higher cardinality (one series per bucket). Summaries are cheaper but only work for single-instance metrics.

The 2026 answer: histograms by default, summaries only for true single-instance metrics like 'time since last GC'. Native histograms (Prometheus 2.40+) further reduce this trade-off by using sparse exponential buckets. Cost comparison for the interview: a classical histogram with 10 configured buckets costs 12 series per label combination (11 `_bucket` series including the mandatory `+Inf`, plus `_sum` and `_count`), while a summary with three quantiles costs 5.

Two other summary facts get asked. The client computes quantiles over a sliding time window you configure (`MaxAge`, ten minutes by default in the Go client), so a summary quantile is not 'since process start' and cannot be re-windowed at query time the way a histogram can. And observing into a summary costs meaningfully more CPU than incrementing histogram buckets, because it maintains a streaming quantile estimator per series rather than doing a handful of integer increments, which matters on hot paths at high request rates.

Answer

Service discovery (SD) tells Prometheus what to scrape WITHOUT hardcoding IP addresses. The major mechanisms: (1) **kubernetes_sd_configs**, queries the K8s API for pods, services, endpoints, ingresses, nodes. The Prometheus Operator's `ServiceMonitor` CRD is a thin wrapper around this. (2) **consul_sd_configs**, queries Consul for registered services.

Common at companies running HashiCorp stack. (3) **ec2_sd_configs**, queries AWS for EC2 instances; you typically use tags to filter what's relevant. (4) **file_sd_configs**, Prometheus watches a JSON/YAML file you generate from your own source of truth. Great for hybrid setups or when you don't fit any of the above. (5) **dns_sd_configs**, uses DNS SRV records, useful for simple cases. Each SD mechanism returns a set of `__meta_*` labels that you can mutate with relabeling rules before they become real labels on your series.

One operational detail is worth naming because it is where debugging starts: SD is re-evaluated continuously, so targets appear and disappear with no config reload, and the `/service-discovery` page in the Prometheus UI shows the raw `__meta_*` labels discovered before relabeling alongside the final label set after it. When a target you expect is missing, that single page tells you whether SD never returned it (an RBAC or selector problem) or a `keep`/`drop` rule silently discarded it (a relabeling problem), which are two completely different fixes.

# Kubernetes service discovery for all pods with the prometheus.io/scrape: "true" annotation
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name]
        action: replace
        target_label: instance

Answer

Relabeling rules transform labels, they're the glue between raw service discovery output and the final label set on your scraped metrics. Two phases: (1) **relabel_configs** runs BEFORE scrape, used to filter which targets to scrape, set the `__address__` (host:port), and rename meta labels into real labels. (2) **metric_relabel_configs** runs AFTER scrape but before storage, used to drop noisy series or rename labels. Actions: `keep` (only proceed if regex matches), `drop` (skip if regex matches), `replace` (substitute a value), `labelmap` (bulk rename), `labelkeep`/`labeldrop` (filter labels).

Mastering relabeling is the difference between a junior and senior SRE in interview, it's where the most-asked 'how would you scrape X' questions go. Mechanics that get tested: multiple `source_labels` are concatenated with `separator` (default `;`) before the `regex` is applied, the regex is fully anchored so you rarely need `^` and `$`, and in a `replace` action the `replacement` string (default `$1`) can reference capture groups. There is a third phase most candidates forget, `alert_relabel_configs`, which rewrites labels on alerts on their way to AlertManager and is exactly how you strip a `replica` external label so an HA pair dedupes into one notification instead of paging twice.

Order matters, since rules run top to bottom against the accumulated label set, so a `keep` placed after a `labeldrop` that removed the label it matches on will silently discard every target. And note the limit of `metric_relabel_configs`: dropping series there saves storage, memory and query time, but not scrape bandwidth or parse cost, because the payload was already transferred and decoded before the rules ran.

# Drop a high-cardinality metric series entirely
metric_relabel_configs:
  - source_labels: [__name__]
    regex: "go_gc_.*"
    action: drop

# Rename a meta label from Kubernetes SD
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    target_label: app
  - source_labels: [__meta_kubernetes_pod_label_version]
    target_label: version

Answer

Recording rules pre-compute frequently-used or expensive queries and store the result as a new time series. Use them when: (1) A dashboard query takes > 1 second to render, pre-compute the result so the dashboard is instant. (2) The same complex expression appears in many alerts or panels, compute it once. (3) Quantile calculations over wide time ranges, those touch a lot of bucket data. Naming convention: `level:metric:operation`, e.g. `instance:http_requests:rate5m`.

Recording rules run on Prometheus's evaluation interval (typically 15-30s) and write results back to the TSDB. The trade-off: extra storage and a slight delay (one eval interval) vs much faster queries. Details that matter in production: rules inside a single group evaluate sequentially in file order, so a rule can safely depend on one declared above it, while separate groups evaluate in parallel and must never depend on each other.

Each group has its own `interval`, and if an evaluation overruns it Prometheus logs `rule group evaluation took longer than the interval` and you get visible gaps in the recorded series. Recording rules are also not retroactive: create one today and it has zero history, which is why a team adding SLO rules has to wait out the full compliance window before the dashboard means anything, and why `promtool tsdb` backfill exists. Watch `prometheus_rule_group_last_duration_seconds` against `prometheus_rule_group_interval_seconds` plus `prometheus_rule_evaluation_failures_total`, and validate every file with `promtool check rules` in CI.

# rules/recording.yml
groups:
  - name: http_request_rules
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration:p99
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Answer

AlertManager is a separate process that handles alerts fired by Prometheus. Its job is to dedupe, group, route, and deliver. (1) **Grouping**, multiple alerts with the same labels (e.g. same alertname + cluster) are batched into one notification, so you don't get spammed with 100 pages when a whole cluster goes down. (2) **Routing tree**, incoming alerts walk a tree of matchers; each matcher can route to a different receiver (Slack, PagerDuty, email, webhook). Use `continue: true` to fall through. (3) **Inhibition**, suppress a low-severity alert when a higher-severity one is firing (e.g. if the whole cluster is down, don't page about a single pod). (4) **Silences**, manually mute alerts for a time window during planned maintenance.

Silences are matched by label and created via the UI or API. The notification template is fully customizable per receiver; most teams template Slack messages to include runbook links. Be able to explain the three timing knobs apart, because candidates routinely blur them: `group_wait` (default 30s) is how long to buffer the first notification for a brand-new group so related alerts arrive in one message, `group_interval` (default 5m) is the minimum gap before sending an update about a group that already notified, and `repeat_interval` (default 4h) is how often to re-notify about an alert that is still firing.

HA is gossip-based rather than leader-elected: two or more AlertManagers form a mesh via `--cluster.peer` and coordinate so exactly one sends each notification, and Prometheus is deliberately configured with every peer in its `alerting` block so it fires each alert at all of them. The consequence to mention is that silences and notification logs propagate through gossip, so a silence created seconds before an alert fires can occasionally be missed by a peer.

# alertmanager.yml
route:
  group_by: [alertname, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: default-slack
  routes:
    - match: { severity: critical }
      receiver: pagerduty-oncall
    - match: { team: payments }
      receiver: payments-slack

inhibit_rules:
  - source_match: { severity: critical, alertname: ClusterDown }
    target_match: { severity: warning }
    equal: [cluster]

receivers:
  - name: default-slack
    slack_configs:
      - api_url: $SLACK_WEBHOOK
        channel: "#alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: $PAGERDUTY_KEY

Answer

A good alert has four parts: (1) **A meaningful expression** that fires only on user-impacting issues, not on every CPU spike. SRE wisdom: alert on symptoms (latency, errors, saturation), not causes (high memory). (2) **A `for` clause** to require the condition to persist before firing, typically 5m for noisy metrics, 1m for binary up/down. This prevents flapping. (3) **Labels** for severity (critical/warning) and team routing. (4) **Annotations** with `summary`, `description`, and a `runbook_url` so the on-call engineer knows what to do at 3 AM without paging the original author.

The four golden signals (latency, traffic, errors, saturation) are the starting template. Two timing details a senior interviewer expects. `for:` is not a notification delay, it is a requirement that the expression return a value for that exact series at every single evaluation in the window; one missing evaluation drops the alert back to inactive and restarts the clock, which is why intermittently-empty expressions never fire even though the graph looks bad. And the real time-to-page is `for` plus AlertManager's `group_wait` (30s by default), so a 5-minute `for` on a critical alert is closer to five and a half minutes of unnoticed outage.

Ratio alerts also need a traffic floor, or a 100% error rate computed from two requests at 3 AM will page you: gate it with `and sum by (service) (rate(http_requests_total[5m])) > 1`. Finally, check the alert in the `/alerts` page's pending state before shipping it, because a rule that is syntactically valid can still match zero series.

# rules/alerts.yml
groups:
  - name: api_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total[5m]))
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "5xx error rate above 5% on {{ $labels.service }}"
          description: "{{ $labels.service }} has had {{ $value | humanizePercentage }} errors for 5 minutes."
          runbook_url: "https://wiki/runbooks/high-error-rate"

Answer

PromQL aggregation operators reduce a vector across labels: `sum`, `min`, `max`, `avg`, `count`, `stddev`, `stdvar`, `topk`/`bottomk`, `quantile`, `group`. They all take an optional `by (labels...)` or `without (labels...)` clause to control which labels survive the aggregation. By default, ALL labels are dropped, that's almost never what you want. `topk(5, ...)` and `bottomk(5, ...)` are useful for surfacing the top contributors in dashboards.

Combining: `sum by (status) (rate(...))` is the most common pattern, get the per-second rate per status code summed across all instances. Gotchas worth naming. `count()` counts series, `count_values()` counts series grouped by their sample value, and `count_over_time()` counts samples inside a range: three different functions with confusingly similar names. `avg()` across instances is almost always the wrong choice for latency because it hides the one slow replica, so prefer `max()` or aggregate the histogram buckets and take a quantile. `topk()` is re-evaluated at every step of a range query, so the membership of the top five changes per timestamp and the graph flickers; for a stable list use it on an instant query or behind a recording rule. Aggregations drop the metric name, which is why `sum(a) / sum(b)` needs no matching modifier while a bare `a / b` does. `by ()` with an empty label list collapses everything into one series, and `group()` is the cheap way to say 'I want the label set, not the value', which matters when the operand is only there to filter another vector.

# Sum across all instances, keep status label
sum by (status) (rate(http_requests_total[5m]))

# Top 5 noisiest endpoints by error rate
topk(5, sum by (endpoint) (rate(http_requests_total{status=~"5.."}[5m])))

# Average latency per service
avg by (service) (rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))

# All labels except instance (sum across instances)
sum without (instance) (rate(http_requests_total[5m]))

Answer

Prometheus's TSDB stores data in 2-hour blocks on disk. Within each block, samples are compressed using Gorilla-style XOR encoding (typically 1-3 bytes per sample). Each block is immutable once written; the active block (currently being written to) lives in a WAL (write-ahead log) until it's flushed.

Compaction merges adjacent blocks over time. Retention is controlled by `--storage.tsdb.retention.time` (default 15d) or `--storage.tsdb.retention.size`. For most production setups, 15-30 days of local retention is enough for incident response, and you ship to long-term storage (Thanos, Mimir) for anything beyond that.

Disk sizing rule: ~1-2 bytes per sample × samples per second × retention seconds. For 10k series scraped every 15s for 30 days: 10000 × (1/15) × 60×60×24×30 × 1.5 bytes ≈ 25 GB. Failure modes to name, because this is where the follow-ups live.

An unclean shutdown replays the WAL on startup, and on a large head block that replay can take several minutes during which `/-/ready` returns 503 and the instance serves nothing, so an HA pair must never be restarted simultaneously. If both `retention.time` and `retention.size` are set, whichever triggers first wins, and size-based retention counts the whole data directory including the WAL. Deleting block directories by hand while the process is running corrupts the index: use `POST /api/v1/admin/tsdb/delete_series` followed by `clean_tombstones`, both of which require `--web.enable-admin-api`. Note that a deletion hides the series from queries immediately but only frees disk after the next compaction, which is why 'I deleted the bad metric and the disk is still full' is a common and correct observation.

Answer

Prometheus's local storage caps at single-node disk and ~15-30 days of retention. For longer retention or multi-cluster federation, you use **remote write**, Prometheus streams samples to a long-term storage system over HTTP. The three major options in 2026: (1) **Thanos**, a CNCF project that sidecars Prometheus, uploads 2-hour blocks to S3/GCS, and provides a query layer that fans out across all Prometheus + object storage.

Simplest to start with. (2) **Cortex**, multi-tenant, horizontally scalable, ingester-based. CNCF, used by Grafana Cloud's older tier. (3) **Mimir**, Grafana Labs' fork of Cortex, optimized for huge scale (billions of series), and used by Grafana Cloud now. Mimir scales the best and has the most active development, but Thanos is easier to operate for under-100M-series setups.

Indian unicorns at scale (Razorpay, Swiggy) typically run Thanos or Mimir behind their Prometheus fleet. Know the remote-write tuning surface, because that is the practical follow-up. The sending queue is configured under `queue_config` with `max_shards`, `min_shards`, `capacity` and `max_samples_per_send`, and when the receiver slows down you watch `prometheus_remote_storage_samples_pending`, `prometheus_remote_storage_samples_dropped_total` and `prometheus_remote_storage_shards`.

Sustained growth of shards up to `max_shards` means the receiver, not Prometheus, is the bottleneck, and raising `max_shards` there just moves the queue. Remote write reads from the WAL rather than from memory, so a receiver outage shorter than your WAL retention (2 hours by default) is replayed with no data loss once the endpoint recovers, and a longer one drops samples permanently.

Answer

Federation is a mechanism where one Prometheus server scrapes selected time series from another Prometheus server's `/federate` endpoint. Two valid patterns: (1) **Hierarchical federation**, a global Prometheus aggregates pre-computed recording rules from per-datacenter or per-cluster Prometheus instances. This is the original design and still works well for top-level dashboards. (2) **Cross-service federation**, pulling specific metrics across team boundaries.

Hard rule: NEVER federate raw metrics. The `/federate` endpoint is meant for already-aggregated series (typically the output of recording rules); pulling raw high-cardinality data through federation kills both Prometheus instances. In 2026, federation is being replaced by remote_write to Thanos/Mimir for most use cases, federation is still useful when you need a simple aggregation for a single executive dashboard but don't want a full long-term storage stack.

Two implementation details make or break it. `honor_labels: true` is mandatory on a federation job, otherwise the parent overwrites the child's `job` and `instance` labels with its own and every federated series collapses into one. And `/federate` returns the most recent sample for each matched series along with its original timestamp, so a parent scraping every 30s against a child scraping every 15s will silently skip samples, which makes `increase()` computed on the parent lower than the same query on the child. Federation is also synchronous and unbounded: a `match[]` selector that resolves to a million series will pin the child's HTTP handler and blow through the parent's `scrape_timeout`, taking out both instances.

# Global Prometheus federating aggregated metrics from cluster-level Prometheis
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      match[]:
        - '{__name__=~"job:.*"}'   # only recording rule output (job:metric:operation)
        - '{__name__=~"up"}'
    static_configs:
      - targets:
          - 'prometheus-cluster-mumbai:9090'
          - 'prometheus-cluster-singapore:9090'

Answer

The four golden signals (from Google's SRE book) are: (1) **Latency**, time to serve a request, split between successful and failed requests since failures often return faster than successes and would otherwise hide the real problem. Measure with a histogram: `histogram_quantile(0.99, ...)` for p99. (2) **Traffic**, demand on the system, typically request rate. Measure with a counter: `rate(http_requests_total[5m])`. (3) **Errors**, rate of failed requests, broken down by category (HTTP 5xx, business-logic errors, policy-violation errors). (4) **Saturation**, how full your service is, relative to its constraint.

For an HTTP service, that's typically queue depth (gauge), worker pool utilization, or CPU saturation. Pick the resource closest to running out first. The RED method (Rate, Errors, Duration) is a derivation focused on services; the USE method (Utilization, Saturation, Errors) is for resources.

Together they cover almost every alerting question. The dashboards you import from grafana.com almost always follow one of these structures, and most interview discussions about 'what alerts would you set up' boil down to picking SLOs against these four indicators. Interviewers usually push hardest on saturation, because it is the one people hand-wave.

Saturation is not CPU utilisation, it is how close the constraining resource is to the point where latency goes non-linear, and the constraint differs per service: for a Go API it is often goroutine count or `container_cpu_cfs_throttled_seconds_total` rather than raw CPU, for a queue worker it is backlog depth divided by drain rate (which yields time-to-full, the only actionable form), and for a database-bound service it is connection pool wait time. A good answer also names the one signal the four miss, which is correctness: a service can be fast, available and error-free while returning wrong data, and that needs a business-level metric rather than an infrastructure one.

Answer

That message is the Go HTTP client giving up: the scrape ran past `scrape_timeout` (10s by default, and never permitted to exceed `scrape_interval`). Prometheus keeps nothing partial from a timed-out scrape, so the series just gets holes, `rate()` bridges them, and the symptom often reaches you as a jagged dashboard rather than a page. Start with the four synthetic series Prometheus writes per scrape. `scrape_duration_seconds` tells you whether the target is slow constantly or only in bursts, `scrape_samples_scraped` tells you whether the payload has grown, and the gap between `scrape_samples_scraped` and `scrape_samples_post_metric_relabeling` shows how much you are transferring and parsing only to throw away in drop rules.

Then reproduce by hand from inside the network: `time curl -s -o /dev/null http://target:9100/metrics`. Fast from your laptop but slow from the Prometheus pod means DNS, NetworkPolicy or a service mesh sidecar, not the exporter. Ranked by how often they are the actual cause: an exporter collector that queries the underlying system synchronously on every scrape (node_exporter's `--collector.systemd`, mysqld_exporter's `perf_schema` collectors, a postgres_exporter custom query doing a sequential scan), a target whose series count has quietly grown into six figures, CPU throttling on the target pod visible as `container_cpu_cfs_throttled_seconds_total`, and a proxy in front with its own shorter timeout. The fix is to disable the expensive collector or cache it inside the exporter, and only then raise `scrape_timeout` and `scrape_interval` together, because raising the timeout alone eventually collides with the interval and Prometheus refuses the config at load.

# Is the job slow, or just big?
topk(10, scrape_duration_seconds)
topk(10, scrape_samples_scraped)

# How much is being transferred and parsed, then dropped?
scrape_samples_scraped - scrape_samples_post_metric_relabeling

# Which jobs are currently failing their scrape
sum by (job) (up == 0)

# Reproduce from inside the cluster, not from your laptop
#   kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- \
#     sh -c 'time curl -s -o /dev/null http://mysqld-exporter:9104/metrics'

# Give a genuinely slow exporter room, and cap the blast radius
scrape_configs:
  - job_name: 'mysqld'
    scrape_interval: 60s
    scrape_timeout: 45s     # must stay <= scrape_interval
    sample_limit: 50000     # fail loudly instead of poisoning the TSDB
    static_configs:
      - targets: ['mysqld-exporter:9104']

Answer

`promtool test rules tests/*.yml` evaluates a rule file against synthetic data with no Prometheus server running, which is the answer when an interviewer asks how you know an alert works before it is needed at 3 AM. The test file names the `rule_files` to load, an `evaluation_interval`, and a list of `tests`. Each test declares `input_series` in the expanding notation, where `0+10x60` means start at 0 and add 10 sixty times, `_` is a deliberately missing sample, and `stale` writes an explicit staleness marker so you can test what happens when a target disappears.

You then assert with `alert_rule_test`, which takes an `eval_time` and the exact set of alerts that should be firing including every label and rendered annotation, or with `promql_expr_test`, which checks the value an expression returns. Assertions are exact and the failure output is a diff, so a label typo or a differently rendered `{{ $value | humanizePercentage }}` fails the build. The bugs this catches are the ones a syntax check never will: a `for:` window that the metric never satisfies continuously, a ratio that returns NaN instead of firing because the denominator went to zero, a `severity` label that no AlertManager route matches, and a recording rule whose output name collides with a metric that already exists.

Run it in CI next to `promtool check rules`, which validates YAML and PromQL but asserts nothing about behaviour. It stops at 'the alert fired': to test routing you need `amtool config routes test` against your AlertManager config separately.

# rules/tests/api_alerts_test.yml
rule_files:
  - ../alerts.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      # 12 5xx/min against 108 2xx/min = a steady 10% error ratio
      - series: 'http_requests_total{service="api",status="500"}'
        values: '0+12x20'
      - series: 'http_requests_total{service="api",status="200"}'
        values: '0+108x20'

    promql_expr_test:
      - expr: sum(rate(http_requests_total{status="500"}[5m]))
        eval_time: 10m
        exp_samples:
          - labels: '{}'
            value: 0.2          # 12 per 60s

    alert_rule_test:
      - eval_time: 3m           # `for: 5m` has not elapsed yet
        alertname: HighErrorRate
        exp_alerts: []
      - eval_time: 15m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              severity: critical
              team: backend
              service: api
            # promtool compares ALL annotations; list every one
            exp_annotations:
              summary: "5xx error rate above 5% on api"

# CI step:
#   promtool check rules rules/*.yml
#   promtool test rules rules/tests/*_test.yml

Answer

A binary operator between two instant vectors matches samples by their full label set, with the metric name dropped first. One-to-one is the default: for every series on the left, PromQL looks for exactly one series on the right with an identical label set, and any series without a partner silently disappears from the result. That is why `a / b` frequently returns fewer series than either operand and why people conclude their data is missing when the real problem is a stray `pod` label on one side. `on (labels...)` narrows matching to just the listed labels, `ignoring (labels...)` matches on everything except them, so `rate(errors[5m]) / ignoring(status) rate(total[5m])` works when `status` is the only difference.

Many-to-one matching must be declared with `group_left` or `group_right` pointing at the higher-cardinality side, otherwise you get `multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)`. The dominant real use is the info-metric join: `kube_pod_labels` and `target_info` always have the value 1 and exist only to carry labels, so `* on (pod, namespace) group_left(label_team) kube_pod_labels` attaches team ownership to a metric that never had it, with the labels in the parentheses being the extras copied across. Two follow-ups: `found duplicate series for the match group` means the match key is not unique on the many side, almost always because `namespace` or `cluster` was left out; and arithmetic operators drop the metric name, so a join yields an unnamed series that Grafana renders as `{}` until you wrap it in `label_replace()` or a recording rule. Set operators (`and`, `unless`, `or`) keep the left side's name but do not accept `group_left` at all.

# One-to-one: label sets must match exactly (metric name is dropped)
rate(http_errors_total[5m]) / rate(http_requests_total[5m])

# Per-status share of traffic: only `status` differs, so ignore it
sum by (service, status) (rate(http_requests_total[5m]))
  / ignoring(status) group_left
sum by (service) (rate(http_requests_total[5m]))

# Info-metric join: attach the owning team from kube_pod_labels
sum by (pod, namespace) (rate(container_cpu_usage_seconds_total[5m]))
  * on (pod, namespace) group_left(label_team)
    kube_pod_labels{label_team!=""}

# Same idea for OTel resource attributes carried on target_info
up * on (job, instance) group_left(k8s_cluster_name) target_info

# Set operator: pods that are firing AND not already silenced upstream
ALERTS{alertstate="firing"} unless on (pod) kube_pod_status_phase{phase="Pending"}

# Errors you will see:
#   multiple matches for labels: many-to-one matching must be explicit
#     -> add group_left/group_right on the high-cardinality side
#   found duplicate series for the match group {pod="api-0"}
#     -> the match key is not unique; add namespace or cluster

Answer

A series that stops being exposed does not go to zero, it stops existing, and every expression referencing it returns an empty vector. An alert with an empty expression is not firing, so 'the exporter died' produces total silence, which is the most common hole in a new alerting setup. `absent(up{job="payments"})` returns the scalar 1 carrying the labels from the selector when that selector matches nothing, and returns nothing at all when it matches something, making it an inverter you can alert on. Its limitation is that it can only synthesise labels it can read literally out of the expression, so `absent(up{job=~"payment.*"})` produces an alert with no `job` label and therefore no AlertManager routing. `absent_over_time(metric[15m])` is usually the one you want, because it tolerates a single missed scrape instead of firing on one flaky evaluation.

Underneath both sits staleness handling: since Prometheus 2.0, when a target disappears or stops exposing a series, Prometheus writes an explicit staleness marker, so the series stops resolving at the very next evaluation rather than lingering for the 5-minute lookback delta the way 1.x did. That is also why `up` for a deleted target vanishes entirely instead of sitting at 0, and why `up == 0` cannot catch a target that dropped out of service discovery. A complete answer names three layers: `up == 0` for a discovered target failing its scrape, `absent_over_time()` for a target or metric that disappeared, and a permanently firing Watchdog alert routed to an external heartbeat service such as Dead Man's Snitch, so that silence from the whole pipeline is itself detectable.

# 1) Discovered target failing its scrape
- alert: TargetDown
  expr: up{job="payments"} == 0
  for: 5m

# 2) Target gone from service discovery: up == 0 cannot see this,
#    because the series itself no longer exists
- alert: PaymentsExporterMissing
  expr: absent_over_time(up{job="payments"}[15m])
  for: 5m
  labels: { severity: critical, team: payments }
  annotations:
    summary: "No successful scrape of job=payments in 15 minutes"

# 3) A healthy target stopped exposing one business metric
- alert: OrdersMetricMissing
  expr: absent_over_time(orders_created_total[30m])
  for: 10m

# 4) Watchdog: always firing, routed to an external heartbeat.
#    Silence from it means Prometheus or AlertManager is down.
- alert: Watchdog
  expr: vector(1)
  labels: { severity: none }

# absent() only invents labels it can read literally:
#   absent(up{job="payments"})    -> {job="payments"}   routable
#   absent(up{job=~"payment.*"})  -> {}                 unroutable

Answer

Prometheus exposes its own `/metrics`, and the standard setup has every instance scrape itself plus a small meta-monitoring Prometheus (or simply the HA peer) scraping all the others, because an instance cannot page about its own death. The metrics that matter fall into four groups. Ingestion: `prometheus_tsdb_head_series` is your active series count, alerted on growth rate rather than an absolute threshold since the fleet legitimately grows, and `prometheus_target_scrape_pool_exceeded_sample_limit_total` counts targets rejected for blowing `sample_limit`.

Rule evaluation: `prometheus_rule_evaluation_failures_total`, plus `prometheus_rule_group_last_duration_seconds` compared against `prometheus_rule_group_interval_seconds`, because a group overrunning its interval leaves gaps in recorded series and delays every alert built on top of them. Notification delivery: `prometheus_notifications_dropped_total` and `prometheus_notifications_queue_length` against `prometheus_notifications_queue_capacity` mean Prometheus cannot reach AlertManager, which is an outage of your alerting rather than of your application. Remote write, if you use it: `prometheus_remote_storage_samples_pending`, `prometheus_remote_storage_samples_failed_total` and `prometheus_remote_storage_shards` against `prometheus_remote_storage_shards_max`.

Add `prometheus_config_last_reload_successful == 0`, which catches the nasty case where someone pushed a broken config, the reload was rejected, and the process is cheerfully serving the previous one while the new alerts simply do not exist. On the process side, `process_resident_memory_bytes` against the container limit matters because the usual death is an OOM kill during head compaction, and `prometheus_tsdb_compactions_failed_total` with `prometheus_tsdb_wal_corruptions_total` catch disk trouble early. The kube-prometheus-stack already ships all of these, so the strongest interview answer is naming what each rule protects you from rather than reciting the list.

# Meta-monitoring: run these against your OTHER Prometheus instances
groups:
  - name: prometheus_meta
    rules:
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 5m

      - alert: PrometheusRuleEvaluationsFailing
        expr: increase(prometheus_rule_evaluation_failures_total[10m]) > 0

      - alert: PrometheusRuleGroupOverrunning
        expr: |
          prometheus_rule_group_last_duration_seconds
            > prometheus_rule_group_interval_seconds
        for: 15m

      - alert: PrometheusCannotReachAlertmanager
        expr: increase(prometheus_notifications_dropped_total[5m]) > 0
        for: 5m

      - alert: PrometheusRemoteWriteSaturated
        expr: |
          prometheus_remote_storage_shards
            >= prometheus_remote_storage_shards_max
        for: 15m

      - alert: PrometheusSeriesGrowthProjected
        expr: |
          predict_linear(prometheus_tsdb_head_series[6h], 86400)
            > 1.5 * prometheus_tsdb_head_series
        for: 1h
        annotations:
          summary: "Active series projected to grow over 50% in 24h"

Answer

Out of the box Prometheus has no TLS and no authentication: anything that can reach port 9090 reads every metric you hold, and if `--web.enable-admin-api` is on it can also delete series through `/api/v1/admin/tsdb/delete_series`. Since 2.24 the built-in answer is a separate web config passed as `--web.config.file=web.yml`, holding `tls_server_config` (cert, key, `min_version`, and `client_auth_type` plus `client_ca_file` for mTLS) and `basic_auth_users`, a map of username to bcrypt hash generated with `htpasswd -nBC 12 "" | tr -d ':\n'`. The same file format is understood by AlertManager, node_exporter and the other official exporters, which is the point: one mechanism instead of a reverse proxy bolted onto each component.

Name what it does not give you, because that is the follow-up: there is no authorisation, so every authenticated caller sees everything, and there is no per-tenant scoping. SSO, RBAC or per-team views mean an oauth2-proxy in front, or moving queries onto the Thanos or Mimir multi-tenant path. The scrape direction is configured separately in each job with `scheme: https`, a `tls_config` block, and one of `bearer_token_file`, `authorization`, `basic_auth` or `oauth2`; on Kubernetes that token is `/var/run/secrets/kubernetes.io/serviceaccount/token`, which is exactly how kubelet and cAdvisor scrapes authenticate. Three more hardening items: leave `--web.enable-admin-api` off and `--web.enable-lifecycle` off unless a config-reloader sidecar needs it, bind `--web.listen-address` to a private interface, and pass credentials as mounted files through the `*_file` variants rather than inline strings, since Prometheus does not expand environment variables in scrape configs.

# web.yml, passed as --web.config.file=/etc/prometheus/web.yml
tls_server_config:
  cert_file: /etc/prometheus/tls/tls.crt
  key_file:  /etc/prometheus/tls/tls.key
  min_version: TLS12
  client_auth_type: RequireAndVerifyClientCert   # mTLS, optional
  client_ca_file: /etc/prometheus/tls/ca.crt

basic_auth_users:
  # generate with: htpasswd -nBC 12 "" | tr -d ':\n'
  observability: $2y$12$hUEBQzQ0h1s5rTQNvY1kOe9tqL5xQnQ3rZ8Wn0v2yQ1sX7pC9mKzS

# ---------------------------------------------------------------
# The scrape direction is configured per job, NOT in web.yml
scrape_configs:
  - job_name: 'kubernetes-cadvisor'
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node

  - job_name: 'partner-api'
    scheme: https
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/partner-password

# Verify before rolling out:
#   promtool check web-config web.yml
#   curl -u observability:PASS --cacert ca.crt https://prometheus:9090/-/healthy

Answer

The 2026 reference architecture for high-scale Prometheus looks like this: (1) **One Prometheus per cluster** (or even per namespace at huge scale), Prometheus is not designed to be a global tool; keep blast radius small. (2) **Local retention 15-30 days** for incident response. (3) **Remote write to a global tier**, Thanos or Mimir, backed by S3/GCS, with retention of 1-2 years. (4) **Global query layer**, Thanos Querier or Mimir Querier deduplicates across replicas and provides a single PromQL endpoint that spans all clusters. (5) **Per-cluster AlertManager in HA pairs** to ensure alerts fire even if a Prometheus dies. (6) **Federation only at the recording-rule level**, never federate raw metrics, it doesn't scale. For replicated Prometheus HA: run two identical Prometheus instances scraping the same targets, and let Thanos/Mimir dedupe on query. At Indian scale: Swiggy/Razorpay run ~50-200 Prometheus instances writing into Thanos/Mimir, processing billions of samples per day.

Two things distinguish a real answer here. External labels are the backbone: every Prometheus sets `external_labels` with `cluster`, `region` and a replica identifier such as `__replica__` or `prometheus_replica`. The first two make a cross-cluster query meaningful, and the last is precisely what Thanos Querier strips via `--query.replica-label` to collapse an HA pair back into one series, which is why the two replicas must be otherwise label-identical.

Sharding is by target, not by metric name: use `hashmod` on `__address__` in `relabel_configs` so each shard scrapes a deterministic slice, because splitting by metric name breaks any query that joins two metrics living in different shards. Plan the query fan-out too, since a global Querier hitting fifty stores is only as fast as its slowest one unless you set `--store.response-timeout` and decide explicitly whether partial responses are acceptable.

Answer

Native histograms (stable in Prometheus 3.x, introduced experimentally in 2.40) replace the classical bucket-per-series model with a single time series that contains sparse exponential buckets internally. Trade-offs: (1) **Cardinality**: classical histogram with 10 buckets × 1000 endpoints × 5 services = 50,000 series; native histogram = 1000 × 5 = 5,000 series. Massive reduction. (2) **Auto-bucketing**: the schema parameter controls resolution; you don't have to pre-define cuts. (3) **Better quantile accuracy**: exponential buckets give consistent relative precision across the entire range.

To use: client library opts in (`prometheus_client` 0.20+ in Python, `prometheus/client_golang` 1.16+ in Go), and Prometheus enables ingest with `--enable-feature=native-histograms`. The wire format also benefits, OpenTelemetry's exponential histogram maps cleanly to Prometheus native histograms, which is why the OTel collector integration in 2026 is converging on native histograms as the default. Still a few rough edges in 2026: not all client libraries support them yet, and some long-tail PromQL functions need to be re-implemented.

Two operational caveats decide whether a migration succeeds. Native histograms are not representable in the classic text exposition format, so the scrape must negotiate protobuf: you set `scrape_protocols` to prefer `PrometheusProto` and verify the target actually offers it, otherwise you silently keep getting classic buckets. And remote write must be version 2.0, or a receiver that explicitly understands native histograms, or the samples are dropped at the boundary, which is the classic 'the quantile works in the local Prometheus but nothing arrives in Mimir' bug. The safe migration is dual emission: keep classic buckets alongside native histograms for one full retention window so dashboards, recording rules and SLO burn-rate alerts can be cut over and compared before the old series are dropped.

# 1) Client opts in (Go, prometheus/client_golang)
#      NativeHistogramBucketFactor:     1.1   // ~10% relative precision
#      NativeHistogramMaxBucketNumber:  160
#      NativeHistogramMinResetDuration: time.Hour

# 2) Prometheus must negotiate protobuf to receive them
global:
  scrape_protocols:
    - PrometheusProto
    - OpenMetricsText1.0.0
    - PrometheusText0.0.4

# 3) Query: no _bucket suffix, no `by (le)`
histogram_quantile(0.99, rate(http_request_duration_seconds[5m]))

# 4) The classic equivalent, for comparison during migration
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# 5) Native-histogram accessors replace _sum / _count
histogram_count(rate(http_request_duration_seconds[5m]))
histogram_sum(rate(http_request_duration_seconds[5m]))

Answer

Workflow for diagnosing cardinality explosions: (1) **Find the worst metric names**, `topk(20, count by (__name__)({__name__=~".+"}))`. This shows you which metric has the most series. (2) **Within that metric, find the worst label**, `topk(20, count by (LABEL)(METRIC_NAME))` for each suspected label. The label whose unique values explode is your culprit. (3) **Check the recent rate of series growth**, `prometheus_tsdb_head_series` over time.

A sudden step-change tells you when the bad metric was deployed. (4) **Use the /api/v1/status/tsdb endpoint**, built-in cardinality stats by series, label name, label value, memory. (5) **Apply a fix**, drop the offending label at the source (preferred), or use `metric_relabel_configs` with `action: labeldrop` as a hotfix. The damage is permanent for already-written data, you wait for the retention window to free the storage. The 2026 best practice: enforce cardinality limits per scrape job with `sample_limit` and `label_limit` to prevent a single bad service from killing the cluster.

Two more tools belong in the answer. `promtool tsdb analyze /path/to/data` runs offline against the block directory and prints the highest-cardinality metric names, the label names with the most unique values, and the worst label pairs, which is the only route that works when the server is too sick to answer a query at all. And `count({__name__=~".+"})` gives you total active series as a single number, so you can watch it fall as drop rules land instead of guessing. Prevention is the part senior interviewers actually care about: `sample_limit` per job so a bad service fails its own scrape rather than the cluster, a CI check that fails the merge when a new metric introduces an unbounded label, and a standing alert on the growth rate of `prometheus_tsdb_head_series` rather than an absolute threshold, since the absolute number legitimately drifts upward as the fleet grows.

# Diagnose worst metrics
topk(20, count by (__name__)({__name__=~".+"}))

# Within http_requests_total, which label is exploding?
topk(20, count by (endpoint)(http_requests_total))
topk(20, count by (user_id)(http_requests_total))  # this is probably the problem

# Hotfix: drop the bad label at scrape time
metric_relabel_configs:
  - regex: "user_id"
    action: labeldrop

# Permanent fix: stop labeling per-user, sample at the application layer

Answer

The Prometheus/OTel story has been converging since 2023. In 2026, the dominant pattern is: applications emit OTLP metrics via OpenTelemetry SDKs, the OTel collector receives them, and the collector either (a) exposes a Prometheus-scrapable endpoint via the `prometheusexporter`, or (b) writes directly to a Prometheus remote_write endpoint via `prometheusremotewriteexporter`. The OTel collector is increasingly used as a metric processing pipeline, filtering, transforming, and routing, before data lands in Prometheus.

Key wins: (1) one SDK for traces + metrics + logs across all languages, (2) language SDKs no longer need to expose `/metrics` themselves, (3) cardinality control via OTel processors before write. Trade-offs: extra component to operate, mapping rules between OTel resource attributes and Prometheus labels can be tricky, and exponential histograms only map cleanly with native histograms enabled. At GoodSpace, we're using OpenTelemetry collectors to ingest into SignOz, which itself uses a Prometheus-compatible backend.

Two mechanics decide whether the pipeline actually works. Prometheus 3.x can receive OTLP directly: start it with `--web.enable-otlp-receiver` and the collector posts to `/api/v1/otlp/v1/metrics`, with no scrape in the path at all. Names are normalised on the way in, so `http.server.request.duration` becomes `http_server_request_duration_seconds`, and the `otlp` block in `prometheus.yml` controls that through `translation_strategy` and `promote_resource_attributes`, which selects the few resource attributes that become real labels; everything else lands on a `target_info` series you join at query time with `group_left`.

The second trap is temporality. Prometheus stores cumulative counters only, so an SDK exporting delta temporality needs the collector's `deltatocumulative` processor in front of it. The symptom when nobody does that is `rate()` returning near-zero or wildly spiky numbers, because every delta sample looks like a counter reset to the query engine.

# OTel collector writing straight into Prometheus (remote write)
exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
    resource_to_telemetry_conversion:
      enabled: false   # do NOT flatten every resource attr into a label

processors:
  deltatocumulative: {}          # required for delta-temporality SDKs
  filter/drop_noise:
    metrics:
      exclude:
        match_type: regexp
        metric_names: ['^otelcol_.*']

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [deltatocumulative, filter/drop_noise, batch]
      exporters: [prometheusremotewrite]

# Alternative in Prometheus 3.x: skip the exporter, push OTLP in directly
#   prometheus --web.enable-otlp-receiver
#   collector endpoint: http://prometheus:9090/api/v1/otlp/v1/metrics

# Joining promoted resource attributes back on at query time
#   sum by (k8s_cluster_name) (
#     rate(http_server_request_duration_seconds_count[5m])
#     * on (job, instance) group_left(k8s_cluster_name) target_info
#   )

Answer

An SLO is a target for a Service Level Indicator (SLI) over a window, e.g. '99.9% of requests succeed over 28 days'. Implementation in Prometheus: (1) **Define the SLI** as a PromQL expression, typically `good_events / total_events`. (2) **Compute the error budget** = 1 - SLO. For 99.9%, budget = 0.1% = 0.001. (3) **Use multi-window, multi-burn-rate alerts**, a page that fires when you're burning 14× the budget over 1 hour AND 2× the budget over 6 hours, etc. This is the Google SRE pattern.

The Prometheus operator project Sloth (or Pyrra) generates the rule files for you given a YAML SLO definition, they expand into recording rules (for fast computation), alerting rules (for burn rate), and Grafana panels. The hardest part is picking the right SLI: latency p95 < 500ms? success rate > 99.9%? Make sure it's user-visible, 'CPU < 80%' is NOT an SLI, 'request error rate < 0.1%' is.

Memorise the burn-rate table, because interviewers ask for the numbers: page at 14.4x burn over a 1h long window guarded by a 5m short window, page at 6x over 6h guarded by 30m, and open a ticket rather than a page at 3x over 1d and 1x over 3d. The short window exists so the alert clears within minutes of a burst ending instead of staying lit for the entire long window. Compute all of it from recording rules, never from raw counters: a 30-day ratio over `http_requests_total` re-reads millions of samples on every evaluation, which is why Sloth emits `slo:sli_error:ratio_rate5m` through `ratio_rate3d` and the alert expressions only ever touch pre-aggregated series. Keep `for:` short (2m is typical) on burn-rate alerts, because the multi-window structure already supplies the damping that `for:` provides elsewhere.

# Error budget burn rate alert (1h + 6h windows)
- alert: ErrorBudgetBurn
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14 * 0.001
      and
      sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) > 6 * 0.001
    )
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Burning 14x error budget over 1h"

Answer

Prometheus 3.0 landed in November 2024 and 3.5 was designated the LTS of the line, which is what most enterprises pin to. The headline additions: a rewritten React UI (the old one survives behind `--enable-feature=old-ui`), Remote Write 2.0 selected per endpoint with `protobuf_message: io.prometheus.write.v2.Request`, a native OTLP ingestion endpoint at `/api/v1/otlp/v1/metrics` behind `--web.enable-otlp-receiver`, UTF-8 metric and label names, and agent mode promoted out of experimental. The breaking changes are the interview material.

Range selectors became left-open and right-closed, so a sample sitting exactly on the `t-5m` boundary is no longer included; the practical effect is that `rate()` and `increase()` values move very slightly and any `promtool test rules` file asserting exact floats needs re-baselining. UTF-8 names mean the old character restrictions are gone, but selecting such a series needs the quoted syntax `{"http.server.duration", job="api"}`, and you can keep the old behaviour with `metric_name_validation_scheme: legacy`. The `le` label on classic histograms and `quantile` on summaries are normalised to a canonical float form on ingestion, which quietly breaks any dashboard or recording rule that hardcodes `le="1"` instead of `le="1.0"`. `holt_winters` was renamed `double_exponential_smoothing` and now requires `--enable-feature=promql-experimental-functions`.

The long-deprecated `--storage.tsdb.retention` flag is gone in favour of `--storage.tsdb.retention.time`, so a container that still passes it will not start. Upgrade order that avoids pain: bump `promtool` and re-run rule tests in CI first, upgrade one non-critical replica of an HA pair and diff its graphs against the 2.x peer for a day, and snapshot the data directory before touching the rest, since a rollback to 2.x is not guaranteed to read blocks written by 3.x.

# 1) Range selectors are left-open in 3.x: the sample at exactly
#    t-5m is excluded, so rate() shifts and hardcoded test floats break.
rate(http_requests_total[5m])

# 2) UTF-8 names need the quoted selector syntax
{"http.server.request.duration_seconds_count", job="api"}

# ...or keep 2.x behaviour while you migrate
global:
  metric_name_validation_scheme: legacy

# 3) Remote Write 2.0 is opt-in per endpoint
remote_write:
  - url: http://mimir:8080/api/v1/push
    protobuf_message: io.prometheus.write.v2.Request

# 4) OTLP straight in, no exporter round-trip
#    prometheus --web.enable-otlp-receiver
#    POST /api/v1/otlp/v1/metrics

# 5) Renamed function, now behind a feature flag
#    old: holt_winters(node_load1[1h], 0.3, 0.3)
#    new: --enable-feature=promql-experimental-functions
double_exponential_smoothing(node_load1[1h], 0.3, 0.3)

# 6) Removed flag: --storage.tsdb.retention
#    use --storage.tsdb.retention.time=30d or the pod will not start

# 7) Normalised bucket labels: le="1" is now le="1.0"
http_request_duration_seconds_bucket{le="1.0"}

Answer

That is `--query.max-samples` (50,000,000 by default) refusing to materialise the result. It is a guardrail protecting the process from an OOM kill, not a bug to be flagged away. The engine counts every sample it must hold simultaneously across the whole evaluation, so a 30-day range query at a 15s step against twenty thousand series blows the ceiling long before it returns anything a human could read.

Diagnose before you touch the flag. The culprit is nearly always one of three things: a Grafana range query whose step collapsed because someone widened the time picker on a panel pinned to a fixed interval, a selector with no `job` or `namespace` matcher that therefore matches the entire TSDB, or a `rate()` over a long range against a high-cardinality classic histogram, where the `_bucket` series multiply the cost by the bucket count. Turn on `--query.log-file` to capture the exact query and parameters, watch `prometheus_engine_query_duration_seconds` broken down by its `slice` label to see whether time goes to `queue_time` or `inner_eval`, and pull `/api/v1/status/tsdb` for the largest series counts.

Fixes in priority order: bound the selector with real matchers, move the expression into a recording rule so dashboards read one pre-aggregated series, raise the step (Grafana's max data points and `$__rate_interval` control this), and only then raise the limit. Neighbouring knobs are `--query.timeout` (2m, surfaced as `query timed out in expression evaluation`), `--query.max-concurrency` (20, where excess queries queue rather than error, so the symptom is everything being slow at once), and `GOMEMLIMIT`, which is the correct way to bound Go heap growth in a container. If the log shows `found unfinished queries` after a restart, the active query tracker recorded what was in flight when the process died: those are your OOM suspects.

# The error, in full:
#   query processing would load too many samples into memory in
#   query execution (limit: 50000000)

# Capture the offenders with their exact parameters
#   prometheus --query.log-file=/prometheus/query.log
#   jq -r 'select(.stats.timings.evalTotalTime > 1) | .params.query' \
#     /prometheus/query.log | sort | uniq -c | sort -rn | head

# Built-in cardinality report (what the query is actually loading)
#   curl -s localhost:9090/api/v1/status/tsdb \
#     | jq '.data.seriesCountByMetricName[:10]'

# Where is engine time going, and are queries queueing?
sum by (slice) (rate(prometheus_engine_query_duration_seconds_sum[5m]))
prometheus_engine_queries_concurrent_max - prometheus_engine_queries

# BAD: recomputed from raw buckets on every dashboard refresh
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# GOOD: compute once in a rule, query one series
- record: job:http_request_duration_seconds:p99
  expr: histogram_quantile(0.99, sum by (job, le) (
          rate(http_request_duration_seconds_bucket[5m])))

# Runtime bounds that prevent the OOM kill in the first place
#   GOMEMLIMIT=6GiB              # container limit 8Gi, ~75%
#   --query.max-samples=50000000
#   --query.timeout=2m
#   --query.max-concurrency=20

Answer

Cost here is driven by active series first, samples per second second, and object storage a distant third, so any plan that opens with 'shrink retention' is starting in the wrong place. Model it with numbers you can defend: roughly 8KB of head memory per active series, so five million series is tens of gigabytes of RAM before a single query runs, and about 1-2 bytes per sample on disk after compression, which puts 10,000 series at a 15s interval over 30 days near 25GB. The levers, largest first.

Drop what nobody queries, at scrape time, with `metric_relabel_configs`: a default Go or JVM exporter ships hundreds of `go_*` and `jvm_buffer_pool_*` series per instance that no dashboard has ever opened, and dropping them saves memory, disk and query time together. Measure with `count by (__name__)` first instead of guessing. Widen `scrape_interval` from 15s to 60s on jobs that only feed capacity dashboards, cutting samples per second by four with no loss of fidelity for any alert carrying a 5m `for:`.

Move classic histograms to native histograms wherever the client supports it, since that collapses a dozen bucket series into one. On the long-term tier the money is in downsampling: the Thanos compactor's `--retention.resolution-raw`, `--retention.resolution-5m` and `--retention.resolution-1h` let you hold raw data for 30 days, 5m data for six months and 1h data for two years. Two costs people forget: S3 request charges, where a compactor churning small blocks can spend more on LIST and GET calls than on stored bytes, and cross-region egress on remote write, which a per-region receiver eliminates. On compute, `GOMEMLIMIT` set near 75% of the container limit lets you run a smaller instance without the OOM risk that makes teams over-provision by reflex.

# 1) Measure before you cut: which metric names dominate?
topk(20, count by (__name__) ({__name__=~".+"}))
count({__name__=~"go_.*"})          # runtime noise nobody queries

# 2) Drop at scrape time: saves memory, disk AND query cost
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_gc_duration_seconds.*|go_memstats_.*|jvm_buffer_pool_.*'
    action: drop

# 3) Cheaper interval for capacity-only jobs
- job_name: 'batch-workers'
  scrape_interval: 60s      # 4x fewer samples than 15s
  scrape_timeout: 30s

# 4) Downsampling decides the object-storage bill (Thanos compactor)
#   thanos compact \
#     --retention.resolution-raw=30d \
#     --retention.resolution-5m=180d \
#     --retention.resolution-1h=730d

# 5) Numbers worth quoting
#   ~8KB head memory per active series
#   ~1-2 bytes per sample on disk after compression
#   10k series @ 15s for 30d  ~=  25GB local disk
#   GOMEMLIMIT ~= 75% of the container memory limit

Answer

Agent mode, started with `prometheus --agent` and promoted out of experimental in 3.0, strips the server down to service discovery, scraping and remote write. There is no queryable local TSDB, no rule evaluation, no alerting and no graphing UI: samples land in a WAL that exists only to survive a receiver outage, and are truncated once shipped. The point is edge collection.

If a cluster's metrics all end up in Mimir, Thanos Receive or a vendor endpoint anyway, a full Prometheus there is paying for a query engine, a head block and 15 days of disk that nobody reads, and the agent runs in a fraction of the memory because it never holds a queryable head. Know when not to use it, because that is the real question. You lose local alerting, so a WAN partition takes out both your metrics pipeline and your ability to page about the cluster that is partitioned; the common compromise is agent mode for high-volume low-criticality telemetry and a full server for anything whose alerts must survive isolation.

You also lose the local UI, which is the single most useful thing you have during an incident, so make sure the agent's own `/metrics` is scraped from elsewhere and that `prometheus_remote_storage_samples_pending` is on a dashboard. Operationally it is the same binary and mostly the same config file minus `rule_files` and `alerting`, and the process refuses to start if you pass those or the normal TSDB flags alongside `--agent`; storage moves to `--storage.agent.path`. The alternative in 2026 is the OpenTelemetry collector with a `prometheus` receiver and `prometheusremotewrite` exporter, which covers traces and logs too. Agent mode wins when the fleet is already pure Prometheus and you want relabeling that behaves byte-for-byte like the server.

# Same binary, different mode
#   prometheus --agent \
#     --config.file=/etc/prometheus/agent.yml \
#     --storage.agent.path=/prometheus/wal \
#     --web.listen-address=:9090

# agent.yml: scrape + remote_write only.
# rule_files or alerting here make the process exit at startup.
global:
  scrape_interval: 30s
  external_labels:
    cluster: mumbai-prod
    __replica__: agent-0

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

remote_write:
  - url: https://mimir.internal/api/v1/push
    protobuf_message: io.prometheus.write.v2.Request
    queue_config:
      max_shards: 50
      max_samples_per_send: 2000
      capacity: 10000
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop

# The agent has no local data to fall back on, so watch:
#   prometheus_remote_storage_samples_pending
#   prometheus_remote_storage_samples_failed_total