Grafana Interview Questions and Answers

Answer

Grafana is an open-source observability and visualization platform, originally forked from Kibana in 2014 by Torkel Ödegaard. It solves the problem of a unified pane of glass over many different telemetry backends. Before Grafana, teams had one UI for Prometheus, another for ElasticSearch logs, a third for CloudWatch, switching between them on incident calls was brutal.

Grafana connects to 100+ data sources (Prometheus, Loki, Tempo, Mimir, MySQL, PostgreSQL, CloudWatch, Elasticsearch, Datadog, etc.) and lets you build dashboards, alerts, and exploration views over all of them in one place. In 2026, Grafana Labs has bundled their LGTM stack, Loki (logs), Grafana (dashboards), Tempo (traces), Mimir (metrics at scale), making it the dominant open-source observability story. Companies use it for application performance monitoring, infrastructure dashboards, business KPIs, and on-call alerting.

Answer

A dashboard is a single page that groups related visualizations together, for example, an 'API latency' dashboard for a service or a 'Kubernetes node health' dashboard for a cluster. Each visualization on the dashboard is a panel. A panel has its own query (against a data source), a visualization type (time series, gauge, stat, bar chart, table, heatmap, etc.), and display options.

Panels can be arranged in rows for grouping, resized via drag-handles, and given variables that change what they show. Dashboards are stored as JSON, can be version-controlled in git, and are imported/exported as JSON files (or via the new file-based provisioning in Grafana 11). Dashboards also support time-range selection at the top, the chosen range is passed to every panel's query automatically.

Answer

A data source is a connection to a backend that holds telemetry, Prometheus, Loki, Tempo, MySQL, CloudWatch, Elasticsearch, and so on. Grafana itself stores almost nothing about your metrics; it queries the data source at render time and visualizes the response. You configure data sources at the Grafana org/instance level (Connections → Data sources), and every panel picks one.

Each data source has its own query editor, PromQL for Prometheus, LogQL for Loki, SQL for relational DBs. In production, never click-configure data sources in the UI for a real environment, use provisioning files (YAML in `/etc/grafana/provisioning/datasources/`) so they're reproducible across environments and survive a reinstall.

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
    jsonData:
      timeInterval: 30s
      httpMethod: POST

Answer

PromQL (Prometheus Query Language) is the query language for Prometheus and Mimir. It operates on time-series data, every metric is a (name, labels, value, timestamp) tuple. The basic building blocks: an instant vector selector picks one value per series at the current time (`http_requests_total{status='200'}`), a range vector selector picks a window (`http_requests_total[5m]`), and functions like `rate()`, `sum()`, `avg()` aggregate across series and time.

The most common pattern in dashboards is `rate()` over a range vector wrapped in `sum() by (...)`. The trap most beginners hit: you can't graph a counter directly, counters always increase, so you must apply `rate()` or `increase()` to get something meaningful.

# Requests per second by HTTP status, last 5 min:
sum by (status) (rate(http_requests_total[5m]))

# 95th percentile latency:
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# Error ratio (5xx / total):
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Answer

LogQL is Loki's query language, designed to feel similar to PromQL but for logs. A LogQL query has two parts: a log stream selector with label matchers (just like PromQL), followed by an optional filter expression. Selectors pick streams by labels (`{app='api',env='prod'}`), filters narrow lines (`|= 'error'`, `!= 'health-check'`, `|~ 'regex'`).

LogQL also has metric queries that turn log lines into time series, `rate({app='api'} |= 'error' [5m])` gives errors per second. Key gotcha: Loki indexes only labels, not log content. Stuffing high-cardinality fields (request_id, user_id) into labels destroys Loki performance, keep those in the log line and filter with `|=` or extract on the fly with `| json`.

# Last 1000 error lines from the api in prod:
{app="api", env="prod"} |= "error"

# Error rate per second:
sum(rate({app="api"} |= "error" [5m])) by (env)

# Parse JSON logs and filter by extracted field:
{app="api"} | json | status >= 500

Answer

MetricsQL is the query language used by VictoriaMetrics, a popular Prometheus-compatible TSDB. It's a strict superset of PromQL, every PromQL query runs unchanged, but MetricsQL adds extra functions (`keep_last_value`, `running_sum`, `alias`, default values for missing series, friendlier `WITH` template expressions). It comes up when you're interviewing at a company that's swapped Prometheus for VictoriaMetrics to save infra cost (about 10× less RAM and disk for the same metrics in many benchmarks). In Grafana, you point a Prometheus-type data source at VictoriaMetrics's URL and queries 'just work', but enabling the VictoriaMetrics datasource plugin unlocks the MetricsQL-specific extensions.

Answer

Variables (also called template variables) are dropdowns at the top of a dashboard that let you parameterize panel queries, pick an environment, a service, a pod, a customer, and every panel updates. Without variables, you'd build one dashboard per service or per cluster, which doesn't scale. Common types: Query (populated from a data source, `label_values(up, job)` to get all job names from Prometheus), Custom (hardcoded values), Constant, Data source (switch the data source itself), Ad hoc filters (add arbitrary label filters that apply globally).

The variable is referenced in queries as `$varname` or `${varname}`. Variables support 'All' (`.*` or a regex), multi-select, and chained dependencies (one variable populates another).

# Variable definition (in the dashboard JSON or via UI):
Name: service
Type: Query
Query: label_values(up, job)
Multi-value: true
Include All: true

# Use in a panel:
sum by (status) (rate(http_requests_total{job=~"$service"}[5m]))

Answer

In Grafana 11, alerting is centralized, Alerting → Alert rules. You define a rule with: (1) a query against a data source, (2) a threshold condition (`is above 0.05`), (3) an evaluation interval (every 30s), and (4) a duration the condition must hold ('Pending → Firing' after 5m). Firing rules emit alerts that match label-based routes to contact points (Slack, PagerDuty, email, webhook).

The biggest shift from the legacy panel-based alerting (pre-v8) is that alerts now live independently of panels, you can have an alert without a dashboard, and one alert can use multiple queries from different data sources. Always set a 'For' duration; without it, a brief blip will page you. The standard SRE rule of thumb: For = at least 5 minutes for non-critical, 1-2 minutes for critical SLO burns.

Answer

Explore is the ad-hoc query view, no dashboard, no panel, just a query editor and a result. It's where on-call engineers live during incidents. Pick a data source, write a query, see the result with a time picker that's easy to slide.

Explore supports split view (two queries side by side, often Prometheus + Loki to correlate metrics with logs) and one-click jumps from logs to traces (via Tempo) using exemplars or trace IDs. The Logs panel in Explore has a histogram on top showing log volume over time, which is invaluable for spotting incident windows. Pro tip: when something looks weird in a dashboard, click the panel title → Explore, it opens the panel's query in Explore where you can mutate it freely without breaking the dashboard.

Answer

Three ways: (1) UI, Dashboards → New → Import, paste JSON or upload a file or paste a Grafana.com dashboard ID (e.g. 1860 for the famous Node Exporter Full dashboard). (2) HTTP API, `POST /api/dashboards/db` with the JSON. (3) File-based provisioning, drop a JSON file in `/etc/grafana/provisioning/dashboards/` with a YAML config pointing at the directory. The third option is the production answer, dashboards live in git, get reviewed in PRs, and Grafana auto-loads them on restart. Common gotcha: data source UIDs are baked into the dashboard JSON. When you import across environments (staging → prod), either replace the UID or use `__inputs` placeholders that the import dialog substitutes.

# /etc/grafana/provisioning/dashboards/main.yaml
apiVersion: 1
providers:
  - name: 'default'
    folder: 'Services'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Answer

A query is what you send to the data source, `rate(http_requests_total[5m])` against Prometheus. A transformation runs in Grafana, in the browser/server, AFTER the query returns. Transformations join multiple query results, rename columns, do math across series, filter rows, group by, organize fields, and so on.

The rule: do as much as you can in the query (it scales with your data source), use transformations only for what the data source can't do (joining results from two different data sources, for example) or for display formatting. Heavy transformations on large result sets kill panel performance because they run client-side. Common transformations: 'Organize fields' (rename/reorder/hide columns), 'Join by field' (outer-join two queries by a key), 'Add field from calculation' (e.g. ratio of two columns).

Answer

Grafana has three default roles: Viewer (read-only), Editor (can create/modify dashboards), Admin (full control of an org). On top of those, Grafana 9+ added fine-grained RBAC (Enterprise / Grafana Cloud), permissions like 'dashboards.read', 'alert.rules.write' that you assign via custom roles. Teams group users together for permission assignment.

Folders hold dashboards and inherit permissions to all dashboards inside them, the standard SRE pattern is one folder per team (Payments, Search, Onboarding), with Editor permission for the team and Viewer for everyone else. For production, integrate with SSO (SAML, OIDC, LDAP) and map external groups to Grafana teams automatically via `team_sync`. Never hand out admin role broadly, service-level dashboard access through teams is enough for almost everyone.

Answer

Run 2-3 stateless Grafana instances behind a load balancer (nginx, ALB, k8s Service). The trick is shared state, by default each instance uses a local SQLite database, which can't be shared. For HA, point all instances at a shared MySQL or PostgreSQL database (`[database]` section in `grafana.ini`) and use a shared object store (S3, GCS, or NFS) for image rendering and snapshots.

Sessions: use the database for session storage so a user staying signed in doesn't require sticky load balancing. For alerting in HA, Grafana 11 uses an internal Alertmanager cluster, set `[unified_alerting] ha_peers` to the addresses of the other instances so they gossip and dedupe firing alerts. The classic mistake: forgetting `ha_peers`, which causes 3 Grafana instances to send 3 copies of every Slack alert.

# grafana.ini for an HA node
[database]
type = postgres
host = grafana-db.internal:5432
name = grafana
user = grafana
password = $__file{/run/secrets/db_pass}

[unified_alerting]
enabled = true
ha_peers = grafana-0:9094,grafana-1:9094,grafana-2:9094
ha_listen_address = 0.0.0.0:9094

[session]
provider = postgres
provider_config = user=grafana password=... host=...

Answer

Unified alerting (introduced v8, refined through v11) consolidates Grafana-managed alerts and external Alertmanager-style routing into one model. The flow: (1) Alert rules query a data source on an interval and emit instances labeled with the query's group-by labels. (2) Each instance has a state: Normal, Pending (firing but not for long enough), Firing, NoData, Error. (3) Firing instances are sent to an internal Alertmanager. (4) The Alertmanager applies routes (tree of label matchers) to deliver each instance to one or more contact points (Slack, PagerDuty, email, OnCall, webhook). (5) Silences and mute timings suppress delivery without changing the firing state. Two killer features: notification policies are a tree (more specific routes match first), and you can group alerts by labels in the contact point so you get one Slack message for 'API down on 12 pods' instead of 12 messages.

Answer

Contact points are the destinations alerts go to, a Slack webhook, a PagerDuty integration key, an email address, a generic webhook, Grafana OnCall. Each contact point can have its own template for the message body. Notification policies route alerts to contact points based on label matchers.

Silences temporarily mute alerts matching a set of label matchers, used during planned maintenance ('silence all api-prod alerts from 2-4 AM tonight'). Mute timings are recurring versions of silences ('every Saturday and Sunday' or 'every day 0-8 IST'), they're attached to notification policies instead of being one-off. The combination is critical for sane on-call: mute timings handle predictable suppression, silences handle ad-hoc maintenance windows, and you never actually disable an alert (which would survive Grafana restarts as a permanent gap in monitoring).

Answer

A slow dashboard is almost always a slow query problem, not a Grafana problem. Steps: (1) Open Query Inspector (panel menu → Inspect → Query), it shows the actual query, the time taken, and the response size. (2) Look for high-cardinality aggregations in PromQL, `sum by (request_id, user_id)` will explode. Aggregate to coarse labels (`sum by (service, status)`) before any per-instance grouping. (3) Avoid `rate(metric[5m])` with very wide time ranges that fetch millions of samples, use a recording rule in Prometheus to pre-aggregate. (4) Reduce panels per dashboard, 30+ panels makes the browser re-render slow. Split into sub-dashboards or use the new Scenes API for virtualization. (5) Set Min step to the resolution your dashboard actually needs (30s, 1m) rather than letting Grafana auto-pick something tiny. (6) For Loki, narrow the stream selector first ('{app=x, env=prod}') before adding filters, selectors with too many series scan a lot of chunks.

Answer

Scenes is Grafana 11's new dashboard rendering framework, a React-based, declarative API for building dashboards and apps inside Grafana. Before Scenes, the dashboard renderer was AngularJS-derived and not pluggable. With Scenes, dashboards become a composition of `SceneObject`s (queries, layouts, variables, behaviors) that re-render reactively as state changes.

Practical impact for engineers: (1) Plugin authors can build rich, dashboard-like apps with the same primitives. (2) Built-in dashboards in Grafana 11 (and Cloud) render faster and use less memory. (3) The new alerting and metrics-drilldown views are Scenes-based. You don't need to learn Scenes to author standard JSON dashboards, but if you build a Grafana plugin or a custom app, Scenes is the API you reach for. Grafana 11 also introduced k8s monitoring improvements, OnCall integration with the core, and a vastly improved metric drilldown UX.

Answer

The LGTM stack is built for this. Three correlation patterns: (1) **Exemplars**, Prometheus histograms can include trace IDs as exemplars. A latency spike on a `http_request_duration_seconds_bucket` panel shows little diamond markers; clicking one opens the matching Tempo trace. (2) **Metric to logs via labels**, set up a 'Derived field' on the Loki data source that extracts the trace_id from log lines and links to Tempo.

Conversely, configure Tempo's 'Trace to logs' to query Loki with `{service='$service'}` and the trace's time range. (3) **Service Graph** (from Tempo), auto-generated topology graph showing which services call which, with latency/error rates. The end goal of the LGTM stack is one-click jumps: see latency spike → click exemplar → see trace → click span → see logs from that exact request, all without leaving Grafana.

Answer

Never click-configure production Grafana, use file-based provisioning so everything is in git, reviewed, and reproducible. The `/etc/grafana/provisioning/` tree has subdirectories for `datasources/`, `dashboards/`, `alerting/`, `notifiers/`, `plugins/`. Each subdir contains YAML config files; the actual dashboards are JSON files referenced by a YAML provider config.

On startup (and on `updateIntervalSeconds`), Grafana reads these files and reconciles state. For Terraform shops, the official `grafana/grafana` Terraform provider can manage data sources, dashboards (as JSON strings), folders, teams, and alert rules, useful if you already manage other infra as Terraform. For Kubernetes, the Grafana Operator (open source) lets you declare `GrafanaDashboard` and `GrafanaDataSource` CRDs. The pattern most production teams settle on in 2026: ConfigMaps + provisioning YAML + a CI pipeline that validates dashboard JSON before merging.

# /etc/grafana/provisioning/alerting/policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: oncall-default
    group_by: ['alertname', 'service']
    routes:
      - receiver: pagerduty-critical
        matchers:
          - severity = critical
        continue: false
      - receiver: slack-warnings
        matchers:
          - severity = warning
        mute_time_intervals: ['weekends']

Answer

Grafana OnCall (open source, integrated into Grafana 11) handles the bit Alertmanager doesn't, schedule rotations, escalation policies, acknowledgements, post-resolution actions. A typical setup: (1) Build a schedule with shifts (e.g. weekly rotation across 4 engineers, with daytime/nighttime layers). (2) Define an escalation chain, page primary on Slack/SMS, after 5 min unacked also page secondary, after 15 min unacked page the manager. (3) Connect Grafana Alerting as the source so firing alerts hit OnCall instead of PagerDuty. (4) Integrate with Slack for ack/resolve from a button in the channel. India twist: most teams here run follow-the-sun rotations across IST and one Western timezone (US/EU), and OnCall's timezone-aware schedules make this less painful than building it in Google Sheets. For postmortems, OnCall's incident timeline export feeds straight into a blameless postmortem template.

Answer

A recording rule is a Prometheus-side feature where the result of a complex query is pre-computed on a schedule and stored as a new time series. Defined in Prometheus's rules file, evaluated every `interval` (typically 30s-1m), the result lives in TSDB with a name you give it (convention: `level:metric:operation`). Grafana then graphs the pre-computed series instead of recomputing on every render.

When to use them: (1) Dashboards that take >2 seconds to load due to expensive aggregations. (2) Alert rules that re-evaluate the same heavy query every interval. (3) Metrics aggregated to a coarser dimension you query often (e.g. `service:requests:rate5m` instead of `sum by (service) (rate(http_requests_total[5m]))` in 30 places). Trade-off: more storage and more Prometheus CPU at evaluation time. In mature SRE practice at scale, recording rules are the default for any panel with a sum-by aggregation across more than a handful of pods, and interviewers at payments and consumer-scale companies will usually ask you where you'd draw that line.

# In Prometheus rules.yaml
groups:
  - name: api-aggregations
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service, status) (rate(http_requests_total[5m]))
      - record: service:http_request_duration_seconds:p95_5m
        expr: histogram_quantile(0.95, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

Answer

Three plugin types: panel (custom visualization), data source (connect to a new backend), app (a bundle of panels, data sources, and pages, what most enterprise integrations look like). Grafana provides `create-plugin` (npx) to scaffold a project, TypeScript + React for the frontend, optional Go backend for data source plugins that need server-side logic (e.g. credentials, network access from inside Grafana's process). The lifecycle: write your TS/React code, run `npm run build`, drop the built bundle into `/var/lib/grafana/plugins/`, restart Grafana.

For local dev, `npm run dev` watches and rebuilds. Signing: in production Grafana refuses to load unsigned plugins by default, you either submit to the Grafana plugin marketplace for signing, or set `allow_loading_unsigned_plugins` in `grafana.ini` (for internal plugins only). In 2026, most custom panels are written using the Scenes API for state management and `@grafana/ui` for visual consistency.

Answer

Every dashboard save creates a new version in Grafana's database. UI: Dashboard settings → Versions shows the diff and lets you restore. The version history persists indefinitely unless you trim it manually (large orgs sometimes have thousands of versions slowing the DB).

For export: 'Share → Export → Save to file' produces a portable JSON. The 'Export for sharing externally' option replaces data source-specific UIDs with `${DS_PROMETHEUS}` placeholders so the JSON can be imported into a different Grafana with different data source UIDs. The real production answer is to skip this UI dance entirely: keep your dashboard JSON in git, do code review on PRs, and use provisioning to load them. Tools like `grafonnet` (Jsonnet library) or `grafana-dashboard-builder` (Python) let you author dashboards as code, generate JSON, and avoid the merge nightmares that come from editing JSON by hand.

Answer

An exemplar is a single concrete observation attached to a Prometheus histogram bucket, typically a trace ID. When your app records a request that took 4.2s, the histogram bucket increments AND an exemplar with the trace ID is recorded. In Grafana, exemplars render as little diamonds on time-series plots.

Click one and Grafana jumps to the Tempo trace for that exact request. This is the single most useful feature for debugging latency tail issues, instead of asking 'why was p99 latency 4s at 14:23' and grepping logs blindly, you click the diamond and see the specific slow trace. Requirements: Prometheus must be running with `--enable-feature=exemplar-storage`, your app must emit exemplars (most OpenTelemetry SDKs do automatically), and Grafana's Prometheus data source must have 'Exemplars' enabled with a Tempo-typed link configured.

Answer

Three layers of multi-tenancy in Grafana: (1) **Orgs**, Grafana's built-in concept, each org is a fully isolated tenant with its own users, data sources, dashboards. Works for low tenant counts but switching orgs is clunky for end users. (2) **Folders + Teams**, single org, one folder per tenant, team-scoped permissions. Works well for SaaS where tenants are internal teams. (3) **Per-tenant Grafana instances**, best isolation, used by Grafana Cloud itself.

Heavy at scale but each tenant gets full freedom. For the data side, Mimir, Loki, and Tempo all support multi-tenancy via the `X-Scope-OrgID` header, Grafana sends it automatically based on the org/team, and the backend filters data accordingly. Critical: enforce the tenant header at the data source level, not just the UI; otherwise a clever PromQL injection could leak cross-tenant data.

Answer

At this scale, you stop running Prometheus and start running Mimir (or Cortex/VictoriaMetrics), Prometheus's single-binary architecture caps out around tens of millions of active series. Reference architecture: (1) **Metrics**: each service ships metrics via OpenTelemetry to a Mimir cluster (distributors → ingesters → store gateway → S3 long-term). 30-day hot retention, 13-month cold via S3. Recording rules aggressively pre-compute service-level dashboards. (2) **Logs**: Loki, sharded by tenant (one tenant per team), object storage backend.

Strict label hygiene, service, env, level only; everything else goes in the line and is parsed at query time. (3) **Traces**: Tempo with tail sampling (sample 100% of error/slow traces, 1% of healthy). (4) **Frontend**: HA Grafana behind a load balancer with SSO. Teams get their own folders, on-call rotations live in Grafana OnCall. (5) **Standards**: every service emits the same four 'golden signals' (RED, Rate, Errors, Duration; plus Saturation), so platform-wide dashboards work without per-service customization. (6) **Cost control**: drop rules at the ingest layer to strip high-cardinality labels before they hit storage; this is where 80% of cost overruns live. In India, this is broadly the shape that payments and consumer-scale platform teams converge on, and it is the answer interviewers at companies like Razorpay, Swiggy, and Postman are listening for when they hand you this design question.

Answer

SLO (Service Level Objective) alerting is the modern replacement for static threshold alerts ('CPU > 80%'). The Google SRE book approach: define an SLI (e.g. 'fraction of HTTP requests answered in < 500ms with a 2xx status'), an SLO target (99.9% over 30 days), then alert on **error budget burn rate**, not on a fixed threshold. Two-window burn rate alerts are the recommended pattern: page if you're burning your 30-day budget at >14× the sustainable rate over both the last 5 min AND the last 1 hour (fast burn, major outage); and a slower 'budget exhausted in 1 day' check over both the last 1 hour and the last 6 hours (slow burn).

Grafana's unified alerting handles this with multi-condition rules. The math: `error_budget_burn = (1 - success_rate) / (1 - SLO)`. The big win: zero alerts during minor blips, but reliable pages during real customer-affecting outages. Sloth (Spotify open source) and Pyrra generate the boilerplate Prometheus rules and Grafana alerts from a simple SLO YAML file.

# Multi-window burn-rate alert (PromQL):
# Fires when 5m and 1h burn rates both exceed 14.4 (fast burn).
(
  (1 - sum(rate(http_requests_total{status=~"2.."}[5m])) / sum(rate(http_requests_total[5m])))
  / (1 - 0.999)
) > 14.4
AND
(
  (1 - sum(rate(http_requests_total{status=~"2.."}[1h])) / sum(rate(http_requests_total[1h])))
  / (1 - 0.999)
) > 14.4

Answer

Tools support the culture; they don't create it. The Grafana side of blameless postmortems: (1) Every incident has a dashboard auto-pinned to the incident's time range, Grafana's 'Share' link with absolute timestamps embeds the exact view in the postmortem doc. (2) OnCall's timeline export gives you 'who got paged, who acked, what action they took' as a CSV, paste into the timeline section of the postmortem. (3) Annotations on dashboards mark deploys, config changes, feature flag flips. When the postmortem asks 'what happened at 14:23?', the annotation says 'deploy of api v1.42.0', no detective work. (4) Compare time-shifted graphs ('this week vs same time last week') to highlight what changed. (5) After the postmortem, action items become alert rules ('if metric X exceeds Y again, page us before it hits prod'). The cultural bit: the postmortem template asks 'how did our monitoring fail to catch this earlier?' rather than 'who pushed the bad deploy', every postmortem ends with at least one new alert rule, recording rule, or dashboard, so monitoring compounds over time instead of stagnating.

Answer

Grafana itself has metrics, point a Prometheus at `/metrics` on the Grafana port (it's exposed by default). Key signals: `grafana_api_response_status_total` (request volume by status), `grafana_api_dataproxy_request_all_milliseconds` (proxy time to data sources), `grafana_alerting_rule_evaluation_duration_seconds` (alert rule evaluation time), `process_resident_memory_bytes`, `go_goroutines`. Common root causes: (1) **Heavy alert rules**, too many rules evaluating too often.

Each rule = one query per interval. 1000 rules at 30s = 33 queries/sec to your data source. Fix by raising intervals on non-critical rules. (2) **Browser load via the API**, a poorly-designed dashboard refreshing every 5s with 30 panels = 6 queries/sec from each open tab. Set sensible refresh intervals (≥30s for most). (3) **PNG rendering**, Grafana's image renderer launches headless Chrome and is heavy.

If you're sending many alert previews with image attachments, run the image renderer as a separate service. (4) **DB latency**, Grafana's metadata DB (Postgres) being slow causes UI lag. Check `grafana_database_conn_inuse`. Profile with Go's pprof endpoint (`/debug/pprof`) for the deep dive.

Answer

Run parallel for at least one full on-call rotation cycle (usually 4-6 weeks). Step by step: (1) **Inventory**: list every alert, every dashboard, every contact point in the old system. Categorize by criticality. (2) **Stand up LGTM**: Mimir (or Prometheus + Thanos), Loki, Tempo, HA Grafana.

Make sure storage, retention, and access controls are sized. (3) **Wire up the data**: instrument apps with OpenTelemetry (single library replaces both Prometheus client and Datadog APM agent, lets you switch backends later without code changes). For infra metrics, deploy node_exporter + cadvisor + kube-state-metrics. (4) **Rebuild dashboards as code**: use grafonnet or a Python builder to define dashboards in version control. Don't try to migrate JSON 1:1 from the old tool, take the chance to standardize on RED/USE metrics. (5) **Rebuild alerts last**, with two-window burn rates instead of static thresholds.

Page yourself in parallel for the cycle to confirm nothing's missed. (6) **Cutover**: announce a date, switch the on-call routing, leave the old system read-only for 30 days. (7) **Decommission**: tear down old infra, document the migration. Critical: don't try to migrate dashboards and alerts in one shot, dashboards are visual and easy to redo, alerts are mission-critical and need extra validation. Quote a realistic timeline in the interview: a migration at this size typically runs 2-3 months end-to-end, and Indian consumer-scale employers hiring for this work (CRED and Cure.fit among them) expect a number in that range rather than a few weeks.