Elasticsearch Interview Questions and Answers

Answer

Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene, first released by Shay Banon in 2010. It solves three problems that traditional databases handle poorly: (1) full-text search across millions or billions of documents with relevance scoring (BM25), (2) horizontally-scalable indexing and querying by sharding data across a cluster, and (3) near-real-time analytics on semi-structured data via aggregations. You write and read JSON documents over a REST API, and the engine handles inverted-index construction, tokenization, scoring, and replication for you. In 2026, the three dominant use cases in India are: e-commerce product search (Flipkart, Myntra, Swiggy), log/observability analytics (the ELK stack at PhonePe, Razorpay), and as a vector store for RAG (since dense_vector + kNN search matured in 8.x).

Answer

An inverted index is a map from every distinct term in your corpus to the list of documents (and positions) that contain it. Compare with a database's normal (forward) index: 'doc → fields'. The inverted index flips this: 'term → docs'.

So for the document set {1: 'the cat sat', 2: 'the dog sat'}, the inverted index is: 'the' → [1, 2], 'cat' → [1], 'sat' → [1, 2], 'dog' → [2]. Searching for 'cat sat' becomes a set-intersection over the postings lists, O(small) instead of scanning every document. Lucene stores this on disk in immutable segments and merges them in the background. This structure is why Elasticsearch can search billions of documents in milliseconds, but also why updates are expensive, you cannot edit a term in place, you have to mark the doc deleted and reindex.

// Conceptual: how 'the cat sat on the mat' is indexed
// Tokens after the standard analyzer:
//   ["the", "cat", "sat", "on", "the", "mat"]
//
// Inverted index (per term → postings list):
//   the → [{doc: 1, freq: 2, pos: [0, 4]}]
//   cat → [{doc: 1, freq: 1, pos: [1]}]
//   sat → [{doc: 1, freq: 1, pos: [2]}]
//   on  → [{doc: 1, freq: 1, pos: [3]}]
//   mat → [{doc: 1, freq: 1, pos: [5]}]

Answer

Every text field is processed by an analyzer at index time AND at query time. The pipeline has three stages, applied in order: (1) Character filters, operate on the raw string before tokenization. Examples: html_strip removes HTML tags, mapping replaces characters, pattern_replace runs regex. (2) Tokenizer, splits the string into tokens.

Exactly one per analyzer. Common choices: standard (Unicode word boundaries), whitespace, keyword (no split), ngram, edge_ngram (for autocomplete). (3) Token filters, transform the stream of tokens. Run in order, and you can chain many.

Common: lowercase, stop (remove stop words), stemmer/porter_stem, synonym, asciifolding (é → e). The same pipeline runs on the query string when you use a match query, that's how 'Running' in the doc matches 'run' in the query: both get stemmed to 'run'. Picking the right analyzer is the single biggest determinant of search quality.

PUT /products
{
  "settings": {
    "analysis": {
      "analyzer": {
        "product_name": {
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "english_stop", "porter_stem"]
        }
      },
      "filter": {
        "english_stop": { "type": "stop", "stopwords": "_english_" }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "product_name" }
    }
  }
}

Answer

These are the two string types in Elasticsearch and confusing them is the #1 beginner mistake. `text` runs through an analyzer, the string is tokenized, lowercased, possibly stemmed, and stored as multiple terms in the inverted index. You use it for full-text search via the match query. You CANNOT sort, aggregate, or do exact filtering on `text` fields without fielddata (which is memory-expensive). `keyword` is stored verbatim as a single term, no analysis.

You use it for filtering, sorting, aggregations, and term-level queries (term, terms, prefix). The standard pattern is to map a string as both, via a multi-field: indexed as `text` for search and as `name.keyword` for aggregations. This is what Elasticsearch does automatically when you let dynamic mapping create string fields.

PUT /products
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 }
        }
      }
    }
  }
}

// Search: GET /products/_search { "query": { "match": { "name": "running shoe" } } }
// Aggregate: GET /products/_search { "aggs": { "brands": { "terms": { "field": "name.keyword" } } } }

Answer

Elasticsearch exposes a REST API for document CRUD. PUT /index/_doc/{id} or POST /index/_doc (auto-id) to index. GET /index/_doc/{id} to fetch.

POST /index/_update/{id} to do a partial update (you supply only the fields that change, Elasticsearch fetches the doc, merges, and reindexes). DELETE /index/_doc/{id} to remove. Important: there is no true in-place update.

Under the hood, an update is a delete + insert because Lucene segments are immutable. This is why high-update workloads are expensive in Elasticsearch, each update creates a new version of the doc and marks the old one for deletion (collected during segment merges).

// Index a doc
POST /products/_doc/1
{ "name": "Running Shoe", "price": 4999, "in_stock": true }

// Get it back
GET /products/_doc/1

// Partial update, only changes the price field
POST /products/_update/1
{ "doc": { "price": 3999 } }

// Delete
DELETE /products/_doc/1

Answer

The _bulk API lets you send many index/update/delete operations in one HTTP request. Each operation is two NDJSON lines: an action header and (for index/update) the document body. Bulk indexing is 10-100× faster than one-by-one indexing because you eliminate HTTP overhead and let Elasticsearch batch refresh and merge work.

The sweet spot is 5-15 MB or 1000-5000 docs per batch, too small and overhead dominates, too large and you get timeouts and circuit breaker trips. For initial loads of large datasets at Flipkart-scale, also: temporarily set refresh_interval to -1, drop replicas to 0, then restore both after ingestion completes. This can cut import time by 5×.

POST /products/_bulk
{ "index": { "_id": "1" } }
{ "name": "Running Shoe", "price": 4999 }
{ "index": { "_id": "2" } }
{ "name": "Yoga Mat", "price": 1299 }
{ "update": { "_id": "3" } }
{ "doc": { "price": 2999 } }
{ "delete": { "_id": "4" } }

Answer

match runs the query string through the same analyzer as the field, so 'Running Shoes' becomes ['run', 'shoe'] and matches docs that contain those terms in any order. Use match for full-text search on `text` fields. term does NOT analyze the input, it looks up the literal value in the inverted index. Use term for exact matches on `keyword` fields, numbers, booleans, dates, IPs.

The classic bug: running term on a text field with a multi-word string ('term: "Running Shoes"') always returns 0 hits, because the indexed terms are 'run' and 'shoe' separately. Rule of thumb: text fields → match family (match, multi_match, match_phrase). keyword/numeric/date/boolean → term family (term, terms, range, prefix, wildcard).

// Full-text search on a text field, analyzed, fuzzy, ordered by relevance
GET /products/_search
{ "query": { "match": { "name": "running shoe" } } }

// Exact match on a keyword field, no analysis, no scoring
GET /products/_search
{ "query": { "term": { "brand.keyword": "Nike" } } }

// Anti-pattern: term on a text field, usually returns nothing
GET /products/_search
{ "query": { "term": { "name": "Running Shoe" } } } // 0 hits!

Answer

bool is the workhorse compound query in Elasticsearch, almost every non-trivial search is wrapped in one. It has four clauses: (1) must, the clause must match, AND it contributes to the score. (2) should, at least one (by default) must match; contributes to score. Acts like an OR with relevance boosting. (3) filter, the clause must match, but is run in 'filter context' so it does NOT contribute to score and is cached.

Filters are dramatically faster on repeated queries, always prefer them when you don't need scoring. (4) must_not, the clause must not match; runs in filter context. The typical pattern: text relevance in must, exact category/price/availability filters in filter, optional boosts in should.

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "running shoe" } }
      ],
      "filter": [
        { "term": { "brand.keyword": "Nike" } },
        { "range": { "price": { "gte": 1000, "lte": 5000 } } },
        { "term": { "in_stock": true } }
      ],
      "should": [
        { "term": { "is_featured": true } } // boosts featured items higher
      ],
      "must_not": [
        { "term": { "discontinued": true } }
      ]
    }
  }
}

Answer

A shard is a single Lucene index, it holds a slice of an Elasticsearch index. When you create an Elasticsearch index with number_of_shards=3, your data is partitioned across 3 primary shards. Each shard is also replicated number_of_replicas times for redundancy (typical: 1 replica per primary).

So an index with 3 primaries and 1 replica = 6 total shards spread across the cluster. Sharding gives you horizontal scalability, searches fan out across shards and results are merged. Replication gives you fault tolerance and read throughput (replicas can serve searches).

Big gotcha: you cannot change number_of_shards on an existing index, you must reindex into a new one. So pick shard count carefully at index-creation time: aim for shards in the 10-50 GB range, with a total shard count under 600 per data node.

Answer

Kibana is the official Elastic UI, it sits on top of Elasticsearch and provides four big things: (1) Discover, an interactive log/document explorer with KQL filtering, the bread and butter for observability teams. (2) Visualize / Lens, drag-and-drop chart builder backed by Elasticsearch aggregations. (3) Dashboards, pin charts together for SREs / business teams. (4) Dev Tools, a console for running raw _search and _cluster API calls. In 2026, most India teams use Kibana primarily for log analytics (ELK stack with Filebeat shipping app logs) and for APM dashboards. It is also the home of the new ES|QL editor introduced in 8.11. Kibana itself stores its config in a `.kibana` system index inside the same Elasticsearch cluster.

Answer

Logstash is the original data-ingestion pipeline tool from Elastic. It pulls data from inputs (file, kafka, http, jdbc, beats), runs it through filters (grok, mutate, date, geoip, json), and pushes to outputs (Elasticsearch, S3, Kafka). In 2026, Logstash is mostly used when you need heavy transformation logic before indexing, e.g., parsing nginx access logs with grok and enriching with GeoIP.

For simpler shipping (logs, metrics from a host), the lighter Beats family (Filebeat, Metricbeat) is preferred, and increasingly Elastic Agent (single-binary replacement). The typical ELK pattern is: app servers run Filebeat → Logstash for parsing → Elasticsearch → Kibana for visualization. For lower-overhead deployments, you skip Logstash and use Filebeat's built-in modules or Ingest Pipelines inside Elasticsearch itself.

Answer

Since Elasticsearch 5.x (2017), the default scoring algorithm is BM25 (Best Match 25), an improvement over TF-IDF that penalizes very long documents less harshly. Three factors determine a doc's score for a query term: (1) Term frequency (TF), how often the term appears in this doc. More is better, with diminishing returns. (2) Inverse document frequency (IDF), how rare the term is across all docs.

Rare terms (like 'kubernetes') boost scores more than common ones (like 'the'). (3) Field length normalization, shorter fields score higher. A term in a title is more meaningful than the same term buried in a 10-page body. BM25 has two tunable knobs: k1 (term frequency saturation, default 1.2) and b (length normalization, default 0.75).

You rarely tune these, instead, use field boosts in multi_match or function_score to bias results. Use GET /index/_search { "explain": true } to see exactly how a score was computed.

Answer

multi_match runs a match query across multiple fields and combines the scores. The `type` parameter changes how scores are combined: (1) best_fields (default), uses the single highest field score (the 'most relevant' field wins). Best when one match in one field is what you care about. (2) most_fields, sums scores across all fields.

Use when matches in many fields are stronger signals (e.g., the same word in title AND description). (3) cross_fields, treats fields as one big field for purposes of term matching. Great for entity searches like a person's name where 'first_name' and 'last_name' are stored separately and you want 'Sundar Pichai' to match across both. (4) phrase, runs match_phrase on each field. (5) phrase_prefix, phrase + the last word treated as a prefix (autocomplete). Field boosts use ^N syntax: `"fields": ["title^3", "description"]` makes title 3× more impactful.

GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "adidas running shoe",
      "type": "best_fields",
      "fields": ["name^3", "brand^2", "description"],
      "tie_breaker": 0.3,
      "fuzziness": "AUTO"
    }
  }
}

Answer

function_score lets you modify the relevance score after the base query runs, useful for blending business signals (popularity, freshness, geo proximity, conversion rate) with text relevance. You wrap a base query and attach one or more functions: (1) weight, flat multiplier. (2) field_value_factor, multiply by a numeric field (popularity score, rating). (3) gauss / linear / exp decay functions, score decays as a field (date, geo_point, number) moves away from an origin. Perfect for 'recent posts score higher' or 'closer restaurants score higher'. (4) script_score, full Painless script for arbitrary math. `score_mode` controls how function scores combine (multiply, sum, avg). `boost_mode` controls how the function score combines with the base query (multiply by default, but `sum` keeps text relevance dominant). Swiggy / Zomato-style search ranking is almost always a function_score with a decay function for distance and a field_value_factor for restaurant rating.

GET /restaurants/_search
{
  "query": {
    "function_score": {
      "query": { "match": { "name": "pizza" } },
      "functions": [
        {
          "gauss": {
            "location": { "origin": "12.97,77.59", "scale": "2km", "decay": 0.5 }
          }
        },
        {
          "field_value_factor": { "field": "rating", "factor": 1.2, "modifier": "sqrt", "missing": 1 }
        }
      ],
      "score_mode": "multiply",
      "boost_mode": "multiply"
    }
  }
}

Answer

By default, Elasticsearch flattens arrays of objects when indexing. An array `[{"name":"alice","age":25}, {"name":"bob","age":30}]` gets stored as `{"name":["alice","bob"], "age":[25,30]}`. The cross-object relationship is lost, a search for `name=alice AND age=30` will match this doc, even though no single sub-object has both.

To preserve the relationship, use `nested` mapping. Each nested sub-object becomes a hidden child document indexed alongside the parent, and you query it with a `nested` query. Trade-offs: (1) Storage and indexing cost, each nested object is a separate Lucene doc. (2) Aggregations need the `nested` aggregation wrapper. (3) Default limit is 50 nested docs per parent (index.mapping.nested_objects.limit) and 50 nested fields per index.

Use nested ONLY when you need cross-field-within-object matching. Otherwise, denormalize or use object, it's much faster.

PUT /reviews
{
  "mappings": {
    "properties": {
      "comments": {
        "type": "nested",
        "properties": {
          "author": { "type": "keyword" },
          "rating": { "type": "integer" }
        }
      }
    }
  }
}

// Now: find reviews where author=alice AND rating=5 in the SAME comment
GET /reviews/_search
{
  "query": {
    "nested": {
      "path": "comments",
      "query": {
        "bool": {
          "must": [
            { "term": { "comments.author": "alice" } },
            { "term": { "comments.rating": 5 } }
          ]
        }
      }
    }
  }
}

Answer

By default, when Elasticsearch sees a field it has never seen before, it auto-creates a mapping for it (text+keyword for strings, long for integers, float for decimals, etc.). This is convenient in development but dangerous in production. Two failure modes: (1) Field type lock-in, once a field is mapped, you can't change its type without reindexing the whole index.

If your first document indexes a numeric-looking ID as long, but later a doc has a string ID, you get a mapping conflict. (2) Mapping explosion, if you index JSON with user-controlled keys (e.g., a `metadata: {}` map where keys come from end users), the number of fields can grow without bound. Each field consumes cluster state memory; clusters with 100k+ fields become unstable. Protections: set `index.mapping.total_fields.limit` (default 1000), set `"dynamic": "strict"` to reject unknown fields, or use `"dynamic": false` to ignore them. For free-form maps, use the `flattened` field type, it stores the whole object as a single field.

PUT /events
{
  "settings": { "index.mapping.total_fields.limit": 2000 },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "event_type": { "type": "keyword" },
      "timestamp": { "type": "date" },
      "custom_attrs": { "type": "flattened" }
    }
  }
}

Answer

When you index a doc, it lands in an in-memory buffer. The doc is NOT searchable yet. Every refresh_interval (default 1 second), Elasticsearch creates a new on-disk Lucene segment from the buffer and opens it for search, this is the 'near real-time' guarantee.

Refresh is cheap-ish but not free; segments multiply quickly. Periodically, segments are merged in the background. flush happens less often: it fsyncs the translog and clears it, ensuring durability. Two big tuning levers: (1) For heavy-ingest, infrequent-search workloads (logs), increase refresh_interval to 30s or 60s, fewer segments, lower CPU, much higher indexing throughput. (2) For bulk loads, set refresh_interval to -1 entirely during the load, then restore. The translog is the WAL, it guarantees indexed docs survive a crash even before flush, but you can sacrifice this for speed with index.translog.durability=async.

Answer

Aggregations are Elasticsearch's analytics engine, comparable to GROUP BY in SQL but much richer. Three families: (1) Metric aggs compute a number over a set of docs: avg, sum, min, max, stats, percentiles, cardinality (HyperLogLog count-distinct). (2) Bucket aggs group docs into buckets: terms (group by field value, like GROUP BY), date_histogram (time buckets), range, histogram, geohash_grid. Each bucket can contain sub-aggregations, that's how you compute 'average price per brand per month'. (3) Pipeline aggs operate on the output of other aggs: cumulative_sum, derivative, moving_avg, bucket_script.

The classic gotcha: terms aggs default to size=10 with approximate counts. For exact top-N, use a composite agg or set shard_size higher. For high-cardinality fields like user_id, terms is expensive, consider cardinality (approximate distinct count) or sampler agg.

GET /orders/_search
{
  "size": 0,
  "aggs": {
    "by_month": {
      "date_histogram": { "field": "created_at", "calendar_interval": "month" },
      "aggs": {
        "by_brand": {
          "terms": { "field": "brand.keyword", "size": 5 },
          "aggs": {
            "revenue": { "sum": { "field": "amount" } }
          }
        }
      }
    }
  }
}

Answer

Elasticsearch nodes have specialized roles you can mix or split: (1) master-eligible nodes, elect a single active master that owns cluster state (mapping changes, index creation, shard allocation). Run 3 master-eligible nodes for quorum-based fault tolerance. In small clusters they double as data nodes; in production clusters above ~5 data nodes, use dedicated masters (small machines, no data). (2) data nodes, store shards, serve searches and indexing.

The CPU/disk/memory of these is what your cluster is sized on. Further split into hot/warm/cold/frozen tiers for tiered storage in observability. (3) ingest nodes, run ingest pipelines (the lighter alternative to Logstash, embedded in Elasticsearch). Any node can be an ingest node; isolating helps when pipelines are heavy. (4) coordinating-only nodes, no data, no master role, just receive client requests and fan them out.

Useful as a load-balancing layer. (5) ML nodes (commercial license), run anomaly detection, NLP inference. Mis-sizing these tiers is the #1 cause of cluster instability in India log-analytics setups.

Answer

A search hits one coordinating node which fans out the query to every shard (primary or replica, whichever is less loaded). Two phases: (1) Query phase, each shard executes the query locally, computes scores, returns just the top N doc IDs and scores back to the coordinator. The coordinator merges and re-sorts these to determine the global top N. (2) Fetch phase, the coordinator goes back to the shards holding those final top N IDs and fetches the full _source.

This is why deep pagination (`from=10000, size=10`) is catastrophic: every shard has to score and return 10010 hits even though only 10 are needed. Use search_after (cursor-based pagination using sort tokens) for deep pagination, or scroll/PIT (point-in-time) for snapshot scans. The other implication: scoring is per-shard.

If your shards have very different doc populations, IDF can vary across shards. For perfectly uniform scoring, use search_type=dfs_query_then_fetch (slower, computes global IDF first).

Answer

Every clause inside a query runs in one of two contexts. Query context, answers 'how well does this doc match?' Computes a _score for each doc and influences ranking.

Used inside bool.must / bool.should and at the top level of `query`. Filter context, answers 'does this doc match yes/no?' No scoring.

Results can be cached in the query cache, dramatically speeding up repeat filters. Used inside bool.filter, bool.must_not, the `constant_score` query, and aggregation filters. Performance implication: for any predicate where you don't care about relevance (status, category, date range, geo box, in-stock flag, tenant_id), use filter context.

The query cache is per-segment and keyed on the filter; popular filters become essentially free on hot indexes. Putting filterable predicates in must is one of the most common performance bugs in production Elasticsearch deployments.

Answer

Ingest pipelines run inside Elasticsearch (on ingest nodes) and transform documents at index time. You define a pipeline as a chain of processors, grok, dissect, date, geoip, script (Painless), enrich, set, remove, rename, foreach, etc., and you target it by name on the index request (`?pipeline=my_pipeline`) or by default in the index settings. Compared to Logstash: ingest pipelines are lower latency (in-process), simpler to operate (no separate cluster), and well-suited for light-to-medium transformation.

Logstash wins for very heavy transformations, complex stateful aggregations, persistent queues, and fan-in from many disparate sources. In 2026, the trend in India is to push as much as possible into Filebeat/Elastic Agent + ingest pipelines, and only run Logstash where you need its expressive grok/ruby filter combinations or its persistent queue.

PUT /_ingest/pipeline/access_logs
{
  "description": "Parse nginx access logs",
  "processors": [
    { "grok": { "field": "message", "patterns": ["%{IPORHOST:client_ip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:ts}\\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:status} %{NUMBER:bytes}"] } },
    { "date": { "field": "ts", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"] } },
    { "geoip": { "field": "client_ip", "target_field": "geo" } },
    { "remove": { "field": "message" } }
  ]
}

Answer

Three common approaches, in increasing sophistication: (1) edge_ngram analyzer, tokenize each word into prefixes ('shoe' → 'sh', 'sho', 'shoe'). Index-time work, very fast at query time. Best for short fields like product names.

Set min_gram=1, max_gram=15 in token filter. (2) search_as_you_type field type (since 7.2), auto-generates a `field._2gram` and `_3gram` subfield using shingles. Use with a multi_match across `field`, `field._2gram`, `field._3gram`, type=bool_prefix. Simpler than configuring edge_ngram yourself. (3) completion suggester, a separate FST-based data structure optimized for prefix lookups.

Lightning-fast, but limited query types, you can't combine with full bool filters easily. Best for autocomplete dropdowns with no further filtering. Real-world e-commerce search at Flipkart-scale typically uses search_as_you_type for the input box + a heavier multi_match for the result page after submit.

PUT /products
{
  "mappings": {
    "properties": {
      "name": { "type": "search_as_you_type" }
    }
  }
}

GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "runn",
      "type": "bool_prefix",
      "fields": ["name", "name._2gram", "name._3gram"]
    }
  }
}

Answer

ILM automates the lifecycle of indices across hot, warm, cold, and frozen tiers, the standard pattern for log/observability workloads. You define a policy: 'Roll over the index every 50GB or every 7 days. Move to warm after 30 days.

Move to cold after 90 days. Delete after 365 days.' ILM applies this policy to data streams (or rollover-aliased indices).

The tiers map to node types with different hardware: hot = SSDs, fewer docs per node, high CPU. warm = HDDs or cheaper SSDs, more docs per node. cold = searchable snapshots (data lives in S3/blob storage, locally cached). frozen = nearly-archive, very rarely read. The 2026 pattern at PhonePe / Razorpay: 7 days hot, 21 days warm, 60 days cold (searchable snapshot), then deleted. This reduces storage cost 80-90% versus keeping everything on hot tier. ILM also handles the rollover writeup: a write alias always points to the latest index, and rollover creates a new index transparently when the threshold is hit.

Answer

Since Elasticsearch 8.0 (Feb 2022), security is enabled by default, a huge change from 7.x where you had to opt in. On first start, the server auto-generates: a TLS certificate authority, certs for transport (node-to-node) and HTTP, and a random password for the `elastic` superuser. All HTTP traffic is HTTPS, all transport is TLS.

You configure access via the built-in role-based access control: users, roles, role mappings. The roles model is rich, index-level, document-level, field-level, and cluster-level permissions. API keys (issued via _security/api_key) are the recommended way to authenticate services, short-lived, scopeable, and revocable.

For SSO, Elastic supports SAML, OIDC, and Kerberos. The historical 'Elasticsearch ransomware' problem (exposed open clusters wiped by attackers) is mostly gone in 8.x because of these defaults, but you still must NOT bind 0.0.0.0 with discovery.type=single-node in production, which disables auth.

Answer

ES|QL (Elasticsearch Query Language) is a new pipe-based query language introduced in Elasticsearch 8.11 (Nov 2023), promoted to GA in 8.13. The syntax mirrors SQL/KQL/Splunk SPL: data flows through a pipeline of commands separated by `|`. Example: `FROM logs-* | WHERE @timestamp > NOW() - 1h | STATS count = COUNT(*) BY status_code | SORT count DESC | LIMIT 10`.

Key differences from Query DSL: (1) Compositional, you build complex analytics by chaining commands, no nested JSON. (2) New execution engine, ES|QL queries don't go through the same query/fetch path; they use a columnar 'compute' engine that's often 5-10× faster on analytics workloads. (3) SQL-friendly, easier for analysts coming from BI tools. (4) Supported commands include FROM, WHERE, STATS, EVAL, KEEP, DROP, RENAME, SORT, LIMIT, ENRICH, DISSECT, GROK, LOOKUP JOIN (8.15+). The Query DSL is not going away, it's still the way to do full-text search with relevance scoring and complex bool queries. ES|QL is targeted at the analytics/observability/security use case, where SQL-like ergonomics matter more than relevance scoring. Expect interview questions on when to choose which.

POST /_query
{
  "query": """
    FROM metrics-*
    | WHERE @timestamp > NOW() - 1h AND host.region == "ap-south-1"
    | STATS avg_cpu = AVG(cpu.pct), p95_cpu = PERCENTILE(cpu.pct, 95) BY host.name
    | WHERE avg_cpu > 0.8
    | SORT p95_cpu DESC
    | LIMIT 20
  """
}

Answer

Since 8.0 Elasticsearch supports a dense_vector field type backed by HNSW (Hierarchical Navigable Small World) for approximate nearest-neighbor search. The pattern: (1) Map a field as dense_vector with the correct dims (matches your embedding model, 768 for BERT, 1536 for OpenAI ada-002, 1024 for E5-large) and similarity (cosine, dot_product, l2_norm). (2) Generate embeddings at index time using your model, store them in the vector field alongside the text. (3) Query with a knn block on the search request. Production RAG patterns in 2026: (1) Hybrid search, combine knn (semantic) with a standard bool/match (lexical) using RRF (Reciprocal Rank Fusion, native in 8.9+) to get the best of both. (2) Quantization, int8 quantization reduces memory 4× with ~1% recall loss (`index_options.type: int8_hnsw`), int4 reduces 8× (8.14+). (3) Embedding via inference processor, set up a model in Elasticsearch (uploaded via Eland), then your ingest pipeline runs inference automatically.

This is what most India RAG teams now do; it eliminates the need for a separate vector DB like Pinecone or Weaviate. Note: Lucene's HNSW has a per-segment graph; force-merging to 1 segment before serving gives big query-latency wins.

PUT /docs
{
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "embedding": {
        "type": "dense_vector",
        "dims": 1024,
        "similarity": "cosine",
        "index_options": { "type": "int8_hnsw", "m": 16, "ef_construction": 100 }
      }
    }
  }
}

// Hybrid search, semantic (knn) + lexical (match) blended via RRF
GET /docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "title": "running shoe care" } } } },
        { "knn": { "field": "embedding", "query_vector": [/*1024 floats*/], "k": 50, "num_candidates": 100 } }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  }
}

Answer

The reference stack at this scale: (1) A dedicated 'search' Elasticsearch cluster, separate from logging, 20-40 data nodes, 3 dedicated master nodes, 2-4 coordinating-only nodes behind a load balancer. (2) Hardware: i3/i4 (NVMe SSDs), 64-128GB RAM per data node, half of RAM as heap up to 30GB (compressed oops cliff). (3) Index design: typically multiple indices (catalog_active, catalog_inactive) using aliases. Primary shards count = (total size / 30GB) rounded up, with 1-2 replicas for search throughput. (4) Aggressive use of filter context, filters on category, brand, price bucket, in_stock, geo all cached. (5) Custom analyzer per language (Hindi, Tamil, English), with synonym files loaded from S3 and reloaded via _reload_search_analyzers. (6) Multi-tier ranking: a fast first-stage retrieval (top 1000) using lexical + semantic, then a learning-to-rank model (LTR plugin) re-ranks the top 200 using business features (CTR, conversion, margin, seller rating). (7) Update strategy: avoid hot-doc writes during peak; daily catalog rebuilds reindex into a new index, swap aliases atomically. Inventory and price (high-update) live in a sidecar cache or are updated via partial-update with refresh_interval relaxed. (8) Observability: every search request gets a profile snapshot sampled at 1%, fed back to Kibana. SLA is typically p99 <200ms for the search API.

Answer

In 2021, Elastic NV changed Elasticsearch's license from Apache 2.0 to a dual SSPL/ELv2 license, making it source-available but no longer fully open source. AWS forked the last Apache-licensed version into OpenSearch and continues to develop it under the Apache 2.0 license. By 2026 the two projects have diverged meaningfully.

Elasticsearch (Elastic): faster on the search/ML/vector roadmap, ES|QL, ELSER (sparse embedding model bundled in), int4 quantization, more frequent BM25/HNSW improvements, the full commercial 'platinum' tier with machine learning, alerting, RBAC, and cross-cluster replication. OpenSearch (AWS / Linux Foundation as of 2024): truly open source (Apache 2.0), no per-feature commercial gating, default choice on AWS Managed Service. Has its own vector engine (uses both Lucene HNSW and Faiss), supports SQL and PPL (Piped Processing Language, a rough ES|QL analog).

Choose Elasticsearch when: you want the latest ML/vector/ES|QL features, you can afford the commercial license or run the basic tier, or you're already on Elastic Cloud. Choose OpenSearch when: license matters (some Indian govt and PSU contracts mandate Apache 2.0), you're deep in AWS and want managed service simplicity, or you need to embed it in a product you'll redistribute. Many India SaaS teams in 2026 stick with Elasticsearch for greenfield search and use OpenSearch for log analytics on AWS.

Answer

A systematic debug pass: (1) Reproduce with the profile API: GET /index/_search with `"profile": true`. The output shows per-shard, per-query-clause breakdown of time spent, you'll see which leaf query is dominating (often a phrase query, a wildcard, or a script). (2) Check if predicates are in filter context, anything that doesn't need scoring should be in bool.filter to hit the query cache. (3) Look at the explain output for one matching doc, `?explain=true`, to verify scoring isn't doing something pathological like loading fielddata on a high-cardinality text field. (4) Check shard sizes with _cat/shards, shards over 50GB or under 1GB are problematic. Too many small shards cause coordination overhead; too few large shards cause CPU starvation. (5) Look at hot threads: GET /_nodes/hot_threads, frequently shows the actual bottleneck (regex on wildcard, expensive aggs). (6) Check the slow log, index.search.slowlog.threshold.query.warn=1s captures queries crossing thresholds with their source. (7) Look at the field types, a sneaky cause is sorting/aggregating on a `text` field via fielddata, which loads all terms into JVM heap.

Always sort/agg on the `.keyword` subfield. (8) Aggregation memory: high-cardinality terms aggs can blow the circuit breaker. Use `cardinality` for approximate counts, or switch to a `composite` agg with pagination. (9) Network: cross-cluster searches and deeply paginated requests (from=10000) are slow by design, switch to search_after / PIT. (10) If all else fails, force-merge to 1 segment on read-only indices, fewer segments = faster queries (but expensive operation, never on hot indices).