Cassandra Interview Questions and Answers

Answer

Cassandra hashes the partition key with Murmur3Partitioner (the default since 1.2) into a 64-bit token somewhere in the range -2^63 to 2^63-1. That token space is the ring. Each node owns a set of token ranges, and with virtual nodes each physical node owns many small ranges instead of one contiguous slice, which is what makes streaming, bootstrap and rebuild parallel across the whole cluster.

Recent versions ship num_tokens: 16 in cassandra.yaml, while clusters built years ago are usually still on 256, and you should pair a low vnode count with allocate_tokens_for_local_replication_factor so ownership stays balanced. To find the primary replica the coordinator walks clockwise from the token until it hits the first node that owns it, then keeps walking to collect RF distinct replicas. With NetworkTopologyStrategy it skips candidates until it has replicas in distinct racks inside each datacenter, using rack and DC labels that the snitch reads (GossipingPropertyFileSnitch reads cassandra-rackdc.properties).

Gossip exchanges membership and state between nodes roughly every second, so every node knows the full ring and any node can act as coordinator. A token-aware driver skips the extra hop by sending the request straight to a replica. The classic gotcha: you cannot simply change num_tokens on a running node, the supported path is to stand up a new datacenter with the new setting and rebuild into it.

-- See the token a partition key hashes to
SELECT token(user_id), user_id FROM app.users LIMIT 5;

-- Which nodes actually hold it?
$ nodetool getendpoints app users 7f3a9c21-0c4e-4a4b-9f2d-11c3e4f5a6b7
10.0.1.24
10.0.2.31
10.0.3.18

-- Ownership and token counts across the ring
$ nodetool ring | head
$ nodetool status app

# cassandra.yaml (per node)
num_tokens: 16
allocate_tokens_for_local_replication_factor: 3
endpoint_snitch: GossipingPropertyFileSnitch

# cassandra-rackdc.properties
dc=blr-1
rack=rack-a

Answer

A Cassandra PRIMARY KEY has two parts. The first component (or the parenthesised group of components) is the partition key, which decides the token and therefore which nodes store the data. Everything after it is a clustering column, which decides the sort order of rows inside that one partition on disk.

So PRIMARY KEY ((tenant_id, day), created_at, event_id) means the composite of tenant_id and day picks the node set, and rows within that partition are stored physically sorted by created_at then event_id. This layout drives every query restriction people trip over. You must supply the full partition key with equality (or IN) in the WHERE clause, because without it Cassandra has no idea which node to ask.

Clustering columns can be restricted with equality or ranges, but only left to right, you cannot skip event_id's parent and range on event_id alone. Ordering with ORDER BY is limited to the clustering order or its exact reverse, which is why you set WITH CLUSTERING ORDER BY (created_at DESC) at create time if newest-first is your dominant read. The interviewer is usually checking two things: whether you understand that the partition key controls distribution while clustering columns control on-disk sort, and whether you know that a bad partition key choice (something low cardinality like a country code, or something monotonically increasing like a bare date) creates hot partitions no amount of tuning will save you from.

CREATE TABLE app.order_events (
  tenant_id   uuid,
  day         date,
  created_at  timestamp,
  event_id    timeuuid,
  status      text,
  amount      decimal,
  PRIMARY KEY ((tenant_id, day), created_at, event_id)
) WITH CLUSTERING ORDER BY (created_at DESC, event_id DESC);

-- Legal: full partition key + left-to-right clustering restriction
SELECT * FROM app.order_events
 WHERE tenant_id = ? AND day = '2026-08-11'
   AND created_at >= '2026-08-11 09:00:00'
 LIMIT 200;

-- Illegal without ALLOW FILTERING: partition key incomplete
SELECT * FROM app.order_events WHERE tenant_id = ?;

-- Illegal: skipping a clustering column
SELECT * FROM app.order_events
 WHERE tenant_id = ? AND day = ? AND event_id = ?;

Answer

The client sends the mutation to a coordinator, which computes the token, finds the replicas, and forwards the mutation to all of them in parallel. On each replica the write is appended to the commit log (durability) and applied to the memtable (an in-memory sorted structure per table). Once enough replicas acknowledge to satisfy the requested consistency level, the coordinator returns success.

Note what is not in that list: there is no read before the write, no uniqueness check, no constraint validation. That is precisely why Cassandra writes are fast and why every write is really an upsert. INSERT and UPDATE compile to the same thing internally.

When the memtable exceeds its threshold (memtable_heap_space plus memtable_cleanup_threshold, or a commit-log size limit, or an explicit nodetool flush), it is flushed to an immutable SSTable and the corresponding commit-log segments are recycled. Compaction later merges SSTables, resolving duplicate cells by timestamp and dropping expired data. Commit-log durability is controlled by commitlog_sync: periodic (default, fsync every commitlog_sync_period, faster but a crash can lose the last window) versus batch or group (fsync before ack, safer, slower). Production gotchas interviewers like: writes that touch collections of type list, counters, materialized views, or lightweight transactions do read before writing and are therefore much more expensive; and an unavailable replica does not fail the write if the consistency level is still satisfiable, the coordinator just stores a hint.

# cassandra.yaml knobs that shape the write path
commitlog_sync: periodic
commitlog_sync_period: 10000ms
commitlog_segment_size: 32MiB
memtable_allocation_type: offheap_objects
memtable_flush_writers: 4
hinted_handoff_enabled: true
max_hint_window: 3h

# Force a flush before taking a snapshot or restarting
$ nodetool flush app order_events

# Flush + stop accepting writes, the correct pre-restart step
$ nodetool drain

# Watch memtable size and flush counts per table
$ nodetool tablestats app.order_events | grep -E 'Memtable|SSTable count'

Answer

A read is a merge. The replica has to reconstruct the current value of each requested cell from potentially several places: the memtable, and every SSTable that could contain the partition. It checks the row cache first if enabled (it is off by default, row_cache_size_in_mb: 0, and is only sensible for tiny, extremely hot, read-mostly partitions).

Then for each SSTable it consults the bloom filter, a probabilistic off-heap structure sized by bloom_filter_fp_chance, which cheaply says definitely not here or maybe here. For a maybe, the key cache may already hold the offset; otherwise the partition index (or the trie-based BTI index in 5.0) is consulted to seek into the data file via compression offsets. Cells from all sources are merged by write timestamp, last write wins, and tombstones shadow older data.

Finally read repair may compare digests from other replicas and push the newest version back to the stale ones. The single most useful number here is SSTables per read, visible in nodetool tablehistograms. Under Leveled Compaction it should typically be one or two; under Size Tiered it grows, and if you see double digits on a latency-sensitive table your compaction is lagging or your strategy is wrong.

Tombstones scanned per read is the sibling metric and the cause of most read latency incidents. Turn on TRACING in cqlsh to see all of this per query.

cqlsh> TRACING ON;
cqlsh> SELECT * FROM app.order_events
   ... WHERE tenant_id = 7f3a... AND day = '2026-08-11' LIMIT 50;
-- trace shows: Bloom filter allows N sstables, Merged data from memtable and 3 sstables,
-- Read 50 live rows and 4213 tombstone cells   <-- the smoking gun

# Percentiles for SSTables per read and partition size
$ nodetool tablehistograms app order_events
Percentile  SSTables  Write Latency  Read Latency  Partition Size  Cell Count
50%              1.00         35.43        124.00            2299          60
99%              4.00        155.00       1109.00          126934        3311

# Cache hit rates
$ nodetool info | grep -E 'Key Cache|Row Cache|Chunk Cache'

Answer

Cassandra lets you choose, per statement, how many replicas must acknowledge before the coordinator returns. The levels are ANY, ONE, TWO, THREE, LOCAL_ONE, QUORUM, LOCAL_QUORUM, EACH_QUORUM and ALL, plus SERIAL and LOCAL_SERIAL for lightweight transactions. The rule that matters is R + W > RF: if the number of replicas read plus the number written exceeds the replication factor, the read set and the write set must overlap on at least one replica holding the newest value, which gives you strong consistency for that key.

With RF=3, LOCAL_QUORUM on both reads and writes gives 2 + 2 > 3 and is the standard production choice inside a single datacenter. ONE on both is eventually consistent and is fine for metrics, feeds, and anything where a stale read is harmless. ALL gives you no fault tolerance at all, one node down and every request fails, so it is almost never right.

ANY is the trap answer: it lets the coordinator satisfy a write by storing a hint even when no replica took it, so the data is not readable at any consistency level until the hint is delivered. Interviewers also probe the failure semantics: a WriteTimeoutException does not mean the write failed, it means not enough replicas acked in time, and the mutation may still land on some replicas. Because writes are idempotent upserts, the correct response is usually to retry.

cqlsh> CONSISTENCY LOCAL_QUORUM;
Consistency level set to LOCAL_QUORUM.

// DataStax Java driver 4.x, per-statement override
SimpleStatement stmt = SimpleStatement.newInstance(
        "SELECT balance FROM ledger.accounts WHERE account_id = ?", accountId)
    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM)
    .setIdempotent(true);
Row row = session.execute(stmt).one();

# Python driver, set a default on the execution profile
from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT
from cassandra import ConsistencyLevel

profile = ExecutionProfile(consistency_level=ConsistencyLevel.LOCAL_QUORUM)
cluster = Cluster(['10.0.1.24'], execution_profiles={EXEC_PROFILE_DEFAULT: profile})

Answer

Production keyspaces use NetworkTopologyStrategy with an explicit replication factor per datacenter. SimpleStrategy ignores racks and datacenters entirely, places replicas by walking the ring, and is deprecated in recent versions, so using it outside a laptop is an immediate red flag in an interview. NetworkTopologyStrategy takes a map of datacenter name to RF, where the datacenter names must match exactly what the snitch reports in cassandra-rackdc.properties and what nodetool status prints.

RF=3 per datacenter is the near universal choice: it tolerates one replica loss while still satisfying LOCAL_QUORUM, and it spreads replicas across three racks or availability zones if you have labelled them properly. RF=2 means LOCAL_QUORUM needs both replicas, so a single node loss makes the partition unavailable, and RF greater than 3 mostly buys read parallelism at the cost of write amplification. durable_writes defaults to true and should stay true, setting it false skips the commit log and trades durability for a small write gain, which is only defensible for a scratch keyspace you can rebuild. After changing replication you must run nodetool repair on the affected keyspace, because ALTER KEYSPACE only changes the placement rules, it does not move a single byte of existing data.

CREATE KEYSPACE IF NOT EXISTS ledger
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'blr-1': 3,
  'mum-1': 3
}
AND durable_writes = true;

-- Verify the DC names the snitch is reporting
$ nodetool status
Datacenter: blr-1
=================
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID   Rack
UN  10.0.1.24   184.2 GiB  16      33.4%             7f3a...   rack-a

-- Never do this in production
CREATE KEYSPACE demo WITH replication =
  {'class': 'SimpleStrategy', 'replication_factor': 1};

-- After any replication change
$ nodetool repair -full ledger

Answer

QUORUM counts replicas across the entire cluster. With RF=3 in Bengaluru and RF=3 in Mumbai, total RF is 6 and a global QUORUM needs 4 acknowledgements, which forces at least one cross-region round trip on every single request. Inter-region latency inside India is roughly 20 to 40 ms one way, so you have just added that to the p50 of every write, and you have coupled your availability to the health of the remote region: lose two nodes in Mumbai and Bengaluru requests start failing even though Bengaluru is perfectly healthy.

LOCAL_QUORUM counts only replicas in the coordinator's own datacenter, so with RF=3 locally it needs 2 acks and stays inside the region. Writes still replicate asynchronously to the other datacenter, and anti-entropy repair plus hinted handoff close the gap. This gives you the usual production shape: application servers in each region pin their driver to the local datacenter, both reads and writes run at LOCAL_QUORUM, and each region survives the other going dark.

EACH_QUORUM exists for the rare write that must be durable in both regions before you acknowledge, for example a compliance-critical ledger entry, and it is normally used only for writes because EACH_QUORUM is not supported for reads. The interviewer's follow-up is usually about read-your-writes across regions: LOCAL_QUORUM gives you that only within one datacenter, a user whose session moves regions can read stale data until replication catches up.

Answer

SSTables are immutable, so Cassandra cannot erase a value in place. A DELETE writes a new marker called a tombstone, carrying a timestamp, which shadows every older value for that cell, row or range. During a read, the merge step has to load those tombstones and apply them before it can decide what is live, so a partition where you have deleted 50,000 rows costs you the work of reading 50,000 markers even if only ten rows remain.

That is why heavy-delete workloads on Cassandra degrade in a way that surprises people coming from Postgres. Tombstones come in several shapes: a cell tombstone from deleting one column, a row tombstone from DELETE FROM ... WHERE full-primary-key, a range tombstone from deleting a clustering slice, and a partition tombstone from deleting the whole partition (the cheapest, because one marker shadows everything).

The trap almost everyone hits is that inserting NULL creates a tombstone. Writing an unset optional field as null through a naive ORM or a prepared statement bound with null generates a delete marker per column per row, and a bulk load can manufacture millions of tombstones without a single DELETE statement. Use unset values in the driver instead. The guardrails are tombstone_warn_threshold (1000 by default, logged) and tombstone_failure_threshold (100000, which aborts the query with TombstoneOverwhelmingException).

-- Four different tombstone shapes
DELETE status FROM app.order_events WHERE tenant_id=? AND day=? AND created_at=? AND event_id=?; -- cell
DELETE FROM app.order_events WHERE tenant_id=? AND day=? AND created_at=? AND event_id=?;        -- row
DELETE FROM app.order_events WHERE tenant_id=? AND day=? AND created_at < '2026-08-01';          -- range
DELETE FROM app.order_events WHERE tenant_id=? AND day=?;                                         -- partition

-- This INSERT writes a tombstone for `status`, silently
INSERT INTO app.order_events (tenant_id, day, created_at, event_id, status)
VALUES (?, ?, ?, ?, null);

// Java driver: bind unset, not null, for absent values
BoundStatement bs = prepared.bind()
    .setUuid("tenant_id", tenantId)
    .setLocalDate("day", day);
bs = (status == null) ? bs.unset("status") : bs.setString("status", status);

# cassandra.yaml
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000

Answer

gc_grace_seconds is the minimum age a tombstone must reach before compaction is allowed to drop it. The default is 864000, which is ten days. It exists because a delete is just another write that has to propagate.

If a replica was down when the DELETE happened, and you purged the tombstone before that replica came back and got repaired, the replica would still hold the old live value. Anti-entropy would then see one node with data and two with nothing, conclude the data is newer, and resurrect the deleted row. That zombie-data scenario is the whole reason for the grace window: it is the deadline by which every replica must have been repaired.

So the real contract is that you must complete a repair of every table within gc_grace_seconds, every cycle, forever. If your repairs take twelve days and gc_grace_seconds is ten, you are risking resurrection. If repairs finish comfortably in two days, lowering gc_grace_seconds to something like three days reclaims disk faster and shortens read scans.

Setting it to zero is only defensible for a table you never delete from and never update in a way that shadows data, or for a strict TWCS time-series table where whole SSTables are dropped on expiry and you accept the risk. The other thing candidates forget: TTL expiry also produces tombstones, and those obey gc_grace_seconds too, so a TTL table with the default grace holds expired rows on disk for ten extra days.

-- Inspect and tune per table
DESCRIBE TABLE app.order_events;

ALTER TABLE app.sessions WITH gc_grace_seconds = 259200;  -- 3 days

-- Only safe when the table is append-only or strictly TWCS+TTL
ALTER TABLE metrics.raw_1s WITH gc_grace_seconds = 0;

-- The obligation this creates: finish repair inside the window
$ nodetool repair -full -pr app
$ nodetool tpstats | grep -i repair

-- Force purge of droppable tombstones on one table (heavy, off-peak only)
$ nodetool garbagecollect app order_events

Answer

TTL is set per cell, not per row. When you write with USING TTL 86400, each cell gets a local deletion time equal to the write time plus the TTL. Once that moment passes, the cell is treated as an expiring tombstone: reads skip it immediately, but the marker stays on disk until compaction can drop it, and it only becomes droppable after gc_grace_seconds have elapsed past expiry.

You can set default_time_to_live on the table so every write inherits it, and you can inspect remaining lifetime with the TTL() function and the original write time with WRITETIME(). Several details catch people out. Updating a cell without specifying USING TTL resets that cell to no TTL at all if the table has no default_time_to_live, so a partial update can accidentally make one column immortal while its siblings expire.

TTL applies to cells, so a row whose every cell has expired still leaves a row shell until compaction cleans up. Primary key columns cannot carry a TTL by themselves. And the big operational point: TTL at scale generates enormous tombstone volume under Size Tiered compaction, because expired cells sit inside SSTables mixed with live data and only disappear when those SSTables happen to get compacted. The correct pairing is TimeWindowCompactionStrategy, which groups writes by time window so an entire SSTable expires at once and can be dropped whole, with no tombstone scanning at all.

-- Per-statement TTL, in seconds
INSERT INTO app.sessions (session_id, user_id, created_at)
VALUES (?, ?, toTimestamp(now())) USING TTL 3600;

-- Table default, applied to every write that does not override it
CREATE TABLE metrics.raw_1s (
  device_id text,
  bucket    text,
  ts        timestamp,
  value     double,
  PRIMARY KEY ((device_id, bucket), ts)
) WITH default_time_to_live = 604800   -- 7 days
  AND gc_grace_seconds = 3600
  AND compaction = {
    'class': 'TimeWindowCompactionStrategy',
    'compaction_window_unit': 'HOURS',
    'compaction_window_size': 6
  };

-- Inspect what is left and when it was written
SELECT ts, value, TTL(value), WRITETIME(value)
  FROM metrics.raw_1s WHERE device_id=? AND bucket='2026-08-11';

Answer

Because there are no joins, and because the partition key is the only cheap way to find data. In a relational database you normalise once and let the query planner figure out access paths with indexes and joins. In Cassandra you invert the process: you list the access patterns first, then create one table per access pattern, each with a primary key shaped exactly for that query.

If the product needs orders by customer, orders by status for an ops dashboard, and a single order by its id, that is three tables holding overlapping data. Storage is the cheap resource, random reads across the ring are the expensive one. Writes are cheap enough that fanning one logical event into three or four tables is normal.

The consequence you must own in the interview is consistency between those tables. Cassandra will not keep them in sync for you. Options are to write all copies from the application (accepting that a crash between writes leaves a divergence you reconcile later), to wrap same-partition-key writes in a logged BATCH which gives atomicity but not isolation, or to derive the copies from an event log in Kafka so a replay repairs them.

Materialized views promise to do this automatically but remain disabled by default in recent versions for good reasons. The senior answer names the reconciliation strategy, not just the duplication.

-- Access pattern 1: recent orders for one customer
CREATE TABLE shop.orders_by_customer (
  customer_id uuid,
  created_at  timestamp,
  order_id    uuid,
  status      text,
  total       decimal,
  PRIMARY KEY (customer_id, created_at, order_id)
) WITH CLUSTERING ORDER BY (created_at DESC, order_id ASC);

-- Access pattern 2: ops dashboard, orders in one status for one day
CREATE TABLE shop.orders_by_status_day (
  status     text,
  day        date,
  created_at timestamp,
  order_id   uuid,
  customer_id uuid,
  PRIMARY KEY ((status, day), created_at, order_id)
) WITH CLUSTERING ORDER BY (created_at DESC, order_id ASC);

-- Access pattern 3: point lookup by order id
CREATE TABLE shop.orders_by_id (
  order_id uuid PRIMARY KEY,
  customer_id uuid,
  status text,
  total decimal,
  created_at timestamp
);

Answer

ALLOW FILTERING tells the coordinator that you accept a query which cannot be answered by seeking directly to the requested data. Without it, Cassandra rejects any query where it would have to read rows and discard them. With it, the coordinator will read candidate rows and filter in memory, and if the partition key is not fully specified it will do that across every node in the ring.

On a table with a few hundred million rows that is a cluster-wide scan that saturates read stages, blows through read_request_timeout, and can take a production cluster down while returning an error to the caller. It is the single most common way an unaware developer causes an outage. There are two situations where it is genuinely fine.

First, when the full partition key is supplied and you are only filtering on a non-clustering column inside one bounded partition, the work is limited to that partition and the cost is predictable. Second, in ad-hoc analytics against a small table or through a Spark connector that intentionally does a full scan with token-range splits. Everything else should be solved by adding a table shaped for the query, or by a Storage Attached Index in 5.0 for genuinely low-selectivity ad-hoc predicates. Recent versions also ship guardrails, so operators can set allow_filtering_enabled: false cluster-wide and simply prevent applications from using it at all, which is what mature teams do.

-- Rejected: no partition key restriction
SELECT * FROM shop.orders_by_customer WHERE status = 'FAILED';
-- InvalidRequest: Cannot execute this query as it might involve data filtering
-- and thus may have unpredictable performance.

-- Dangerous: cluster-wide scan
SELECT * FROM shop.orders_by_customer WHERE status = 'FAILED' ALLOW FILTERING;

-- Acceptable: bounded to a single partition
SELECT * FROM shop.orders_by_customer
 WHERE customer_id = ? AND status = 'FAILED' ALLOW FILTERING;

# cassandra.yaml: stop it at the cluster level
guardrails:
  allow_filtering_enabled: false
  page_size_warn_threshold: 5000

Answer

A static column belongs to the partition rather than to any individual row inside it. In a table with clustering columns, marking a column STATIC means there is exactly one value of it per partition key, shared by every clustering row. The classic use is partition-level metadata that you do not want to repeat on ten thousand rows: the customer name on a table of that customer's orders, the device model on a table of that device's readings, a per-conversation title on a table of messages.

Two practical benefits follow. Storage shrinks, because the value is written once per partition instead of once per row. And you can read or update the metadata without touching the clustering rows at all, which means a SELECT of just static columns with only the partition key restricted is a legal and cheap query.

There are real limits. A table with no clustering columns cannot have static columns, because every row would already be its own partition. Static columns cannot be part of the primary key, and they cannot be part of the partition key by definition.

Lightweight transactions behave differently on them: a conditional update on a static column is applied at partition granularity. And the concurrency gotcha interviewers like is that two clients updating the same static column from different rows are simply racing on the same cell, and last write wins by timestamp resolves it, which is not always what the application intended.

CREATE TABLE chat.messages (
  conversation_id uuid,
  title           text STATIC,   -- one value per conversation
  participants    set<uuid> STATIC,
  sent_at         timestamp,
  message_id      timeuuid,
  sender_id       uuid,
  body            text,
  PRIMARY KEY (conversation_id, sent_at, message_id)
) WITH CLUSTERING ORDER BY (sent_at DESC, message_id DESC);

-- Update partition metadata without writing any message row
UPDATE chat.messages SET title = 'Release war room'
 WHERE conversation_id = ?;

-- Read only the static part, cheap, no clustering scan
SELECT conversation_id, title, participants
  FROM chat.messages WHERE conversation_id = ?;

-- Conditional update at partition granularity
UPDATE chat.messages SET title = 'Archived'
 WHERE conversation_id = ? IF title = 'Release war room';

Answer

A counter is a distributed 64-bit value you increment or decrement rather than set. Cassandra keeps per-replica shards internally and resolves them on read, which is how it avoids a global lock. The rules are unusually strict.

A counter column can only live in a table where every non-primary-key column is a counter, so you cannot mix a counter and a text column in one table. You cannot INSERT into a counter table, only UPDATE with an increment expression. Counters cannot carry a TTL, cannot be indexed, and their columns cannot be part of the primary key.

The behavioural difference that matters most in interviews is idempotence. A counter update is read-modify-write on the replica, so it is not idempotent: if the coordinator times out and your driver retries, the increment can be applied twice, and there is no way for Cassandra to detect the duplicate. That is why the DataStax drivers treat counter statements as non-idempotent by default and refuse to retry them on write timeouts.

Counters are also more expensive than normal writes, they involve a local read on the replica, and heavy counter workloads on a hot partition produce contention that shows up as elevated CounterMutationStage latency in nodetool tpstats. For accurate money-like counting, most teams either accumulate events in Kafka and periodically write an authoritative total, or use lightweight transactions on a normal bigint despite the cost.

CREATE TABLE analytics.page_views (
  page_id text,
  day     date,
  views   counter,
  uniques counter,
  PRIMARY KEY ((page_id, day))
);

-- No INSERT allowed, only UPDATE with +/-
UPDATE analytics.page_views SET views = views + 1
 WHERE page_id = '/jobs/data-engineer' AND day = '2026-08-11';

UPDATE analytics.page_views SET views = views - 3
 WHERE page_id = '/jobs/data-engineer' AND day = '2026-08-11';

-- Rejected: mixing counter and non-counter columns
CREATE TABLE bad.mix (id text PRIMARY KEY, hits counter, label text);
-- InvalidRequest: Cannot mix counter and non counter columns in the same table

$ nodetool tpstats | grep -i counter
CounterMutationStage   0   0   184203   0   0

Answer

A non-frozen collection is stored as individual cells, one per element, all sharing the row's primary key. That is what makes element-level updates possible: adding one item to a set writes one cell, and removing one writes one tombstone, without rewriting the whole collection. A FROZEN collection is serialised into a single blob cell, so any change rewrites the entire value, but in exchange it can be used as a primary key component, nested inside another collection, or compared for equality.

The three types are not equivalent operationally. Sets and maps are well behaved: add and remove touch only the affected elements. Lists are the problem child.

Because a list preserves order and allows duplicates, operations like setting by index or removing by value require a read-before-write on the replica, and prepending or replacing the whole list writes a range tombstone covering the old contents. Recent versions expose read_before_write_list_operations_enabled as a guardrail so operators can disable those patterns outright. The standard advice is to model with a set or a map wherever order does not matter, and to promote genuinely ordered data into clustering columns instead of a list.

Size is the other constraint: collections are read entirely into memory, there is no paging inside one, so a collection with thousands of elements bloats every read of that row. Guardrails such as items_per_collection_warn_threshold and collection_size_warn_threshold exist precisely because teams keep growing collections without noticing.

CREATE TABLE app.profiles (
  user_id   uuid PRIMARY KEY,
  skills    set<text>,                 -- element-level updates
  meta      map<text, text>,
  history   frozen<list<text>>,        -- single blob, replaced wholesale
  location  frozen<tuple<double,double>>
);

-- Cheap: writes one cell / one tombstone
UPDATE app.profiles SET skills = skills + {'cassandra'} WHERE user_id = ?;
UPDATE app.profiles SET skills = skills - {'hbase'}     WHERE user_id = ?;
UPDATE app.profiles SET meta['tier'] = 'gold'           WHERE user_id = ?;

-- Expensive: full overwrite writes a range tombstone first
UPDATE app.profiles SET skills = {'cassandra','kafka'}  WHERE user_id = ?;

# cassandra.yaml guardrails worth enabling
guardrails:
  read_before_write_list_operations_enabled: false
  items_per_collection_warn_threshold: 20
  collection_size_warn_threshold: 16KiB

Answer

A UDT groups related fields into one named type so you can reuse the shape across tables and keep a row readable. An address with line1, city, state, pincode is the canonical example: as flat columns it clutters every table that needs it, as a map<text,text> it loses typing, and as a UDT it stays typed and named. Inside a table a UDT is usually declared frozen, which means the whole value is serialised into one cell and any change replaces it entirely.

Non-frozen UDTs exist in recent versions and permit updating individual fields, but frozen remains the common and safer choice, especially if the UDT appears inside a collection or as part of a primary key, where frozen is mandatory. The important trade-off is schema evolution. You can ALTER TYPE ...

ADD a field, and old rows simply return null for it, which is fine. You cannot remove a field, and renaming is restricted, so a UDT is close to a one-way door. That is why many teams reach for a UDT only where the shape is genuinely stable, and otherwise use flat columns, which are trivially addable and droppable. The other consideration is query shape: you cannot filter on a field inside a frozen UDT without ALLOW FILTERING or a Storage Attached Index, so anything you need to search on should be a real column, not buried inside a type.

CREATE TYPE shop.address (
  line1   text,
  line2   text,
  city    text,
  state   text,
  pincode text
);

CREATE TABLE shop.customers (
  customer_id uuid PRIMARY KEY,
  name        text,
  city        text,                          -- duplicated flat for querying
  billing     frozen<shop.address>,
  shipping    list<frozen<shop.address>>     -- frozen required inside a collection
);

INSERT INTO shop.customers (customer_id, name, city, billing)
VALUES (uuid(), 'Ananya R',
        'Bengaluru',
        {line1:'12 Residency Rd', city:'Bengaluru', state:'KA', pincode:'560025'});

-- Additive evolution only
ALTER TYPE shop.address ADD country text;

Answer

nodetool status is the first command anyone runs on a Cassandra cluster. The two-letter code is state plus status: the first character is Up or Down as seen by gossip, the second is the node's operational state, N for Normal, L for Leaving (decommissioning), J for Joining (bootstrapping), M for Moving. So UN is healthy, DN is a node gossip believes is dead, UJ is a node still streaming data in and not yet serving reads for its ranges.

Load is the on-disk size of SSTables for that node, including data that has not yet been compacted away, so a node mid-compaction or holding many snapshots looks fatter than its peers. Tokens is the vnode count, and if you see mixed values across nodes someone changed num_tokens without a proper DC migration. Owns shows the percentage of the token space the node is responsible for, and when you pass a keyspace name it becomes effective ownership including replication, which is the number you actually want.

Host ID is the stable identity used by nodetool removenode and by replacement procedures. Rack matters because NetworkTopologyStrategy uses it for replica placement, and a cluster where every node reports the same rack has no rack-level fault tolerance no matter how many availability zones the instances are actually spread across. Two red flags to call out in an interview: Owns skew of more than a few percent, and a Load that differs sharply between replicas of the same ranges.

$ nodetool status ledger
Datacenter: blr-1
=================
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
--  Address     Load        Tokens  Owns (effective)  Host ID    Rack
UN  10.0.1.24   184.21 GiB  16      33.4%             7f3a9c21   rack-a
UN  10.0.1.25   181.06 GiB  16      33.2%             a10b4d55   rack-b
DN  10.0.1.26   402.77 GiB  16      33.4%             c92e7f08   rack-c

# Follow-ups when something looks wrong
$ nodetool describecluster        # schema agreement across nodes
$ nodetool netstats               # streaming and pending messages
$ nodetool compactionstats -H     # is compaction the reason Load is high?
$ nodetool gossipinfo | grep -A3 10.0.1.26

Answer

cqlsh is more than a REPL, and knowing four of its directives separates people who have debugged Cassandra from people who have read about it. TRACING ON attaches a full server-side trace to every subsequent query, printing which replicas were contacted, how many SSTables the bloom filters allowed through, how many live rows and tombstone cells were merged, and the microseconds spent in each stage. That trace is the fastest way to prove whether a slow SELECT is a tombstone problem, a too-many-SSTables problem, or a cross-datacenter problem.

CONSISTENCY sets the level for the session so you can reproduce exactly what the application does, and SERIAL CONSISTENCY does the same for lightweight transactions. EXPAND ON switches output to a vertical, one-field-per-line format, which is the only way to read a wide row without wrapping. PAGING sets or disables the fetch size when you want to see how a query behaves under a particular page size.

DESCRIBE KEYSPACE and DESCRIBE TABLE print the full DDL including compaction, compression, gc_grace_seconds and caching options, which is what you paste into a ticket. COPY TO and COPY FROM handle small CSV imports and exports, useful for fixtures, but they run through the normal write path and are the wrong tool for bulk loading anything large, where sstableloader or a Spark job belongs instead. Finally, cqlsh --request-timeout is worth remembering, because the default client timeout will kill a legitimately slow diagnostic query before the server does.

$ cqlsh 10.0.1.24 -u appuser -p '***' --request-timeout=60

cqlsh> CONSISTENCY LOCAL_QUORUM;
cqlsh> SERIAL CONSISTENCY LOCAL_SERIAL;
cqlsh> PAGING 100;
cqlsh> EXPAND ON;
cqlsh> TRACING ON;

cqlsh> SELECT * FROM app.order_events
   ... WHERE tenant_id = 7f3a9c21-0c4e-4a4b-9f2d-11c3e4f5a6b7
   ...   AND day = '2026-08-11';

cqlsh> DESCRIBE TABLE app.order_events;

-- Small fixture export / import only
cqlsh> COPY app.order_events (tenant_id, day, created_at, event_id, status)
   ... TO '/tmp/events.csv' WITH HEADER = true;

Answer

Two separate wins that people often conflate. Preparing a statement sends the CQL string to the cluster once, where it is parsed and cached against a query id, and every subsequent execution ships only that id plus the bound values. You save parsing on the server and string building on the client, and you get correct type handling and protection from CQL injection.

Prepare once at startup and keep the PreparedStatement, preparing inside a request handler on every call is a classic bug that shows up as growing prepared-statement cache pressure and log warnings about re-preparing. The second win is routing. A prepared statement carries routing information (the keyspace and the serialised partition key), so the driver's default load balancing policy can compute the token itself and send the request directly to a replica that owns it, in the local datacenter, shuffled among replicas to spread load.

An unprepared SimpleStatement has no routing key unless you set one manually, so the driver picks a coordinator that then has to forward the request, adding a network hop and extra coordinator work. In driver 4.x the local datacenter is mandatory: CqlSession.builder().withLocalDatacenter(...) or the basic.load-balancing-policy.local-datacenter setting in application.conf. Omitting it fails fast at startup by design, because silently spraying traffic across regions was the single most common misconfiguration in the 3.x era.

import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.cql.*;

CqlSession session = CqlSession.builder()
    .addContactPoint(new InetSocketAddress("10.0.1.24", 9042))
    .withLocalDatacenter("blr-1")          // mandatory in driver 4.x
    .withKeyspace("shop")
    .build();

// Prepare ONCE, at startup, and hold the reference
private final PreparedStatement byCustomer = session.prepare(
    "SELECT order_id, status, total FROM orders_by_customer "
  + "WHERE customer_id = ? AND created_at > ?");

BoundStatement bound = byCustomer.bind(customerId, since)
    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM)
    .setIdempotent(true)
    .setPageSize(500);

for (Row row : session.execute(bound)) {
  handle(row.getUuid("order_id"), row.getString("status"));
}

Answer

Cassandra never returns an unbounded result set in one message. The driver sets a page size (5000 rows by default in the DataStax drivers) and the server returns that many rows plus an opaque paging state, a binary cursor describing where to resume. Iterating the result set transparently fetches the next page when you cross the boundary, which is why a for-each over a ResultSet can silently make many round trips.

Two things follow. First, page size is a real tuning knob: pages that are too large cause coordinator heap pressure and long GC pauses on wide rows, pages that are too small multiply round trips. Recent versions ship a page_size_warn_threshold guardrail so operators can catch applications requesting enormous pages.

Second, the paging state is how you build stateless pagination in an HTTP API. You return the base64-encoded state as a cursor to the client, and the next request resumes exactly there, which is dramatically cheaper than offset-based pagination (Cassandra has no OFFSET at all). The caveats worth naming in an interview: a paging state is tied to the exact query and to the cluster's data layout, it is not a durable bookmark and should not be persisted for long or handed between different queries, and it is not a snapshot, so rows inserted between pages may or may not appear. For genuine full-table iteration, do not page a naive SELECT *, split by token ranges instead, which is what the Spark connector does.

from cassandra.cluster import Cluster
from cassandra.query import SimpleStatement
import base64

session = Cluster(['10.0.1.24']).connect('shop')

stmt = SimpleStatement(
    "SELECT order_id, status FROM orders_by_status_day "
    "WHERE status=%s AND day=%s",
    fetch_size=500)

# First page
rs = session.execute(stmt, ('FAILED', '2026-08-11'))
page = list(rs.current_rows)
cursor = base64.b64encode(rs.paging_state).decode() if rs.has_more_pages else None

# Resume from a client-supplied cursor, no OFFSET needed
state = base64.b64decode(cursor) if cursor else None
rs2 = session.execute(stmt, ('FAILED', '2026-08-11'), paging_state=state)

# Full-table iteration: split by token range, do not page SELECT *
rows = session.execute(
    "SELECT * FROM orders_by_id WHERE token(order_id) > %s AND token(order_id) <= %s",
    (start_token, end_token))

Answer

CQL BATCH is not a performance optimisation and it is not a transaction. A logged batch writes the statements to a batchlog on two replicas first, then applies them, and retries any that fail until they all land. That gives you atomicity in the sense of eventually all or nothing, but no isolation whatsoever: other readers can see partial results while the batch is applying.

The cost is real, a logged batch is roughly two extra writes plus coordination, so using it to group unrelated inserts makes throughput worse, not better. The one case where a batch is genuinely good is when every statement targets the same partition. Then it becomes a single mutation on a single replica set, it is atomic and isolated for that partition, and it is cheaper than the equivalent individual writes.

The other legitimate case is a logged batch across the denormalised copies of the same logical entity, where you accept the cost in exchange for the guarantee that orders_by_customer and orders_by_status_day do not diverge. UNLOGGED BATCH skips the batchlog and gives you nothing except fewer round trips, which only helps when the statements share a partition. The failure mode to name in interviews is batch size: batch_size_warn_threshold defaults to 5KiB and batch_size_fail_threshold to 50KiB, and an application that loops a thousand inserts into one batch will hit Batch too large and fail, or worse will pass the threshold and pin one coordinator handling mutations for the whole ring.

-- GOOD: same partition, atomic and isolated, cheaper than separate writes
BEGIN BATCH
  INSERT INTO chat.messages (conversation_id, sent_at, message_id, sender_id, body)
    VALUES (?, ?, ?, ?, ?);
  UPDATE chat.messages SET title = ? WHERE conversation_id = ?;
APPLY BATCH;

-- DEFENSIBLE: keeping denormalised copies in sync, accept the batchlog cost
BEGIN BATCH
  INSERT INTO shop.orders_by_id (order_id, customer_id, status, total) VALUES (?,?,?,?);
  INSERT INTO shop.orders_by_customer (customer_id, created_at, order_id, status) VALUES (?,?,?,?);
APPLY BATCH;

-- ANTI-PATTERN: 1000 unrelated partitions through one coordinator
BEGIN UNLOGGED BATCH
  INSERT INTO metrics.raw_1s (device_id, bucket, ts, value) VALUES (?,?,?,?);
  ... x1000
APPLY BATCH;
-- WARN  Batch of prepared statements for [metrics.raw_1s] is of size 71 KiB,
--       exceeding specified threshold of 50 KiB

# cassandra.yaml
batch_size_warn_threshold: 5KiB
batch_size_fail_threshold: 50KiB

Answer

A lightweight transaction is compare-and-set implemented with Paxos. When you write IF NOT EXISTS or IF column = value, the coordinator runs a consensus round among the replicas: a prepare and promise phase, a read of the current value, a propose and accept phase, and finally a commit. In the classic implementation that is four round trips instead of one, so an LWT is commonly an order of magnitude slower than a normal write, and it takes locks on the partition that serialise concurrent LWTs against the same key.

Cassandra 4.1 introduced a second Paxos implementation, enabled with paxos_variant: v2, which cuts the common case down and materially improves throughput, and it is worth naming if the interviewer goes deep. LWTs have their own consistency levels: SERIAL and LOCAL_SERIAL control the consensus round, and LOCAL_SERIAL is what you want in a multi-datacenter cluster because SERIAL will drag consensus across regions. The result set of an LWT always includes an [applied] boolean column, and when the condition failed it also returns the current values so the client can decide what to do without a second read.

Practical guidance for an interview: use LWTs for genuinely rare operations where correctness demands it, unique username registration is the canonical example, and never on a hot write path. Mixing LWT and non-LWT writes to the same partition is unsafe, because the plain write bypasses the Paxos round entirely and can silently clobber a value another client just conditionally set.

-- Unique registration: the canonical safe use
INSERT INTO auth.users_by_email (email, user_id, created_at)
VALUES ('ananya@example.com', ?, toTimestamp(now()))
IF NOT EXISTS;

 [applied] | email                | user_id  | created_at
-----------+----------------------+----------+------------
     False | ananya@example.com   | 8c1e...  | 2026-03-04

-- Conditional state transition, returns current value on failure
UPDATE shop.orders_by_id SET status = 'SHIPPED'
 WHERE order_id = ?
 IF status = 'PACKED';

cqlsh> SERIAL CONSISTENCY LOCAL_SERIAL;

// Java driver: read the applied flag, do not assume success
ResultSet rs = session.execute(lwt.bind(orderId));
if (!rs.wasApplied()) {
  String current = rs.one().getString("status");
  throw new IllegalStateException("expected PACKED, found " + current);
}

# cassandra.yaml (4.1+)
paxos_variant: v2

Answer

When a coordinator cannot deliver a mutation to a replica that is down, it stores a hint locally: the mutation plus the target node, written into the system hints storage. When gossip reports the node back up, the coordinator replays the hints to it. This heals short outages, a rolling restart, a five minute network blip, without any operator action, and it is why a LOCAL_QUORUM write can succeed while one replica is missing and the missing replica still ends up consistent minutes later.

The limits are what interviews probe. Hints are only kept for max_hint_window, three hours by default, and a node that stays down longer than that has data holes no hint will fill. Hints are stored on the coordinator, so if that coordinator itself dies before replaying, the hints die with it.

Hints are only created for mutations the coordinator actually attempted, so any write that happened while a node was already excluded from the ring, or any corruption or bit rot, is invisible to the mechanism. And hint replay is throttled (hinted_handoff_throttle) precisely so it does not overwhelm a node that just came back, which means a large backlog takes a long time to drain and can itself become an incident. So hinted handoff is an optimisation for brief unavailability. Anti-entropy repair, which compares Merkle trees of actual data between replicas, is the only mechanism that guarantees convergence, and it is the one bound by gc_grace_seconds.

# cassandra.yaml
hinted_handoff_enabled: true
max_hint_window: 3h
hinted_handoff_throttle: 1024KiB
max_hints_delivery_threads: 2
hints_directory: /var/lib/cassandra/hints

# Operational checks
$ nodetool statushandoff
Hinted handoff is running

$ du -sh /var/lib/cassandra/hints/
2.4G    /var/lib/cassandra/hints/

$ nodetool tpstats | grep -i hint
HintsDispatcher      1   0   48213   0   0

# Pause replay if a recovering node is being crushed by the backlog
$ nodetool pausehandoff
$ nodetool resumehandoff

# Drop hints for a node you are decommissioning anyway
$ nodetool truncatehints 10.0.1.26

Answer

Repair builds Merkle trees over token ranges on each replica, compares them, and streams the differing ranges so replicas converge. A full repair validates all data in the range every time, which is correct but expensive and re-reads data that was already verified. Incremental repair marks SSTables as repaired once they have been validated, so subsequent runs only examine unrepaired data, which is much cheaper on a steady-state cluster, at the cost of extra complexity: repaired and unrepaired data are kept in separate compaction pools, and a failed incremental repair historically left data in an awkward pending state.

The default mode has changed between major lines, so the professional habit is to pass -full or -inc explicitly rather than rely on it. Subrange repair narrows a run to a specific token range with -st and -et, which is how you make repair resumable and schedulable: instead of one enormous job you run thousands of small ones, each of which can fail and be retried without redoing everything. The -pr flag repairs only the node's primary ranges, which avoids repairing the same range RF times when you loop over the whole cluster, but it must then be run on every node.

On a 60-node cluster nobody hand-rolls this. The standard answer is Cassandra Reaper, which segments the token ring, schedules segments with configurable parallelism and intensity, backs off on load, resumes after failures, and reports coverage, so you can prove every table was repaired inside gc_grace_seconds. Managed offerings and the cass-operator ecosystem on Kubernetes wrap the same idea.

# Full repair of primary ranges only, one keyspace, run on every node in turn
$ nodetool repair -full -pr ledger

# Incremental, explicit
$ nodetool repair -inc ledger accounts

# Subrange: resumable unit of work
$ nodetool repair -full -st -9223372036854775808 -et -9223372036000000000 ledger

# Parallelism and progress
$ nodetool repair -full -j 2 -pr ledger
$ nodetool netstats | head -20
$ nodetool compactionstats -H | grep -i validation

# What most teams actually run: Reaper schedules segments cluster-wide
$ curl -X POST 'http://reaper:8080/repair_schedule?clusterName=prod\
&keyspace=ledger&owner=platform&segmentCountPerNode=64\
&repairParallelism=DATACENTER_AWARE&intensity=0.7&scheduleDaysBetween=7'

Answer

SizeTiered groups SSTables of similar size and merges a bucket once min_threshold of them (4 by default) exist. Write amplification is low, but a partition can be spread across many SSTables so reads suffer, and the largest compaction needs free disk roughly equal to the size of the tables being merged, which is why STCS clusters are sized for about 50 percent headroom. Leveled keeps non-overlapping SSTables in levels each around ten times larger than the one above, so any partition lives in at most one SSTable per level and SSTables-per-read is typically one or two.

You pay for that with roughly an order of magnitude more write amplification, so LCS suits read-heavy, update-heavy tables on SSD and is a bad fit for a firehose ingest table. TimeWindow buckets writes by wall-clock window, set with compaction_window_unit and compaction_window_size, and never compacts across windows, so an entire SSTable of expired TTL data can be dropped whole with no tombstone scanning. TWCS is the right answer for append-only time series, and its two failure modes are out-of-order writes and long-tail TTLs that keep an old window alive.

Cassandra 5.0 adds UnifiedCompactionStrategy, which collapses the choose-one-strategy decision into a single strategy tuned by scaling_parameters: negative or L-style values behave like Leveled, positive or T-style values behave like SizeTiered, and you can set a different value per level. UCS also shards the token range so compactions run in parallel and stay bounded by target_sstable_size, which removes the multi-hundred-gigabyte SSTable that used to make STCS clusters unrecoverable.

-- Read-heavy, frequently updated table
ALTER TABLE shop.orders_by_id WITH compaction = {
  'class': 'LeveledCompactionStrategy',
  'sstable_size_in_mb': 160
};

-- Append-only time series with TTL
ALTER TABLE metrics.raw_1s WITH compaction = {
  'class': 'TimeWindowCompactionStrategy',
  'compaction_window_unit': 'HOURS',
  'compaction_window_size': 6,
  'unchecked_tombstone_compaction': 'true'
};

-- Cassandra 5.0 Unified Compaction
ALTER TABLE app.order_events WITH compaction = {
  'class': 'UnifiedCompactionStrategy',
  'scaling_parameters': 'T4, T4, L10',
  'target_sstable_size': '1GiB',
  'base_shard_count': 4
};

$ nodetool compactionstats -H
$ nodetool tablestats app.order_events | grep -E 'SSTable count|Compacted partition maximum'
$ nodetool compact app order_events   -- major compaction, rarely a good idea

Answer

A legacy secondary index is a hidden local table on every replica, keyed by the indexed value and pointing back at primary keys held by that node. Because it is local, a query that does not also restrict the partition key must fan out to every node and gather, so its cost grows with cluster size. Cardinality then decides how badly it fails: index a near-unique column like email and you get millions of one-row index partitions, index a five-value status column on a billion rows and you get five gigantic index partitions.

The only genuinely safe shape is an index used alongside a partition key restriction, which keeps the work on one node. SASI added tokenised text and range support but never left experimental status and ships disabled behind sasi_indexes_enabled. Cassandra 5.0 introduced Storage Attached Indexes, built into the SSTable lifecycle: index components are written when a memtable flushes and are compacted together with the data, so there is no separate index table to keep in sync and no extra write path.

SAI supports numeric ranges, equality, text analysis for prefix and contains matching, and multiple indexes on one table combined with AND, while sharing one on-disk structure per SSTable so the space cost is far lower than several legacy indexes. The limit interviewers want you to state anyway: SAI is still a local index. Without a partition key restriction it remains scatter-gather across the ring, just much cheaper per node. Use SAI for moderately selective, genuinely ad-hoc predicates; keep a purpose-built table for anything on the hot path.

-- Cassandra 5.0 Storage Attached Index
CREATE INDEX orders_status_sai ON shop.orders_by_id (status) USING 'sai';

CREATE INDEX orders_total_sai ON shop.orders_by_id (total) USING 'sai';

-- Text analysis options for prefix / case-insensitive matching
CREATE INDEX cust_name_sai ON shop.customers (name) USING 'sai'
WITH OPTIONS = {
  'case_sensitive': 'false',
  'normalize': 'true',
  'ascii': 'true'
};

-- Multiple SAI predicates combine with AND
SELECT order_id, status, total FROM shop.orders_by_id
 WHERE status = 'FAILED' AND total > 5000;

-- Legacy 2i: only safe when the partition is also pinned
CREATE INDEX ON shop.orders_by_customer (status);
SELECT * FROM shop.orders_by_customer
 WHERE customer_id = ? AND status = 'FAILED';

# cassandra.yaml
sasi_indexes_enabled: false

Answer

A materialized view is a server-maintained denormalised table. Every base-table write forces the replica to read the existing base row so it can work out which view row is being replaced, write a tombstone for the old view row, and insert the new one, usually on a different replica set because the view's partition key differs from the base table's. So one client write becomes a local read plus coordinated remote writes, which is exactly the pattern Cassandra's write path is designed to avoid.

In production the consequences are a write amplification cliff on high-update tables, latency that appears in the base write and is hard to attribute, and, worst of all, silent divergence: if a view write is lost there is no built-in mechanism to detect or repair it, and running nodetool repair on the base table does not reconcile the view. That combination is why cassandra.yaml still ships materialized_views_enabled: false and the CREATE statement emits an experimental warning. If you do use them, the schema rules are strict: the view's primary key must include every column of the base primary key plus at most one additional column, and that extra column must be non-null for a row to appear. The alternatives teams actually run are writing both tables from the application and reconciling asynchronously, wrapping copies that share a partition key in a logged batch, deriving the second table from a Kafka or CDC stream so a replay repairs drift, or, for filter-shaped predicates in 5.0, a Storage Attached Index rather than a whole view.

-- The syntax works, but note the constraints
CREATE MATERIALIZED VIEW shop.orders_by_status AS
  SELECT order_id, customer_id, status, total, created_at
  FROM shop.orders_by_id
  WHERE status IS NOT NULL AND order_id IS NOT NULL
  PRIMARY KEY (status, order_id);

-- Base primary key columns must all appear; at most one extra column
-- and every key column needs an IS NOT NULL guard.

# cassandra.yaml default in 4.x and 5.x
materialized_views_enabled: false

-- Preferred: application-owned second table, reconciled from an event log
BEGIN BATCH
  INSERT INTO shop.orders_by_id (order_id, customer_id, status, total) VALUES (?,?,?,?);
  INSERT INTO shop.orders_by_status_day (status, day, created_at, order_id) VALUES (?,?,?,?);
APPLY BATCH;

$ nodetool viewbuildstatus shop orders_by_status

Answer

The theoretical ceiling is two billion cells per partition, and it is useless as guidance because a cluster becomes miserable long before that. The working budget most teams use is under roughly 100 MB and under roughly 100,000 rows per partition. Beyond that you get p99 read latency that looks random, garbage collection pressure because reading and compacting the partition pulls large structures into heap, compactions of that one partition running for hours, and a single node that is measurably hotter than its peers even though ownership looks balanced.

Detection is straightforward if you know where to look: the Partition Size column of nodetool tablehistograms at the 99th percentile, the compacted partition maximum bytes line in nodetool tablestats, the Writing large partition warnings in system.log, and the partition_size_warn_threshold and partition_tombstones_warn_threshold guardrails in recent versions. The fix is always the same shape, add an extra component to the partition key so it splits. For time-ordered data bucket by time, so the key becomes ((device_id, day), ts) or ((tenant_id, month), created_at).

For unbounded fan-out with no natural time dimension, bucket by a computed hash, ((user_id, bucket), item_id) where bucket is a hash modulo something like 16, and read all buckets in parallel. Migrating live means creating the new table, dual-writing from the application, backfilling with a Spark job that reads the old table by token range rather than paging a naive SELECT star, flipping reads behind a feature flag, and dropping the old table only after you have compared row counts.

# Find the offenders
$ nodetool tablehistograms chat messages
Percentile  SSTables  Read Latency  Partition Size  Cell Count
99%             4.00       1109.00       178851922      2402100   <-- 178 MB

$ grep -i 'Writing large partition' /var/log/cassandra/system.log | tail -5
WARN  Writing large partition chat/messages:9f1c... (214.882MiB) to sstable ...

-- Before: one partition per conversation, grows forever
PRIMARY KEY (conversation_id, sent_at, message_id)

-- After: bounded by month
CREATE TABLE chat.messages_v2 (
  conversation_id uuid,
  month           text,          -- '2026-08'
  sent_at         timestamp,
  message_id      timeuuid,
  body            text,
  PRIMARY KEY ((conversation_id, month), sent_at, message_id)
) WITH CLUSTERING ORDER BY (sent_at DESC, message_id DESC);

# cassandra.yaml guardrails
guardrails:
  partition_size_warn_threshold: 100MiB
  partition_tombstones_warn_threshold: 1000

Answer

In the DataStax drivers the retry policy decides what to do per error class: a ReadTimeout can be retried on the same host if enough replicas responded but the data was not returned, a WriteTimeout is retried only when the write type is BATCH_LOG, an Unavailable moves to the next host once, and a server error or aborted request tries a different node. Sitting on top of that is idempotence, and this is the part candidates get wrong. The driver will not retry a statement on another node after a write timeout unless you have declared it safe, via setIdempotent(true) on the statement or basic.request.default-idempotence in the configuration.

The default is false, which is the right default because counter updates, list append and prepend, and lightweight transactions cannot be replayed safely. A plain INSERT or UPDATE is idempotent, with one trap: if the statement itself calls now() or uuid() then a retry produces a different value, so it is not idempotent even though it looks like one. Speculative execution builds on the same flag.

Configure advanced.speculative-execution-policy with a constant delay and the driver fires a duplicate of a slow request at a second node, taking whichever answer arrives first, which cuts tail latency caused by one struggling replica. It only ever applies to idempotent statements, and it is dangerous on a saturated cluster because it multiplies load exactly when you can least afford it. The server-side sibling is the per-table speculative_retry option, where the coordinator itself sends a redundant read to an extra replica once the p99 threshold is crossed.

# application.conf, DataStax Java driver 4.x
datastax-java-driver {
  basic.request {
    timeout = 4 seconds          # must exceed server read_request_timeout
    consistency = LOCAL_QUORUM
    default-idempotence = false  # opt in per statement instead
  }
  advanced.retry-policy { class = DefaultRetryPolicy }
  advanced.speculative-execution-policy {
    class = ConstantSpeculativeExecutionPolicy
    max-executions = 2
    delay = 60 milliseconds
  }
}

// Safe to replay: no counters, no list append, no now()
BoundStatement safe = upsertOrder.bind(orderId, status).setIdempotent(true);

// NOT idempotent: a retry writes a different timeuuid
// INSERT INTO events (id, at) VALUES (?, now());

-- Server-side redundant read for the slowest replica
ALTER TABLE shop.orders_by_id WITH speculative_retry = '99p'
  AND additional_write_policy = '99p';

Answer

Compression happens per chunk, not per row or per SSTable. chunk_length_in_kb defines that unit, and recent versions default to 16 KiB where older clusters are often still on 64 KiB. To return a single 200-byte cell, Cassandra must fetch and decompress the entire chunk containing it, so read amplification for a point lookup is roughly chunk size divided by row size. On a table serving random point reads, dropping to 4 KiB or 8 KiB visibly cuts disk IO and latency; on a table that is scanned in ranges or archived, a larger chunk gives a better compression ratio and fewer metadata entries.

The algorithm choice is the second lever. LZ4Compressor is the default and is fast with a moderate ratio, ZstdCompressor gives substantially better ratios with a tunable compression_level and more CPU, and Deflate and Snappy are legacy options. Cold time-series data on Zstd routinely halves disk footprint, which on managed cloud storage is a real line item; a latency-critical table stays on LZ4.

The cost people forget is the compression offset map, an off-heap structure sized proportionally to data size divided by chunk length, so quartering the chunk length quadruples that memory. Check it before you change anything, it appears in nodetool tablestats as compression metadata off heap memory used. Two operational notes: a compression change only affects newly written SSTables, so run nodetool upgradesstables -a to rewrite existing ones, and compressing already-compressed blobs such as images just burns CPU, set enabled false there.

-- Point-read heavy table: smaller chunks
ALTER TABLE shop.orders_by_id WITH compression = {
  'class': 'LZ4Compressor',
  'chunk_length_in_kb': 4
};

-- Cold archive: better ratio, more CPU
ALTER TABLE metrics.raw_1s WITH compression = {
  'class': 'ZstdCompressor',
  'compression_level': 6,
  'chunk_length_in_kb': 64
};

-- Already-compressed payloads
ALTER TABLE media.thumbnails WITH compression = { 'enabled': false };

# Measure before and after
$ nodetool tablestats shop.orders_by_id | grep -E 'Compression ratio|off heap'
  SSTable Compression Ratio: 0.3184
  Compression metadata off heap memory used: 41216512

# Existing SSTables keep the old settings until rewritten
$ nodetool upgradesstables -a shop orders_by_id

# cassandra.yaml: cache for decompressed chunks
file_cache_size: 2GiB

Answer

Cassandra publishes everything through Dropwizard metrics exposed over JMX, and the standard production path is a JMX exporter or the MCAC agent scraping into Prometheus, with Grafana on top. Nobody runs a real cluster off nodetool alone, but nodetool is how you confirm a metric at three in the morning. The alerts that actually matter start with dropped messages: nodetool tpstats shows per-stage dropped counts, and any non-zero MUTATION or READ drop means the node accepted work it could not finish inside the timeout, which is the earliest honest signal of overload.

Next are the thread-pool pending counts, particularly MutationStage, ReadStage, CompactionExecutor and MemtableFlushWriter, because a pending queue that never drains is backpressure. Then read and write latency percentiles per table, not cluster-wide averages, since one bad table hides in an average. Pending compactions from nodetool compactionstats is the metric that predicts tomorrow's incident: a steadily rising backlog means compaction throughput is below ingest and SSTables-per-read is about to climb.

Add hints stored and hints delivered, GC pause duration and frequency, disk used against total (Cassandra needs headroom to compact at all), and SSTables per read plus tombstones scanned per read for latency-sensitive tables. Cassandra 4.0 also added structured diagnostic events and full query logging via nodetool enablefullquerylog, which writes a binary log you can replay with fqltool to reproduce a workload. In an interview, naming dropped mutations and pending compactions as your first two dashboards immediately marks you as someone who has operated the thing.

$ nodetool tpstats
Pool Name                Active Pending Completed Blocked All time blocked
MutationStage                 8      412  91238411       0                0
ReadStage                     4       11  40021882       0                0
CompactionExecutor            2       37    182773       0                0

Message type           Dropped
MUTATION                  1842   <-- overload, not a network problem
READ                        11

$ nodetool compactionstats -H
pending tasks: 214            <-- backlog is growing, act now

$ nodetool proxyhistograms
$ nodetool gcstats
$ nodetool getconcurrentcompactors

# Full query logging (4.0+) for reproducing a workload
$ nodetool enablefullquerylog --path /var/lib/cassandra/fql --roll-cycle HOURLY
$ fqltool dump /var/lib/cassandra/fql | head -40
$ nodetool disablefullquerylog

Answer

Mocking the session gives you almost nothing, because the bugs that matter are CQL restrictions, tombstone behaviour and consistency semantics that only a real server enforces. The default in 2026 is Testcontainers with the official cassandra image: the test spins a container, waits for the CQL port, applies the schema, and hands the suite a CqlSession pointed at the mapped port. It is slow enough that you run one container per test class or module rather than per test, and you clean between tests with TRUNCATE rather than DROP KEYSPACE, which is much faster.

For local multi-node behaviour, ccm (Cassandra Cluster Manager) still creates a real multi-node cluster on a loopback range so you can kill a node and verify your retry and consistency handling actually works. Things worth asserting that unit tests usually miss: that a query rejected without ALLOW FILTERING stays rejected, that your writes bind unset rather than null, that a LWT path checks wasApplied, and that your code behaves when the driver throws WriteTimeoutException. Schema management is the other half.

Never let the application create tables at startup, because concurrent DDL from several pods causes schema disagreement, which you can see with nodetool describecluster. Use a versioned migration tool run by exactly one process, such as cassandra-migration or Liquibase's Cassandra support, keep each migration additive, and remember that ALTER TABLE DROP leaves the data until compaction and that adding a column back with a different type is not allowed within the drop retention window.

// JUnit 5 + Testcontainers
@Testcontainers
class OrderRepositoryTest {

  @Container
  static final CassandraContainer<?> CASSANDRA =
      new CassandraContainer<>("cassandra:5.0").withInitScript("schema.cql");

  static CqlSession session;

  @BeforeAll
  static void connect() {
    session = CqlSession.builder()
        .addContactPoint(CASSANDRA.getContactPoint())
        .withLocalDatacenter(CASSANDRA.getLocalDatacenter())
        .withKeyspace("shop")
        .build();
  }

  @AfterEach
  void clean() { session.execute("TRUNCATE orders_by_id"); }

  @Test
  void rejectsUnboundedScan() {
    assertThrows(InvalidQueryException.class, () ->
        session.execute("SELECT * FROM orders_by_customer WHERE status='FAILED'"));
  }
}

# Local multi-node cluster for failure testing
$ ccm create test -v 5.0 -n 3 -s && ccm node2 stop && ccm node1 cqlsh

Answer

nodetool snapshot flushes memtables and creates hard links to the current SSTable files in a snapshots directory under each table. It is instant and consumes no extra space at the moment you take it, because it is just links, but the space is reclaimed only when the original SSTables are compacted away, at which point the snapshot keeps them alive on disk. That is the classic surprise: a forgotten snapshot silently fills a disk, and nodetool listsnapshots plus nodetool clearsnapshot are how you find and remove it.

A snapshot is also local to one node and useless on its own, so the real backup step is shipping those files off the box to object storage, together with the schema, which you capture separately with cqlsh -e DESCRIBE SCHEMA. Incremental backups, enabled with incremental_backups in cassandra.yaml, hard-link every newly flushed SSTable into a backups directory so you can ship deltas between full snapshots, but Cassandra never cleans that directory for you. Snapshots across a cluster are not taken at the same instant, so the restore point is fuzzy by design, which is fine for Cassandra's eventually consistent model but must be stated.

Restore has two shapes. Same-topology restore means stopping the node, copying the SSTables back into the table directory, and running nodetool refresh. Different-topology restore means streaming the files in with sstableloader, which reads the SSTables and distributes rows to the correct owners. Most teams wrap all of this in Medusa, which handles scheduling, object-storage upload and cluster-wide restore orchestration.

# Snapshot one keyspace on every node, tagged
$ nodetool snapshot -t pre-upgrade-2026-08-12 ledger
$ nodetool listsnapshots
$ nodetool clearsnapshot -t pre-upgrade-2026-08-12

# Capture the schema separately, it is not in the snapshot
$ cqlsh 10.0.1.24 -e 'DESCRIBE SCHEMA' > schema-2026-08-12.cql

# cassandra.yaml
incremental_backups: true
snapshot_before_compaction: false

# Restore, same topology
$ nodetool drain && systemctl stop cassandra
$ cp /var/lib/cassandra/data/ledger/accounts-*/snapshots/pre-upgrade-2026-08-12/* \
     /var/lib/cassandra/data/ledger/accounts-*/
$ systemctl start cassandra && nodetool refresh ledger accounts

# Restore into a differently shaped cluster
$ sstableloader -d 10.0.1.24,10.0.1.25 \
    /backup/ledger/accounts-7f3a9c21

Answer

The new node gets the same cluster_name, the same snitch and datacenter and rack labels, seeds pointing at existing nodes (never at itself), and auto_bootstrap left at the default true. On start it joins gossip, is assigned token ranges, and enters the UJ state while replicas stream it the data for those ranges. It does not serve reads for a range until streaming completes and it flips to UN.

You add one node at a time and wait, because concurrent bootstraps can produce overlapping range calculations, and recent versions refuse a second bootstrap while one is in progress unless you deliberately override it. Streaming speed is governed by stream_throughput_outbound and by disk and network, and nodetool netstats is how you watch it. The step people skip is nodetool cleanup on the pre-existing nodes.

When the new node takes ownership of ranges, the old owners still hold that data on disk; it is no longer theirs, it is not served, but it consumes space and it is compacted forever. Cleanup rewrites the SSTables dropping rows the node no longer owns. Skip it and disk usage on the old nodes never falls, which is why teams sometimes add capacity and see no free space.

Cleanup is IO heavy, so run it one node at a time, off peak, and never before the bootstrap has fully finished. The removal side of this is the mirror image: nodetool decommission on a live node streams its data away before it leaves, whereas nodetool removenode is for a node that is already dead and makes the surviving replicas stream to cover it.

# cassandra.yaml on the joining node
cluster_name: 'prod-blr'
auto_bootstrap: true
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.0.1.24,10.0.2.31"
endpoint_snitch: GossipingPropertyFileSnitch
stream_throughput_outbound: 200Mib/s

# Watch it join
$ nodetool status | grep UJ
UJ  10.0.1.27   42.11 GiB  16   ?   d41c8f2a   rack-a
$ nodetool netstats | grep -E 'Receiving|files'

# Only after it shows UN: reclaim space on the OLD nodes, one at a time
$ nodetool cleanup -j 2 ledger
$ nodetool tablestats ledger | grep 'Space used (live)'

# Removing capacity
$ nodetool decommission          # node is alive
$ nodetool removenode c92e7f08   # node is already dead

Answer

On a read above consistency level ONE the coordinator asks one replica for full data and the others for a digest of the same range. If the digests disagree it fetches full data from the mismatched replicas, merges by cell timestamp, and, before returning to the client, writes the merged result back to the replicas that were stale. That write-back is blocking read repair: the client does not get an answer until the repair mutation is acknowledged, which is what preserves monotonic quorum reads.

If the repair write itself cannot reach enough replicas, the coordinator raises a ReadFailure rather than returning a value it cannot guarantee, which surprises people who expect reads to degrade quietly. The 3.x era also had probabilistic background read repair, configured per table with read_repair_chance and dclocal_read_repair_chance, which repaired a fraction of reads asynchronously even at consistency ONE. Those options were removed in 4.0: they had a measurable cost on every read, they gave no guarantee anyone could rely on, and they encouraged teams to treat them as a substitute for scheduled repair.

What 4.0 added instead is a per-table read_repair option with values BLOCKING (the default) and NONE, where NONE turns off the write-back entirely for workloads that cannot tolerate the extra latency and accept slower convergence. The point to land in an interview is that read repair only touches data someone actually reads, so cold data never converges through it. Scheduled anti-entropy repair inside gc_grace_seconds remains mandatory regardless.

-- 4.0+ per-table option, replacing read_repair_chance
ALTER TABLE app.order_events WITH read_repair = 'BLOCKING';   -- default
ALTER TABLE metrics.raw_1s   WITH read_repair = 'NONE';       -- latency over convergence

-- These no longer exist in 4.0+ and are rejected
-- ALTER TABLE app.order_events WITH read_repair_chance = 0.1;
-- ALTER TABLE app.order_events WITH dclocal_read_repair_chance = 0.1;

cqlsh> TRACING ON;
cqlsh> CONSISTENCY LOCAL_QUORUM;
cqlsh> SELECT * FROM app.order_events WHERE tenant_id=? AND day='2026-08-12';
-- trace: Digest mismatch: org.apache.cassandra.service.DigestMismatchException
--        Reading data from /10.0.1.25
--        Repair 1 rows to /10.0.1.26

$ nodetool tablestats app.order_events | grep -i 'repair'

Answer

Out of the box Cassandra runs with AllowAllAuthenticator, meaning any client that reaches port 9042 is a superuser, and clusters exposed that way are still found on the public internet. The first change is authenticator: PasswordAuthenticator plus authorizer: CassandraAuthorizer in cassandra.yaml, then log in as the default cassandra superuser, create your own superuser, and disable or change the default one, because the built-in cassandra role reads at QUORUM and will fail during any partial outage. Roles are the unit of both authentication and permission: CREATE ROLE with LOGIN and PASSWORD, GRANT SELECT or MODIFY on a keyspace or table, and roles can be granted to other roles for grouping.

Critically, the system_auth keyspace holds all of this and is created with a low replication factor, so you must ALTER it to NetworkTopologyStrategy with RF matching your data keyspaces and repair it, otherwise losing one node locks people out. Encryption has two independent settings. server_encryption_options covers internode traffic, with internode_encryption set to all or dc so that cross-datacenter links over the public internet are protected. client_encryption_options covers driver connections, and the driver side needs a matching truststore, optionally with require_client_auth for mutual TLS. Beyond that, 4.0 added audit logging (audit_logging_options) writing who ran which statement, and a role can be restricted with network authorization so it only connects to specific datacenters. Cassandra has no native at-rest encryption in the Apache distribution, so you use encrypted volumes such as LUKS or EBS encryption underneath.

# cassandra.yaml
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
role_manager: CassandraRoleManager
roles_validity: 2000ms

server_encryption_options:
  internode_encryption: all
  keystore: /etc/cassandra/conf/server-keystore.jks
  truststore: /etc/cassandra/conf/server-truststore.jks

client_encryption_options:
  enabled: true
  optional: false
  require_client_auth: true

audit_logging_options:
  enabled: true
  included_categories: DDL,DCL,AUTH

-- Fix system_auth replication FIRST, then repair it
ALTER KEYSPACE system_auth WITH replication =
  {'class':'NetworkTopologyStrategy','blr-1':3,'mum-1':3};

CREATE ROLE app_rw WITH LOGIN = true AND PASSWORD = '***';
GRANT SELECT, MODIFY ON KEYSPACE shop TO app_rw;
ALTER ROLE app_rw WITH ACCESS TO DATACENTERS {'blr-1'};

Answer

First rule: treat GC as a symptom, not the disease. Cassandra allocates heap in proportion to what it is asked to read, so the usual root cause is a workload problem, a wide partition, a huge page size, a collection with thousands of elements, an unbounded IN clause, or a query returning a million tombstones. Check nodetool tablehistograms for partition size at p99, the log for large partition warnings, and the driver's configured page size before you touch a single JVM flag.

Once the workload is clean, sizing is conventional. Heap of 16 to 31 GB, staying strictly under the compressed-oops threshold of about 32 GB because crossing it silently makes every object reference bigger. Leave the rest of RAM for the page cache, which is what serves your SSTable reads, so a 64 GB machine typically runs a 24 to 31 GB heap.

Move memtables off heap with memtable_allocation_type: offheap_objects so flushing does not churn the collector. On collectors, G1GC is the default in 4.x and 5.x and is the right choice for heaps above about 16 GB; set MaxGCPauseMillis around 300 and leave the young generation unfixed, because pinning Xmn with G1 is a common mistake carried over from CMS tuning. ZGC is available on modern JDKs and gives sub-millisecond pauses at some throughput cost, and it is worth trialling on large-heap latency-critical clusters, but G1 remains the well-trodden default. Confirm with nodetool gcstats and the GC log rather than by feel.

# jvm11-server.options / jvm17-server.options
-Xms24G
-Xmx24G                      # equal, and under the ~32G compressed-oops cliff
-XX:+UseG1GC
-XX:MaxGCPauseMillis=300
-XX:InitiatingHeapOccupancyPercent=70
-XX:+AlwaysPreTouch
-XX:+PerfDisableSharedMem
-Xlog:gc*,gc+heap=info:file=/var/log/cassandra/gc.log:time,uptime:filecount=10,filesize=64m

# cassandra.yaml
memtable_allocation_type: offheap_objects
concurrent_reads: 32
concurrent_writes: 64
file_cache_size: 2GiB

$ nodetool gcstats
Interval(ms) Max GC Elapsed(ms) Total GC Elapsed(ms) GC Reclaimed(MB) Collections
     600214                812                21447      184229498412         2211

# Prove it is the workload, not the collector
$ nodetool tablehistograms app order_events
$ grep -c 'Writing large partition' /var/log/cassandra/system.log

Answer

The supported path is a replacement bootstrap, not a fresh join followed by a repair. You build a new machine with the same cassandra.yaml, the same cluster name, snitch, datacenter and rack as the dead node, an empty data directory, and one extra JVM option: cassandra.replace_address_first_boot set to the dead node's IP address. On start the new node takes over the dead node's token ranges rather than being assigned new ones, and streams the data for those ranges from the surviving replicas.

Because it inherits the exact tokens, no other node's ownership changes, which means no cleanup is needed anywhere and the ring stays balanced. Two preconditions matter. The dead node must actually be down and marked DN in gossip; if it comes back mid-replacement you get two nodes claiming the same tokens.

And the replacement must complete within gc_grace_seconds of the failure, because the surviving replicas hold tombstones that will become droppable, and streaming after that window can resurrect deleted rows, which is why you run a repair of the affected keyspaces once the node is UN. Use replace_address_first_boot rather than the older replace_address, since the newer flag is ignored on subsequent restarts and cannot accidentally re-trigger a replacement. If instead you want to permanently shrink the cluster, the dead-node command is nodetool removenode with the host ID, which makes surviving replicas stream to restore replica counts. nodetool assassinate is the last resort that just evicts the node from gossip without moving any data, and it leaves you under-replicated until you repair.

# 1. Confirm the node really is down and grab its Host ID
$ nodetool status ledger
DN  10.0.1.26   402.77 GiB  16   33.4%   c92e7f08-...   rack-c

# 2. New machine: identical cassandra.yaml, EMPTY data/commitlog/hints dirs,
#    same dc/rack in cassandra-rackdc.properties, itself NOT in the seed list.

# 3. cassandra-env.sh (or jvm-server.options)
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot=10.0.1.26"

$ systemctl start cassandra
$ nodetool netstats | grep -E 'Receiving|Mode'
Mode: JOINING

# 4. When it reports UN, repair the ranges it just inherited
$ nodetool status ledger | grep 10.0.1
$ nodetool repair -full ledger

# Alternative: permanently remove instead of replace
$ nodetool removenode c92e7f08-...
$ nodetool removenode status
$ nodetool assassinate 10.0.1.26     # last resort, moves no data

Answer

Order matters more than any single command here, and getting it wrong either drops writes or floods production with streaming. Prerequisites first: the existing cluster must already be on NetworkTopologyStrategy with GossipingPropertyFileSnitch, and applications must already be using LOCAL_QUORUM or LOCAL_ONE rather than QUORUM, otherwise the moment you raise total replication factor the quorum arithmetic changes underneath live traffic. Step one, bring up the new nodes with the new datacenter name in cassandra-rackdc.properties and auto_bootstrap set to false.

This is the counterintuitive bit: you do not want them streaming during join, you want them empty and in the ring. Step two, ALTER KEYSPACE to add the new datacenter to the replication map, for every user keyspace and also for system_auth, system_distributed and system_traces. From this instant, new writes are replicated to the new DC automatically, so nothing written from now on is missing.

Step three, run nodetool rebuild on each new node, naming the source datacenter, which streams the historical data across. Rebuild is resumable and throttled by stream_throughput_outbound; run a few nodes at a time so you do not saturate the inter-region link. Step four, verify with nodetool status per keyspace that effective ownership looks right, then run a full repair. Only then do you point application traffic at the new DC by changing withLocalDatacenter, and only after you have confirmed reads there return complete data.

# New nodes only: cassandra-rackdc.properties + cassandra.yaml
dc=mum-1
rack=rack-a
# cassandra.yaml on the new nodes
auto_bootstrap: false
seeds: "10.0.1.24,10.0.2.31"   # include seeds from BOTH datacenters

# 1. Start them, confirm they appear under the new DC with ~0 Load
$ nodetool status

-- 2. Extend replication (every user keyspace AND the system ones)
ALTER KEYSPACE ledger WITH replication =
  {'class':'NetworkTopologyStrategy','blr-1':3,'mum-1':3};
ALTER KEYSPACE system_auth WITH replication =
  {'class':'NetworkTopologyStrategy','blr-1':3,'mum-1':3};
ALTER KEYSPACE system_distributed WITH replication =
  {'class':'NetworkTopologyStrategy','blr-1':3,'mum-1':3};

# 3. Stream history into each new node, a few at a time
$ nodetool rebuild -- blr-1
$ nodetool netstats | grep -c 'Receiving'

# 4. Verify, then repair, then move traffic
$ nodetool status ledger
$ nodetool repair -full ledger
// driver: .withLocalDatacenter("mum-1")

Answer

Cassandra 5.0 added a native vector type, vector<float, n>, where n is a fixed dimension, and a Storage Attached Index that builds an approximate nearest neighbour structure over that column using a JVector-based graph index. You then query with the ANN OF clause and an ORDER BY on the vector column, and CQL exposes similarity_cosine, similarity_dot_product and similarity_euclidean so you can return the score alongside the row. Because the index is a SAI, it is written when memtables flush and is compacted with the data, and it can be combined with other SAI predicates so you can filter by tenant or category and then rank by vector distance in one statement.

What makes this genuinely useful is not raw ANN performance, it is colocation: the embedding sits in the same row as the operational data, replicated the same way, in the same multi-datacenter cluster you already run, so there is no second system to keep in sync. The limits are worth stating plainly in an interview. Recall and latency depend on the similarity_function and index build options, and there is no free lunch versus a purpose-built engine tuned for that one job.

ANN queries without a partition key restriction still fan out across the ring and then merge per-node top-k results, so cost grows with cluster size. Dimensionality is fixed at table creation. And re-embedding a corpus means rewriting every row, which on a large table is a real migration, not an ALTER.

CREATE TABLE kb.chunks (
  doc_id     uuid,
  chunk_id   int,
  tenant     text,
  body       text,
  embedding  vector<float, 1536>,
  PRIMARY KEY (doc_id, chunk_id)
);

CREATE INDEX chunks_embedding_ann ON kb.chunks (embedding) USING 'sai'
WITH OPTIONS = { 'similarity_function': 'cosine' };

CREATE INDEX chunks_tenant_sai ON kb.chunks (tenant) USING 'sai';

-- Filtered approximate nearest neighbour search
SELECT doc_id, chunk_id, body,
       similarity_cosine(embedding, [0.021, -0.118, 0.334 /* ... */]) AS score
  FROM kb.chunks
 WHERE tenant = 'goodspace'
 ORDER BY embedding ANN OF [0.021, -0.118, 0.334 /* ... */]
 LIMIT 10;

-- Dimension is fixed at DDL time; changing the model means a table migration

Answer

Every cell carries a microsecond timestamp, and on read the merge keeps the cell with the highest one. In modern versions the coordinator generates that timestamp by default, so all mutations in a request share a consistent clock source, but the clocks of different coordinators are only as aligned as your NTP setup. Drift of a few hundred milliseconds between nodes is enough that a later write, routed through a lagging coordinator, loses to an earlier one and simply disappears with no error anywhere.

That is the failure mode to describe: silent data loss that no exception, log line or consistency level protects you from, which is why chrony or an equivalent with tight sync is a hard requirement, and why interviewers treat clock discipline as a Cassandra topic rather than an infrastructure aside. Two more subtleties. Ties are broken by comparing the cell values byte-wise, which is deterministic but arbitrary, so two writes in the same microsecond can resolve to either.

And a delete always beats a write at the same timestamp, because tombstones win ties by design. Setting USING TIMESTAMP explicitly is the right call in a few situations: replaying an event log or backfilling from another system where the true event time should decide precedence, correcting data where you must guarantee your fix outranks whatever is there, and idempotent re-ingestion where the same event replayed must not overwrite something newer. The trap is writing a timestamp far in the future, perhaps to force an overwrite, because nothing can then update that cell until wall-clock time catches up, and the only remedy is a delete with an even larger timestamp.

-- Inspect what is actually stored
SELECT status, WRITETIME(status), TTL(status)
  FROM shop.orders_by_id WHERE order_id = ?;

-- Backfill from an event log: let the EVENT time decide precedence
INSERT INTO shop.orders_by_id (order_id, status, total)
VALUES (?, 'SHIPPED', 4999.00)
USING TIMESTAMP 1786512345678901;   -- microseconds since epoch

-- A delete beats a write at the same timestamp
DELETE FROM shop.orders_by_id USING TIMESTAMP 1786512345678901
 WHERE order_id = ?;

-- DANGEROUS: a far-future timestamp freezes the cell
-- USING TIMESTAMP 99999999999999999

# Guardrail against exactly that mistake (recent versions)
guardrails:
  maximum_timestamp_warn_threshold: 1d
  maximum_timestamp_fail_threshold: 3d

# Clock discipline is part of the database's correctness model
$ chronyc tracking | grep -E 'System time|Last offset'

Answer

Start by reading the exception properly. WriteTimeoutException carries the consistency level, the number of replicas that acknowledged and the number required, and the write type. Received 1 of 2 at LOCAL_QUORUM with writeType SIMPLE tells you one replica is slow or unreachable, not that the cluster is down, and it explicitly does not mean the mutation failed, since the write may have landed on the replicas that did respond.

Next, find whether it is one node or all of them, because the whole shape of the investigation depends on that. nodetool status for a DN, then per-node nodetool tpstats looking at MutationStage pending and, most importantly, the dropped MUTATION counter, which is Cassandra telling you it accepted work and threw it away after write_request_timeout expired. Dropped mutations with idle CPU almost always point at IO or at a stall rather than compute. Check nodetool compactionstats for a pending backlog, disk utilisation and await on the data volume, and the GC log for pauses long enough to blow the timeout, which is the single most common cause of a healthy-looking node timing out.

Then check hints piling up on coordinators, since a growing hints directory names the slow replica for you. Widen the model view too: a hot partition means every write for one key funnels to the same three replicas regardless of cluster size, and that shows up as one node with high MutationStage pending while its peers idle. Immediate mitigations are throttling the offending workload and raising concurrent_writes or compaction throughput; the fix is usually the partition key.

// What the driver actually reports
// com.datastax.oss.driver.api.core.servererrors.WriteTimeoutException:
//   Cassandra timeout during SIMPLE write query at consistency LOCAL_QUORUM
//   (2 replica were required but only 1 acknowledged the write)

$ nodetool status ledger | grep -v '^UN'
$ for h in 10.0.1.24 10.0.1.25 10.0.1.26; do
    echo "== $h"; nodetool -h $h tpstats | grep -E 'MutationStage|^MUTATION';
  done
== 10.0.1.26
MutationStage        32   9214   88123441   0   0     <-- pending queue not draining
MUTATION           4812                              <-- dropped

$ nodetool -h 10.0.1.26 compactionstats -H
pending tasks: 341
$ iostat -x 2 3 | awk '/nvme0n1/ {print $NF, $10}'
$ grep -E 'Pause|Total time for which' /var/log/cassandra/gc.log | tail
$ du -sh /var/lib/cassandra/hints/

# cassandra.yaml knobs involved
write_request_timeout: 2000ms
concurrent_writes: 64
compaction_throughput: 64MiB/s

Answer

Two related pieces of work, both aimed at the memory and IO cost of finding a key. Historically a memtable was a concurrent skip list of partitions, which carries substantial per-entry object overhead on heap and means a partition key is stored in full for every entry. Cassandra 5.0 adds a trie-backed memtable, selectable per table through the memtable configuration in cassandra.yaml and a memtable option on the table.

A trie shares common key prefixes rather than repeating them, so for the long structured keys real schemas use, tenant identifiers, path-like keys, prefixed identifiers, the memory footprint drops noticeably and more data fits before a flush is forced, which in turn means fewer, larger SSTables and less compaction work downstream. The on-disk half is the big-trie-indexed format, BTI, an alternative to the long-standing BIG format, configured with sstable_format or the sstable option on a table. BIG uses a partition index with an index summary sampled into memory, and the summary size scales with the number of partitions, so very large tables spend a lot of heap just to locate keys.

BTI replaces that with byte-ordered tries for both the partition index and the row index, giving lookups with far smaller and mostly memory-mapped structures and no sampling trade-off to tune. Both are opt-in in 5.0, which is the practical point for an interview: they are per-table settings that only apply to newly written SSTables, so adoption is a rolling change plus nodetool upgradesstables, not a big-bang migration.

# cassandra.yaml: declare a trie memtable configuration
memtable:
  configurations:
    trie:
      class_name: TrieMemtable
    default:
      inherits: trie

# Default on-disk format for newly written SSTables
sstable:
  selected_format: bti

-- Or opt in per table
ALTER TABLE app.order_events WITH memtable = 'trie';
ALTER TABLE app.order_events WITH sstable = { 'format': 'bti' };

# Existing SSTables keep their old format until rewritten
$ nodetool upgradesstables -a app order_events
$ ls /var/lib/cassandra/data/app/order_events-*/ | head -4
da-1-bti-Data.db
da-1-bti-Partitions.db
da-1-bti-Rows.db

$ nodetool tablestats app.order_events | grep -i 'off heap'

Answer

Today the honest position is that Cassandra gives you three things and no more: atomicity within a single partition, atomicity without isolation across partitions via a logged batch, and linearizable compare-and-set on one partition via Paxos-based lightweight transactions. There is no general multi-partition transaction, and any candidate who claims otherwise is guessing. The project's answer to this is Accord, proposed as CEP-15, a leaderless consensus protocol designed to give general-purpose transactions across partitions and across datacenters in one round trip in the common case, without the coordinator election that makes classic Paxos and Raft expensive over wide-area links.

It has been in development on the trunk branch alongside the 5.x line, exposed through new CQL transaction syntax, and it is the headline feature the community associates with the next major step rather than something you should assume is production-ready in a cluster you are running now. So the practical advice for a team today has three branches. If the operation genuinely fits inside one partition, restructure the model so it does, and use a single-partition batch or an LWT.

If it spans partitions but tolerates eventual convergence, write the copies from the application or derive them from an event log and build a reconciler, which is what most large Cassandra deployments actually do. If you need real cross-entity ACID with isolation, put that slice of the domain in a relational database and keep Cassandra for the high-volume, high-availability part. Choosing the right store per workload is a stronger answer than bending one store to everything.

-- What you can rely on TODAY

-- 1. Atomic and isolated: everything in one partition
BEGIN BATCH
  INSERT INTO chat.messages (conversation_id, sent_at, message_id, body) VALUES (?,?,?,?);
  UPDATE chat.messages SET title = ? WHERE conversation_id = ?;
APPLY BATCH;

-- 2. Atomic, NOT isolated: logged batch across partitions
BEGIN BATCH
  INSERT INTO shop.orders_by_id (order_id, status) VALUES (?, 'PAID');
  INSERT INTO shop.orders_by_customer (customer_id, created_at, order_id) VALUES (?,?,?);
APPLY BATCH;

-- 3. Linearizable CAS on ONE partition
UPDATE shop.orders_by_id SET status='SHIPPED' WHERE order_id=? IF status='PACKED';

-- What you must NOT assume exists
-- BEGIN TRANSACTION ... COMMIT across arbitrary partitions with isolation

# The CAS path, tuned
paxos_variant: v2

Answer

Cassandra is wrong whenever the access pattern is not known in advance. If analysts need to slice data by arbitrary columns, if the product roadmap keeps inventing new filters, or if the workload is dominated by joins and aggregations, you are fighting the model, and a relational database or a columnar warehouse will beat it on both effort and cost. It is also wrong for small data: a three-node minimum with repair, compaction and monitoring is a real operational commitment, and a hundred gigabytes on Postgres with a read replica is simpler in every dimension.

It is wrong for delete-heavy queue-like workloads, where tombstones make the read path progressively worse. And it is wrong when you need cross-entity transactions with isolation. Against the alternatives, the arguments are specific.

ScyllaDB is CQL-compatible and rewritten in C++ with a shard-per-core, thread-per-core design, so it typically delivers more throughput per node and much better tail latency, at the cost of a smaller ecosystem and a single primary vendor; the honest comparison is per-node efficiency versus community breadth and licensing comfort. DynamoDB removes operations entirely and is excellent if you are all-in on AWS, but it prices per request and per stored gigabyte, its cross-region story is a managed feature you cannot tune, and multi-cloud or on-premise is off the table. MongoDB has a far friendlier document model and ad-hoc query story, but its replication is leader-based, so write availability during a partition is not the same guarantee Cassandra's masterless ring provides.

The question a design review really answers is whether you need always-on multi-region writes. If yes, Cassandra earns its complexity. If no, it usually does not.