MLOps Interview Questions and Answers

Answer

MLOps is the discipline of taking machine learning models from experimentation to reliable production operation: packaging, deploying, monitoring, retraining, and governing them. It borrows heavily from DevOps, version control, CI/CD, containers, infrastructure as code, but ML systems break the core DevOps assumption that behaviour is fully determined by code. An ML system's behaviour is determined by code plus data plus the trained model artifact, and all three change independently.

That creates problems DevOps never had to solve: a deployment can pass every test and still degrade silently because the incoming data distribution shifted; two builds of the same code can produce different models because training is stochastic; and rollback means restoring a model artifact and possibly its feature pipeline, not just redeploying a container. MLOps therefore adds several new pillars on top of the DevOps foundation: experiment tracking so every model is traceable to its code, data, and hyperparameters; data and model versioning; automated validation of data quality before training; model-specific deployment patterns like shadow and canary releases evaluated on prediction quality, not just error rates; and continuous monitoring for drift with retraining loops. In interviews at Indian services firms like Fractal or Quantiphi, a crisp way to frame it is: DevOps ships and operates code, MLOps ships and operates code, data, and models as one versioned unit, and most of the extra machinery exists because data changes even when code does not.

Answer

Experiment tracking is the practice of recording every training run, its code version, hyperparameters, dataset reference, metrics, and output artifacts, in a queryable central store. Without it, a data science team accumulates hundreds of notebook runs where nobody can answer the two questions that matter in production: which exact run produced the model currently serving traffic, and can we reproduce it? MLflow and Weights & Biases dominate because they solve this with a few lines of instrumentation.

MLflow is open source, self-hostable, and ships a tracking server, artifact store, and model registry in one package, which makes it the default at cost-conscious Indian enterprises and services firms. Weights & Biases is a managed product with stronger visualisation, sweep-based hyperparameter search, and collaboration features, common at funded startups and research-heavy teams. In an interview, go beyond naming tools: explain what a complete run record contains.

That means the git commit hash, the data version or snapshot ID, the full parameter set, environment details like library versions and GPU type, evaluation metrics on a fixed validation split, and the serialised model as an artifact. Also mention the organisational benefit: tracking turns model selection from a conversation into a query, and it is the foundation the model registry builds on, since you can only register and promote a model confidently when its lineage is recorded.

import mlflow

mlflow.set_tracking_uri('http://mlflow.internal:5000')
mlflow.set_experiment('churn-model')

with mlflow.start_run(run_name='xgb-baseline') as run:
    params = {'max_depth': 6, 'eta': 0.1, 'num_rounds': 400}
    mlflow.log_params(params)
    mlflow.set_tags({
        'git_sha': get_git_sha(),
        'data_version': 'dvc:transactions@2026-08-01',
    })

    model = train(train_df, params)
    auc = evaluate(model, val_df)

    mlflow.log_metric('val_auc', auc)
    # name= is the MLflow 3.x signature; MLflow 2.x used artifact_path=
    mlflow.xgboost.log_model(model, name='model')
    mlflow.log_artifact('configs/features.yaml')
    # run.info.run_id ties this model to code + data forever

Answer

A model registry is the system of record for models that matter, the ones heading to or already in production, as opposed to the thousands of throwaway experiment runs. Where experiment tracking answers what did we try, the registry answers what is deployed, what is about to be, and who approved it. A registry entry is a named model, say fraud-scorer, with immutable numbered versions.

Each version points back to the training run that produced it, which gives you lineage to code, data, and parameters. On top of versions, registries add lifecycle state: modern MLflow uses aliases like champion and challenger (the older staging and production stages are deprecated), SageMaker Model Registry uses approval status, and Vertex AI Model Registry uses versions with aliases as well. The registry is also where governance hooks live: who promoted a version, when, with what evaluation evidence, and approval workflows requiring a human sign-off before an alias moves.

What should live in it: the serialised model artifact or a pointer to it in object storage, the input and output signature or schema, the evaluation metrics that justified promotion, dependency requirements needed to load it, and tags linking to the data snapshot. What should not live in it: raw training data, credentials, or ad-hoc experiment runs that nobody intends to ship. In production, serving infrastructure should always resolve models through the registry by alias, never by a hardcoded file path, because that single indirection is what makes rollback a one-line operation.

Answer

Data versioning applies the guarantees Git gives code to datasets: any historical state can be named, retrieved, and diffed, so a training run can declare exactly which data produced it. Git alone cannot do this because repositories choke on multi-gigabyte files and binary formats do not diff meaningfully. DVC (Data Version Control) solves it with a pointer-file pattern.

When you run dvc add on a dataset, DVC computes a content hash, moves the file into a local content-addressed cache, and writes a small .dvc metafile containing the hash and size. Git tracks only that tiny metafile; the actual bytes go to a remote you configure, S3, GCS, Azure Blob, or a plain SSH server, via dvc push. A teammate who checks out your branch runs dvc pull and gets byte-identical data, because the hash in the metafile tells DVC exactly which cached object to fetch.

This means a Git commit now pins code and data together: checking out any historical commit and pulling reproduces the complete input state of that training run. DVC also layers pipelines on top, dvc.yaml stages declare dependencies and outputs, and dvc repro re-executes only stages whose inputs changed, giving you make-style incremental builds for ML. In interviews, contrast this with alternatives: Git LFS versions large files but has no ML pipeline awareness, while lakeFS and Delta Lake version data at the lake or table level rather than per-repo, which suits centralised data platforms better.

# Track a large dataset with DVC
dvc init
dvc remote add -d storage s3://gs-ml-data/dvc

dvc add data/transactions.parquet
git add data/transactions.parquet.dvc data/.gitignore
git commit -m 'data: transactions snapshot 2026-08'
dvc push          # bytes go to S3, hash stays in Git

# Reproduce a teammate's exact data state
git checkout experiment-42
dvc pull          # fetches the exact hashed objects

# Pipeline stages re-run only when inputs change
dvc repro

Answer

Training-serving skew is a mismatch between the features a model saw during training and the features it receives in production, and it is one of the most common causes of models that evaluate well offline but underperform live. It usually creeps in through duplicated logic: a data scientist computes a feature like average transaction value over 30 days in a Spark job for training, then a backend engineer reimplements the same feature in Java for the serving path. The two implementations drift, different null handling, different time windows, different rounding, and every discrepancy silently degrades predictions.

A feature store eliminates the duplication by making feature definitions single-sourced. You declare a feature once, and the store materialises it into two synchronised surfaces: an offline store (typically a warehouse or lake, BigQuery, Snowflake, S3 plus Parquet) used to build training sets, and an online store (typically Redis, DynamoDB, or Cassandra) that serves the same feature values at low latency during inference. Tools in this space include the open-source Feast, Tecton on the managed side, Databricks Feature Store inside that platform, and SageMaker Feature Store on AWS.

The second problem feature stores solve is point-in-time correctness: when building a training set, the store joins each label with feature values as they existed at that label's timestamp, preventing data leakage from the future. In interviews, define skew crisply, name the two-store architecture, and mention that a feature store is worth its operational cost mainly when multiple models share features or when online inference needs precomputed aggregates.

Answer

The serving pattern should follow the freshness the product actually needs, not what feels most sophisticated. Batch inference scores a large population on a schedule, nightly churn scores for every customer, weekly demand forecasts per SKU, and writes results to a table or cache that applications read later. It is the cheapest and most operationally forgiving pattern: no latency SLO, easy retries, and you can use spot instances.

Pick it whenever predictions do not need to react to the current session. Online (real-time) inference exposes the model behind a synchronous API and scores one request at a time, fraud checks during payment authorisation at a PhonePe-scale gateway, search ranking, credit decisions at checkout. It brings the full weight of production service engineering: p99 latency budgets, autoscaling, feature lookups that must complete in milliseconds, and fallback behaviour when the model times out.

Streaming inference sits between the two: the model consumes an event stream, Kafka or Kinesis, scores events as they arrive, and emits predictions to another topic or store. It suits continuous signals where a synchronous caller does not exist, transaction anomaly flags, IoT sensor scoring, real-time personalisation features feeding a downstream ranker. A useful interview framing: start from label latency and decision latency.

If the decision can wait hours, batch wins on cost and simplicity. If a user is blocked on the answer, you need online. If events flow continuously and consumers are asynchronous, streaming fits. Many mature systems combine them, batch precomputes heavy features, online does the final low-latency scoring.

Answer

Containerizing training makes runs portable and reproducible: the same image executes identically on a laptop, a CI runner, and a GPU node in Kubernetes. The image should contain the things that define the computation: the Python runtime, pinned dependencies from a lockfile (not a loose requirements.txt with unpinned versions), your training source code, and default configuration. For GPU training, base the image on an nvidia/cuda runtime image matching your framework's CUDA requirement, or use framework-published images like pytorch/pytorch, and rely on the NVIDIA container toolkit on the host to expose the GPU.

Three things must stay out of the image. First, data: bake in a multi-gigabyte dataset and every code tweak rebuilds and re-pushes gigabytes, so data should be mounted or pulled at runtime from object storage using a versioned reference. Second, credentials: secrets enter through the orchestrator (Kubernetes secrets, IAM roles, workload identity), never through ENV lines or COPY.

Third, outputs: models and metrics leave the container by being pushed to the artifact store and tracking server, because the filesystem is ephemeral. Practical hygiene interviewers listen for: order Dockerfile layers so dependency installation is cached and code changes only invalidate the final layers; keep images slim so cold-start pulls on autoscaled nodes are fast; pin the base image by digest for reproducibility; and make the entrypoint a parameterised module so the same image serves smoke tests on sampled data and full training runs.

# Dockerfile for a GPU training job
FROM nvcr.io/nvidia/pytorch:24.12-py3

WORKDIR /app

# Dependency layer first: cached until the lockfile changes
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock

# Code layer last: cheap to rebuild on every commit
COPY src/ src/
COPY configs/ configs/

# No data, no secrets, no output paths baked in.
# Data ref and output URI arrive as arguments at runtime.
ENTRYPOINT ["python", "-m", "src.train"]
CMD ["--config", "configs/default.yaml"]

Answer

Model drift is the umbrella term for a deployed model's performance degrading over time even though its code and weights are unchanged. It happens because the world the model observes keeps moving while the model stays frozen at its training snapshot. Interviewers expect you to split it into two mechanisms.

Data drift (covariate shift) means the distribution of inputs changes while the underlying input-to-output relationship stays the same. Example: a credit model trained mostly on salaried applicants starts receiving a surge of gig-economy applicants after a marketing push. The model may still be directionally correct, but it is extrapolating into regions it saw rarely during training, so error rates rise.

Data drift is detectable without labels, you compare live feature distributions against the training distribution using statistics like PSI or the Kolmogorov-Smirnov test. Concept drift means the relationship between inputs and the target itself changes: the same inputs now map to different outcomes. Fraud is the canonical example, fraudsters actively adapt, so a pattern that indicated fraud last quarter becomes benign and new patterns emerge.

Concept drift generally requires labels (or strong proxies) to detect, because feature distributions can look perfectly stable while accuracy collapses. The distinction matters operationally: data drift can sometimes be handled by reweighting or retraining on recent data, while sustained concept drift demands fresh labels and often feature or architecture changes. Also mention prediction drift, a shift in the model's output distribution, as a cheap early-warning signal that something upstream changed, even before labels arrive.

Answer

Reproducibility means someone can re-execute a training run months later and obtain the same model, or at minimum a statistically equivalent one, from the same inputs. It decomposes into pinning every source of variation. Code: the exact git commit, including preprocessing and feature code, not just the model script.

Data: a versioned snapshot reference (DVC hash, lakeFS commit, Delta table version), because tables that get overwritten in place are the single most common reproducibility killer. Configuration: every hyperparameter and flag recorded by the tracking system, with no hidden defaults that differ between environments. Environment: pinned dependency lockfiles and a container image digest, since a minor library upgrade can change numerical results; for GPU work, CUDA and cuDNN versions matter too.

Randomness: seeds set for Python, NumPy, and the framework, plus deterministic algorithm flags where exactness is required, noting that full determinism on GPUs costs performance because some fast kernels are non-deterministic. Order effects: data shuffling, augmentation, and dataloader worker seeding all inject randomness that must be controlled. A mature answer also acknowledges the pragmatic tiers: bit-exact reproducibility is expensive and usually reserved for regulated domains like credit underwriting, where an auditor may ask you to regenerate a specific decision's model. Most teams target metric-level reproducibility, rerunning yields the same validation metrics within a small tolerance, and enforce it with a periodic CI job that retrains a small model from pinned inputs and compares metrics against the recorded run.

Answer

A notebook on a cron schedule is how many production incidents begin. Notebooks encourage hidden state, cells executed out of order, variables lingering from previous sessions, so the file that works interactively can fail or, worse, silently produce a different model when run top-to-bottom. They also bundle everything into one monolithic execution: if training fails at the evaluation step after two hours of feature computation, you rerun everything.

A training pipeline decomposes the workflow into explicit, individually retryable steps, typically ingest, validate, transform, train, evaluate, register, connected as a DAG in an orchestrator like Airflow, Kubeflow Pipelines, Dagster, or a managed service such as Vertex AI Pipelines or SageMaker Pipelines. That structure buys concrete operational properties. Isolation: each step runs in its own container with declared inputs and outputs, so failures are attributable and retries resume from the failed step with cached upstream outputs intact.

Validation gates: a data-quality step can halt the pipeline before bad data reaches training, which is impossible to enforce inside a free-form notebook. Observability: the orchestrator records step durations, logs, and lineage, so you can answer why last night's run was slow. Parallelism: independent branches, say training per region or per category, fan out concurrently.

And automation: pipelines are triggerable by schedule, by data arrival, or by a drift alert, which is the foundation of continuous training. The honest nuance interviewers appreciate: notebooks remain excellent for exploration, and the skill is extracting stabilised logic into tested pipeline steps, not banning notebooks.

Answer

Production model monitoring has four layers, and accuracy is only part of one of them. First, system health, the same signals as any service: request rate, latency percentiles (p50/p95/p99), error rates, saturation of CPU, memory, and GPU. A model that predicts brilliantly but times out at p99 is a failed deployment.

Second, data quality on the inference path: missing or null feature rates, schema violations, out-of-range values, and feature staleness, how old the precomputed features being served are. A broken upstream job that freezes a feature at last Tuesday's value will not throw an error; only staleness monitoring catches it. Third, statistical behaviour: input feature distributions compared against a training reference (PSI, KS tests), and the prediction distribution itself.

A fraud model whose flag rate jumps from 2% to 9% overnight is alerting you to a problem even with zero labels available. Fourth, actual outcome quality once ground truth arrives: rolling-window precision, recall, AUC, calibration error, and, crucially, business KPIs, approval rates, chargeback rates, click-through, revenue per session, sliced by important segments such as geography, platform, and customer tier, because aggregate metrics routinely hide a collapse in one segment. Tie each layer to an escalation: system alerts page immediately, drift alerts open investigations, and sustained outcome degradation triggers the retraining runbook. Tools here include Evidently, Arize, Fiddler, WhyLabs, and warehouse-native SQL checks; interviewers care less about the vendor and more about whether you can name signals that work before labels arrive.

Answer

Application CI/CD validates code and ships a build; ML CI/CD must additionally validate data and models, and it introduces a third continuous process, continuous training. On the CI side, an ML repository runs everything a normal service runs, linting, unit tests for feature transforms and utility code, type checks, plus stages that have no app-code equivalent. Data validation tests assert schema and distribution expectations on training inputs, using tools like Great Expectations or pandera.

A training smoke test executes the full training loop on a small data sample in minutes, catching broken pipelines cheaply before an expensive full run. Model tests evaluate a freshly trained candidate: threshold tests (AUC must exceed a floor), comparison tests (must not underperform the current production model beyond a tolerance), behavioural and invariance tests (predictions should not change when a protected or irrelevant attribute changes), and slice tests over critical segments. On the CD side, the artifact being deployed is not just a container but a model version from the registry, and promotion is gated on evaluation evidence rather than only on tests passing.

Deployment patterns are model-aware: shadow traffic and canary rollout judged on prediction quality metrics, with rollback meaning re-pointing an alias to the previous model version. The third loop, continuous training, has no application analogue at all: pipelines retrain on schedules or triggers, and their outputs flow through the same gated promotion path. A good closing line for interviews: in ML, a green pipeline is necessary but not sufficient, because the artifact can be statistically wrong while being computationally correct.

Answer

Serialization choice decides portability, safety, and what infrastructure can load your model. Pickle (and joblib, its large-array-friendly wrapper) is the scikit-learn ecosystem default: it snapshots arbitrary Python objects, so it captures full pipelines including custom transformers with zero effort. Its liabilities are serious, though.

Unpickling executes arbitrary code by design, so loading an untrusted pickle is remote code execution; this is why Hugging Face and security scanners flag pickle-based checkpoints. Pickle is also environment-bound: the loading process needs compatible versions of Python and every library referenced in the object graph, so a scikit-learn upgrade can strand old artifacts. ONNX takes the opposite approach: it exports the model as a framework-independent computation graph that ONNX Runtime executes anywhere, C++ services, mobile, edge devices, browsers, usually with inference speedups from graph optimisation and quantisation support.

The cost is that export can be lossy or fiddly for exotic architectures and custom ops, and you serialise the computation, not arbitrary preprocessing code. safetensors, now the standard for deep learning weights, stores pure tensors with a small JSON header: no code execution on load, memory-mappable for fast startup, and framework-agnostic at the tensor level. It stores weights only, so the model class definition must exist in code at load time. Practical guidance interviewers want: pickle only for artifacts you produced and load inside the same trusted, version-pinned environment; safetensors for neural network checkpoints; ONNX when serving needs to be fast, portable, or Python-free.

# Pickle/joblib: convenient, trusted-environment only
import joblib
joblib.dump(sk_pipeline, 'model.joblib')  # captures full pipeline

# ONNX: portable optimised graph, Python-free serving
import torch
torch.onnx.export(
    torch_model, sample_input, 'model.onnx',
    input_names=['features'], output_names=['score'],
    dynamic_axes={'features': {0: 'batch'}},
)

# safetensors: tensors only, no code execution on load
from safetensors.torch import save_file, load_file
save_file(torch_model.state_dict(), 'model.safetensors')
state = load_file('model.safetensors')  # safe + memory-mapped

Answer

The FTI architecture structures an ML system as three independently deployable pipelines connected by shared stores, and it has become a common whiteboard framework in system design rounds. The feature pipeline turns raw data into features: it reads from source systems, warehouse tables, event streams, applies transformations and aggregations, and writes versioned features to a feature store or feature tables. It runs on its own cadence, hourly or streaming, regardless of when anyone trains.

The training pipeline reads features and labels from the offline store, trains and evaluates candidate models, and writes accepted versions to the model registry. It runs on a schedule or on triggers like drift alerts. The inference pipeline reads the current model from the registry and fresh features from the online or offline store, produces predictions, and delivers them via API responses, streams, or batch tables.

The connective tissue is deliberate: pipelines never call each other directly; they communicate only through the feature store and the model registry. That decoupling is the whole point. Each pipeline scales and fails independently, a broken feature backfill does not take down serving, which keeps using the last materialised features.

Teams can own different pipelines. Batch and real-time products reuse the same feature and training pipelines, differing only in the inference pipeline. And the interfaces make testing tractable: you can validate the feature pipeline's output contract without running training at all. When asked to design an ML system end to end, opening with the three pipelines and their two shared stores gives you a clean skeleton to hang every subsequent detail on.

Answer

MLflow is the open-source tool interviewers assume you know, so being precise about its components signals hands-on experience. Tracking is the core: a server plus client API that records runs, parameters, metrics with step-wise history, tags, and artifacts, browsable in a UI where you compare runs side by side. The backend store (a SQL database) holds metadata while the artifact store (S3, GCS, Azure Blob) holds files.

Models is the packaging layer: a saved MLflow model is a directory with the serialised model, a conda or pip environment specification, and an MLmodel descriptor declaring flavors, framework-specific ways to load it (sklearn, xgboost, pytorch, and pyfunc, the universal Python-function interface every downstream tool can call). This standard packaging is what lets one deployment path serve models from any framework. Model Registry adds named models with immutable versions, aliases such as champion and challenger (which replaced the deprecated staging/production stages), annotations, and lineage back to the producing run, making it the promotion and rollback control point.

Projects defines a reproducible way to package and run training code with declared entry points and environments, though in practice many teams standardise on their own containers instead. Since MLflow 3, the platform also covers GenAI workflows: tracing for LLM applications, prompt management, and evaluation tooling, reflecting how much of the 2026 workload is LLM-based. A rounded answer notes deployment reality: teams typically run a shared tracking server with Postgres and S3 behind it, or consume MLflow managed inside Databricks, and treat the registry aliases as the source of truth that serving infrastructure resolves at load time.

Answer

Kubernetes treats GPUs as extended resources exposed by the NVIDIA device plugin, requested as nvidia.com/gpu in a container's resource limits. Three properties make GPU scheduling different from CPU and memory. First, GPUs are integer resources: you request whole devices, and they cannot be oversubscribed or fractionally shared by default the way CPU millicores can (sharing requires explicit mechanisms like time-slicing or MIG, which are their own topic).

Second, GPUs are scarce and expensive, so clusters typically dedicate tainted node pools to them; your workload needs matching tolerations and a nodeSelector or affinity for the right GPU type, because an A100 job scheduled onto a T4 node either fails or silently underperforms. Third, training jobs are batch workloads, not services: they should run as Kubernetes Jobs with restart and backoff policies, not Deployments, and long runs must checkpoint to object storage so eviction or node failure resumes instead of restarting from zero. Operationally, you also drive utilisation: an idle reserved GPU costs the same as a busy one, so teams monitor GPU utilisation and memory via DCGM metrics in Prometheus and consolidate under-utilised workloads.

On managed platforms, SageMaker training jobs or Vertex AI custom jobs hide the node management but the same concepts apply, instance type selection, checkpointing for spot capacity, and utilisation monitoring. Mentioning that driver and CUDA compatibility comes from pairing the NVIDIA container toolkit on hosts with CUDA-matched images shows you have actually debugged one of these clusters.

apiVersion: batch/v1
kind: Job
metadata:
  name: train-recsys-v14
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        gpu-type: a100          # right silicon, not just any GPU
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule    # allowed onto the tainted GPU pool
      containers:
        - name: trainer
          image: registry.internal/recsys-train:1.4.2
          args: ['--config', 'configs/prod.yaml',
                 '--resume-from', 's3://ckpts/recsys-v14/latest']
          resources:
            limits:
              nvidia.com/gpu: 1  # whole GPUs only, no fractions
              memory: 64Gi

Answer

The naive version, one task that runs a training script every Sunday, misses everything that makes retraining safe. A production DAG needs distinct stages with guards between them. Extract snapshots the training data as of a fixed cutoff and writes it to a versioned location, so the run is reproducible and reruns of the same logical date read identical data (Airflow's logical date exists precisely for this).

Validate runs data-quality checks, schema, null rates, volume compared to previous weeks, label balance, and fails the DAG before any GPU money is spent if expectations break. Train executes the containerised training job, ideally via KubernetesPodOperator or a managed training job API rather than inside the Airflow worker, logging everything to MLflow; Airflow orchestrates, it should not compute. Evaluate compares the candidate against the current production model on a held-out window, overall metrics plus critical segment slices.

The promotion guard is the step candidates forget: the new model registers and gains the staging alias only if it beats the incumbent by a configured margin; otherwise the DAG ends cleanly with a notification, and production keeps the proven model. Retraining that unconditionally deploys whatever it produced is how silent regressions ship. Around the DAG itself: catchup disabled unless backfills are intended, retries with exponential backoff on transient steps, SLA alerts so a hung run pages someone, and idempotent tasks throughout, any step must be safe to rerun because the scheduler will eventually rerun it.

from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule='0 2 * * 0', start_date=datetime(2026, 1, 4),
     catchup=False, tags=['ml', 'churn'],
     default_args={'retries': 2, 'retry_exponential_backoff': True})
def churn_weekly_retrain():

    @task
    def extract(logical_date=None):
        # snapshot as-of cutoff -> versioned path, idempotent per week
        return snapshot_features(cutoff=logical_date)

    @task
    def validate(path):
        report = run_expectations(path)   # schema, nulls, volume, labels
        if not report.success:
            raise ValueError(report.summary)
        return path

    @task
    def train(path):
        # launches the training container; Airflow only orchestrates
        return submit_training_job(data=path)   # returns mlflow run_id

    @task
    def gate_and_register(run_id):
        if beats_champion(run_id, metric='val_auc', margin=0.002):
            register_as_challenger('churn-model', run_id)
        else:
            notify_slack('Candidate lost to champion, keeping prod model')

    gate_and_register(train(validate(extract())))

churn_weekly_retrain()

Answer

Frame the choice along three axes: what else the team orchestrates, where infrastructure lives, and how ML-native the workflows need to be. Airflow is the general-purpose incumbent. Most Indian data teams already run it for ELT, so adding ML DAGs means zero new infrastructure and one scheduler for everything; Airflow 3 modernised the stack with a stable REST API, DAG versioning, and event-driven scheduling.

Its weakness is that it is not ML-aware: no built-in artifact passing, caching, or lineage between steps, you assemble those from operators, XComs, and MLflow yourself. Kubeflow Pipelines is Kubernetes-native and ML-first: pipelines compile from Python into containerised steps with typed artifact passing, per-step caching, and experiment tracking hooks. The cost is operating Kubeflow itself, which is notoriously heavy; it makes sense for platform teams already committed to Kubernetes who need multi-tenant ML infrastructure.

Vertex AI Pipelines runs the Kubeflow Pipelines SDK format serverless on GCP, no cluster to manage, per-run pricing, first-class integration with Vertex training, endpoints, and registry. SageMaker Pipelines is the AWS equivalent, tightly integrated with SageMaker jobs, its registry, and Step Functions-style execution. Decision heuristics that land well in interviews: mixed data-plus-ML estate with existing Airflow, keep Airflow and let it trigger containerised training.

Deep single-cloud commitment, prefer that cloud's managed pipelines and spend zero effort on orchestrator operations. Kubernetes platform team serving many ML squads, Kubeflow (or Argo Workflows underneath) earns its complexity. Also acknowledge Dagster and Prefect as modern general orchestrators with better data-asset ergonomics than Airflow, which several 2026 teams choose for greenfield stacks.

Answer

A canary release sends a small slice of real traffic to the new model while the incumbent keeps serving the rest, limiting the blast radius of a bad model to that slice. The mechanics: run both versions simultaneously and split traffic at a routing layer, a service mesh like Istio, an ingress controller, or natively in serving platforms, KServe's canaryTrafficPercent, SageMaker endpoint variant weights, Vertex AI traffic splits. Start at 5-10%, watch, then step up 25, 50, 100, with automated rollback if guardrails breach at any stage.

What makes model canaries different from app canaries is the evaluation. Application canaries watch errors and latency; a model canary must also watch prediction behaviour, score distributions versus the incumbent, downstream business guardrails like approval or flag rates, and segment-level sanity, because a model can serve 200s at great latency while making terrible predictions. Since ground-truth labels usually lag, define proxy guardrails up front: acceptable divergence in prediction distribution, bounded shift in key rates, no latency regression.

Practical details that show seniority: keep feature pipelines identical for both versions, otherwise you are testing two changes at once; make traffic assignment sticky per user or session where repeated decisions must be consistent; log a canary flag on every prediction so offline analysis can attribute outcomes to the right version once labels arrive; and predefine the abort procedure, rollback is re-pointing traffic to the incumbent, which stayed warm the whole time. For high-stakes models, run a shadow phase before the canary so the first live percent is not the first time the model sees production data.

# KServe: canary 10% of traffic to the new model version
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-scorer
spec:
  predictor:
    canaryTrafficPercent: 10        # incumbent keeps 90%
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://gs-models/fraud-scorer/v14
---
# Promotion = raise canaryTrafficPercent stepwise (25 -> 50 -> 100)
# Rollback  = set canaryTrafficPercent: 0, incumbent never went away

Answer

In a shadow (mirror) deployment, production traffic is duplicated to the new model, which scores every request, but its predictions are logged and discarded; the incumbent alone answers users. The new model experiences real production inputs, real feature lookups, real traffic patterns and volumes, with zero user-facing risk. You then compare offline: score distributions between champion and shadow, disagreement rates and where they concentrate, latency and resource behaviour under genuine load, and, once labels arrive for the shadowed period, actual quality on real traffic.

Shadow is preferable to canary in several situations. When mistakes are expensive or irreversible: credit decisions, medical triage, large-ticket fraud blocking, where even 5% canary traffic means real harmed decisions. When you suspect training-serving skew or integration bugs, shadow surfaces missing features, timeout behaviour, and serialization mismatches that offline evaluation cannot.

When validating a major architecture change, new framework, new feature store, where operational behaviour is as uncertain as statistical behaviour. Its limits are equally important to state. Shadow cannot measure feedback effects: if the model's decisions would change user behaviour or downstream data, recommendations changing what people click, fraud blocks changing what fraudsters attempt, shadow evaluation is blind to that, and only live exposure (canary, then A/B) reveals it.

It also doubles inference compute for the shadowed traffic, and for side-effectful pipelines you must guarantee the shadow path is truly side-effect free, no writes, no notifications, no state mutation. The mature rollout sequence interviewers like: offline evaluation, shadow for one to two weeks, canary ramp, then full traffic with monitoring.

Answer

Drift detection compares a live window of a feature (or prediction) distribution against a reference, usually the training distribution or a stable recent baseline. Population Stability Index (PSI) is the workhorse for tabular features: bin the reference distribution (commonly deciles), compute the share of reference and live data per bin, and sum (live share minus reference share) times log of the ratio. It is symmetric, cheap, and interpretable, with widely used rule-of-thumb bands: below 0.1 stable, 0.1 to 0.25 investigate, above 0.25 significant shift.

The Kolmogorov-Smirnov test compares continuous distributions via the maximum gap between empirical CDFs, giving a statistic and p-value. Its trap is sample size: at production volumes of millions of rows, KS will declare microscopic, practically meaningless differences significant, so use the statistic's magnitude (effect size), not the p-value, as your alert signal. For categorical features use chi-square or PSI over categories; Jensen-Shannon divergence is a bounded, symmetric alternative that behaves well for both.

On thresholds, the honest answer is that defaults are starting points, not truth. Calibrate per feature: backtest the metric over months of known-healthy history, set thresholds a comfortable margin above normal seasonal variation, weight features by importance so a drifting top feature alerts harder than a drifting minor one, and alert on persistence, three consecutive windows breaching, rather than single spikes, because paging on every Diwali-weekend distribution wobble trains the team to ignore drift alerts entirely. Route drift alerts to investigation, not to automatic retraining, until the pipeline has earned that trust.

import numpy as np
from scipy import stats

def psi(reference, live, bins=10):
    cuts = np.percentile(reference, np.linspace(0, 100, bins + 1))
    cuts[0], cuts[-1] = -np.inf, np.inf
    ref_pct = np.histogram(reference, cuts)[0] / len(reference) + 1e-6
    live_pct = np.histogram(live, cuts)[0] / len(live) + 1e-6
    return float(np.sum((live_pct - ref_pct) * np.log(live_pct / ref_pct)))

ref = train_df['txn_amount'].to_numpy()
cur = live_window_df['txn_amount'].to_numpy()

score = psi(ref, cur)
# ~rule of thumb: <0.1 stable, 0.1-0.25 investigate, >0.25 shifted

ks_stat, p_value = stats.ks_2samp(ref, cur)
# At production volumes, alert on ks_stat magnitude, not p_value:
# tiny p-values are guaranteed once n is in the millions.

Answer

There are three trigger families, and mature systems layer them rather than pick one. Scheduled retraining, weekly, monthly, retrains regardless of need. Its virtues are predictability and simplicity: capacity is plannable, freshness is bounded, no monitoring dependency.

Its vices: it wastes compute when nothing changed, and it cannot react between ticks, a payment-fraud model on monthly retraining is defenceless against an attack pattern that emerged this week. Drift-triggered retraining fires when input or prediction distributions shift beyond thresholds. It reacts within hours and spends compute only when warranted, but it needs a well-tuned monitoring stack, and it carries a subtle flaw in both directions: data drift does not always hurt performance (retraining on drift alone can be wasted work), and concept drift can destroy performance with no visible input drift (so drift triggers alone can miss real decay).

Performance-triggered retraining is the ground truth: retrain when rolling precision, AUC, or a business KPI degrades past a floor. It targets exactly the failures that matter but is gated on label latency; in domains where labels arrive in weeks (loan defaults), the alarm rings long after users felt the damage. The layered pattern for an interview: a scheduled baseline as backstop (cadence set by known data velocity), drift triggers for early warning between ticks, performance triggers where labels are timely, and every path converging on the same gated pipeline, candidate must beat the incumbent before promotion, with a cooldown or budget so a flapping trigger cannot retrain in a loop. Also say the quiet part: sustained concept drift often needs new features or labels, and retraining on the same stale recipe just automates mediocrity.

Answer

The design constraint is that full training is too slow and expensive for PR feedback, so CI must maximise confidence per minute. Tier one, always on every PR: linting and formatting (ruff), type checks, and fast unit tests for the code that is most bug-prone in ML repos, feature transformations, label construction, metric implementations, config parsing. Feature logic bugs are the classic source of silent model damage, and they are perfectly unit-testable: given this input frame, the transform must produce exactly these values, including the null and edge cases.

Tier two, data contract tests: run schema and expectation suites (Great Expectations, pandera) against a small versioned fixture dataset, so a PR that changes preprocessing incompatibly fails immediately rather than at 2 AM in the weekly pipeline. Tier three, the training smoke test: execute the entire pipeline, ingest through evaluation through model save, on a tiny sampled dataset, targeting minutes. It will not produce a good model; it proves the plumbing works, shapes line up, the loss decreases, and an artifact with the right signature emerges.

Tier four, model behaviour tests against a small trained candidate or the current production model: invariance checks (renaming a user must not change their score), directional expectations (higher income should not increase default probability, all else equal), and minimum-quality floors on the smoke model where meaningful. Full training and champion-challenger evaluation belong in the merge-triggered or scheduled pipeline, not PR CI. GPU-dependent tests run nightly on a self-hosted runner to keep PR feedback fast and cheap.

# .github/workflows/ml-ci.yaml
name: ml-ci
on: [pull_request]

jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements.lock

      - name: Lint and types
        run: |
          ruff check src tests
          mypy src

      - name: Unit tests (features, labels, metrics)
        run: pytest tests/unit -q

      - name: Data contract tests on versioned fixtures
        run: pytest tests/data -q

      - name: Training smoke test (sampled data, minutes)
        run: python -m src.train --config configs/smoke.yaml

      - name: Model behaviour tests on smoke artifact
        run: pytest tests/model -q

Answer

The demo is fifteen lines: load a joblib file at import time, one POST route, uvicorn. Production hardening touches every layer. Model loading: load once at startup (lifespan handler), never per-request, and resolve the artifact from the registry by alias rather than a baked-in file path, so rollback does not require an image rebuild.

Validation: pydantic schemas for the request that mirror the model's training signature, with explicit handling for missing features, silently imputing zeros for a missing feature is a classic skew bug. Concurrency: model inference is CPU-bound, so an async endpoint that calls predict directly blocks the event loop; either use sync def routes (FastAPI runs them in a threadpool) or offload to a process pool, and size uvicorn/gunicorn workers to cores. Batching: if traffic is high-QPS with small payloads, micro-batching requests over a few milliseconds significantly improves throughput on vectorised models.

Operations: timeouts and a defined fallback response when inference exceeds budget, structured logs carrying a request ID and model version on every prediction, Prometheus metrics for latency percentiles, error rates, and prediction distribution summaries, and separate liveness and readiness probes, readiness must fail until the model is loaded, or Kubernetes will route traffic into a pod that cannot score. Image discipline: multi-stage build, non-root user, pinned digests. Finally, log the full feature vector and prediction (sampled if volume demands) to your monitoring pipeline, because serving is also the data collection point for drift detection and future retraining.

from contextlib import asynccontextmanager
from fastapi import FastAPI
import mlflow, prometheus_client as prom

LATENCY = prom.Histogram('predict_latency_s', 'inference latency')
model = None

@asynccontextmanager
async def lifespan(app):
    global model
    # registry alias, not a baked-in path: rollback = alias flip
    model = mlflow.pyfunc.load_model('models:/fraud-scorer@champion')
    yield

app = FastAPI(lifespan=lifespan)

@app.get('/ready')
def ready():
    return {'ok': model is not None}

@app.post('/score')
def score(req: ScoreRequest):          # sync def -> threadpool,
    with LATENCY.time():               # event loop never blocks
        pred = model.predict(req.to_frame())
    log_prediction(req, pred, model_version='champion')
    return {'score': float(pred[0])}

Answer

These tools solve different layers of the serving problem, and knowing which layer each occupies matters more than feature lists. NVIDIA Triton Inference Server is a high-performance inference engine: multi-framework (TensorRT, ONNX, PyTorch, Python backends), dynamic batching, concurrent model execution on shared GPUs, and model ensembles executed server-side. Choose it when GPU utilisation and raw throughput are the bottleneck, computer vision at scale, low-latency ranking, and you can invest in its model-repository conventions.

TorchServe packages PyTorch models with custom pre/post-processing handlers; it is straightforward for PyTorch shops, though its community momentum has faded compared to alternatives, with maintenance slowing through 2025. KServe is not an engine at all but a Kubernetes control plane: InferenceService CRDs give you autoscaling (including scale-to-zero via Knative), canary traffic splits, and pluggable runtimes, Triton or others can run underneath it. Choose it when you operate many models on Kubernetes and want deployment semantics standardised.

BentoML is a Python-first packaging and serving framework: define a service in code, get a container with batching and adaptive scaling, deployable anywhere; it favours developer velocity for teams shipping diverse Python models without deep Kubernetes expertise. vLLM is the specialist: an LLM inference engine built around PagedAttention for KV-cache efficiency and continuous batching, delivering the throughput that makes self-hosted LLM serving economical; in 2026 it is the default open-source choice for serving open-weight LLMs, often run behind KServe or a gateway. Interview synthesis: pick the engine by workload (Triton for GPU-heavy classical DL, vLLM for LLMs, plain FastAPI for small CPU models), and pick the platform layer (KServe, or managed endpoints on SageMaker/Vertex) by how many models and teams you must standardise.

Answer

Feast's core mechanism is a single feature definition compiled against two storage backends. You declare entities (join keys like user_id), feature views (features with their source table or stream, and a TTL), and Feast manages an offline store, your warehouse or lake holding full history, and an online store, Redis, DynamoDB, or similar, holding only the latest value per entity for millisecond lookups. Materialisation jobs copy the newest feature values from offline to online on a schedule or from a stream, so both surfaces derive from the same definition and the same upstream computation.

That single sourcing is what prevents training-serving skew: the serving path reads the same materialised values the training path was built from, rather than a reimplementation. The point-in-time correct join is the subtler half. When building a training set, each label row has an event timestamp; for every feature, the join must fetch the value as it existed at that timestamp, the most recent value at or before it, within TTL, never a later one.

A naive join to current feature values leaks the future: a churn label from March joined against June's activity features produces beautiful validation metrics and a model that falls apart live, because at serving time the future is not available. Feast's get_historical_features performs this as-of join across all requested feature views automatically. Points that earn credit: TTLs bound how stale an online value may be and simultaneously define lookback in as-of joins; streaming sources reduce online staleness for fast-moving features; and the feature store is worth its operational cost mainly once multiple models share features or online inference needs precomputed aggregates, a single-model team can start with plain tables and discipline.

from feast import FeatureStore

store = FeatureStore(repo_path='.')

# TRAINING: point-in-time correct join against label timestamps
training_df = store.get_historical_features(
    entity_df=labels_df,   # user_id, event_timestamp, churned
    features=[
        'user_stats:txn_count_30d',
        'user_stats:avg_ticket_size',
        'user_stats:days_since_last_order',
    ],
).to_df()
# each row gets feature values as-of ITS timestamp: no future leakage

# SERVING: same definitions, latest values, single-digit ms
features = store.get_online_features(
    features=[
        'user_stats:txn_count_30d',
        'user_stats:avg_ticket_size',
        'user_stats:days_since_last_order',
    ],
    entity_rows=[{'user_id': 91042}],
).to_dict()

Answer

Attack in order of leverage. First, utilisation, because an idle reserved GPU is pure waste: measure actual GPU utilisation and memory via DCGM or cloud monitoring before buying anything. Common findings are GPUs starved by input pipelines (fix with more dataloader workers, prefetching, faster storage or caching, moving CPU-heavy augmentation off the training node) and oversized instances scoring 20% utilisation on a model that fits comfortably on cheaper silicon.

Right-sizing, T4 or L4 for small models and inference-adjacent work, A100/H100 reserved for what genuinely needs them, is often a large saving by itself. Second, spot and preemptible capacity, at steep discounts versus on-demand, made safe by engineering for interruption: checkpoint to object storage every N steps, resume automatically on restart, and keep individual jobs short enough that losing a node costs minutes. Managed training on SageMaker supports spot with checkpointing natively; on Kubernetes, run jobs with resume-from-latest arguments as a habit.

Third, spend less compute per result: mixed-precision training (bf16) roughly doubles effective throughput on modern GPUs; early stopping kills doomed runs; hyperparameter search should use Bayesian or successive-halving strategies (ASHA) instead of grid search, which burns most of its budget on hopeless corners; and for LLM fine-tuning, parameter-efficient methods like LoRA/QLoRA reduce hardware requirements by an order of magnitude versus full fine-tuning. Fourth, governance: per-team quotas and queueing (Kueue on Kubernetes), scheduled shutdown of dev notebooks with attached GPUs, idle-run reapers, and cost dashboards per project, because visibility alone changes behaviour. Reserved capacity or savings plans then cover only the stable baseline that survives all the above.

Answer

Lineage is the recorded chain of derivation: this prediction came from this model version, trained by this pipeline run, from this code commit, this data snapshot, these parameters, through these transformations. It is what turns three recurring situations from archaeology into queries. Debugging: predictions went strange on Tuesday, was it the model, a feature pipeline change, or an upstream table?

Compliance: a regulator or auditor, RBI in the Indian credit context, asks why this applicant was declined, which requires reconstructing the exact model and inputs for that decision. Impact analysis: this source table has a bug discovered today, which features, models, and downstream decisions are contaminated, and how far back? Building it is less about one tool than about discipline at every hop.

Training-side lineage largely falls out of good practice: the tracking system records run to code-commit to data-version to parameters, and the registry records model-version to run. Pipeline orchestrators add step-level lineage, and OpenLineage has emerged as the vendor-neutral standard for emitting lineage events from Airflow, Spark, and dbt into catalogs like Marquez or DataHub, connecting ML lineage to the broader data platform's table-level lineage. The hop teams most often miss is serving: every prediction log line must carry the model version (and ideally feature snapshot identifiers or timestamps), because without that final link you can reconstruct any model but cannot say which one made a given decision. The complete answer to the interview question is a walk of that chain backwards: prediction log gives model version and feature timestamps; registry gives the training run; the run gives commit, data hash, and parameters; data versioning gives the exact bytes; and the whole path should be walkable in minutes.

Answer

The promotion path should look like a deployment pipeline where the artifact is a registry version and every advance is gated on evidence. Step one: a training run that passes its own evaluation registers a new version of the named model, carrying lineage to the run and its metrics. Step two: the version receives a challenger alias.

Challenger status triggers automated evaluation against the champion on a fixed, held-out recent window: primary metrics, segment slices, calibration, latency of the packaged artifact, and any fairness or policy checks the domain requires. The comparison must be apples-to-apples, same data window, same metric code, ideally executed by one evaluation job that scores both models, because subtle differences in evaluation data are the classic way a worse model looks better. Step three: a promotion gate.

For low-stakes models this can be fully automated, challenger beats champion by the configured margin, alias flips. For high-stakes models (credit, fraud, pricing) insert a human approval: the gate assembles an evidence bundle, metric deltas, slice tables, drift context, and a designated owner approves in the registry UI or via a ticketed workflow; the registry records who approved and when, which is your audit trail. Step four: the champion alias moves to the new version, and serving, which always loads models:/name@champion, picks it up on next refresh, often after a canary phase where infrastructure splits traffic between the two aliases.

Rollback is the payoff for this discipline: flip the alias back, no rebuild, no redeploy. Modern MLflow made aliases the mechanism precisely because the old rigid stage names encouraged teams to treat stage transitions as informal bookkeeping rather than gated promotion.

from mlflow import MlflowClient
import mlflow

client = MlflowClient()

# 1. Register the candidate produced by a tracked run
mv = client.create_model_version(
    name='fraud-scorer',
    source=f'runs:/{run_id}/model',
    run_id=run_id,
)

# 2. Mark as challenger -> triggers champion-vs-challenger eval
client.set_registered_model_alias('fraud-scorer', 'challenger', mv.version)

report = evaluate_pair('fraud-scorer', champion='champion',
                       challenger='challenger', window='last_14d')

# 3. Gate: metric margin + (for high-stakes models) human approval
if report.challenger_wins(margin=0.002) and approval_granted(report):
    # 4. Promotion is one alias flip; so is rollback
    client.set_registered_model_alias('fraud-scorer', 'champion', mv.version)

# Serving always resolves the alias, never a version number:
model = mlflow.pyfunc.load_model('models:/fraud-scorer@champion')

Answer

All three let you name and recover historical data states, but they operate at different levels of the stack, and the interview signal is knowing which level a given team needs. DVC versions data at the repository level: content-hashed files referenced from Git metafiles, with the bytes in object storage. Its unit of versioning is a project's datasets, and its natural home is the ML team's workflow, a model repo where code and its training data snapshot travel in the same commits, plus lightweight pipeline caching via dvc repro.

It does not scale gracefully to petabyte lakes or to datasets shared by dozens of teams, and it versions files, not tables. lakeFS versions data at the object-store level: it fronts S3/GCS/MinIO with Git-like semantics, branches, commits, merges, over entire buckets, using zero-copy metadata operations so branching a petabyte is instant. That enables genuinely powerful patterns: branch the lake, run a risky backfill or a new ETL version on the branch, validate, then merge or discard atomically; or pin a training job to a lakeFS commit for perfect reproducibility across teams. It is infrastructure a platform team operates, overkill for one squad, transformative for a data platform.

Delta Lake (like Apache Iceberg and Hudi) versions at the table format level: every write creates a new table snapshot in the transaction log, and time travel queries any prior version by number or timestamp. It is transparent, no new workflow, and ideal in warehouse-centric Databricks or Spark estates; its limits are retention (old snapshots are vacuumed away, so long-term reproducibility needs explicit cloning or exporting of pinned snapshots) and scope, it versions tables individually, not a coherent multi-table state. Rule of thumb: DVC for ML-repo-scale, Delta/Iceberg time travel inside lakehouse platforms, lakeFS when the whole lake needs branch-and-merge semantics.

Answer

Delayed labels are the norm, not the exception, loan defaults take months, churn takes a billing cycle, even fraud chargebacks take weeks, so a monitoring strategy that waits for labels is a strategy of finding out last. The toolkit for the label-free window has several layers. Proxy signals: identify early behavioural indicators correlated with the eventual label, for churn, logins and order frequency drop before the subscription lapses; for credit, early missed payments precede default, and monitor the model's performance against proxies while acknowledging their imperfection.

Prediction and input monitoring: drift in feature distributions and in the score distribution is fully label-free and catches upstream breakage and population shift immediately; a stable model on a stable population should produce a stable score distribution, so a moving one demands explanation. Confidence and uncertainty signals: rising average uncertainty, or a growing share of predictions near the decision threshold, indicates the model is being asked about inputs it knows less well. Downstream business guardrails: approval rates, flag rates, queue volumes for human review, conversion, these move within hours and are what stakeholders actually feel.

There are also estimation techniques designed for exactly this gap: confidence-based performance estimation (CBPE, implemented in NannyML) estimates a classifier's post-deployment AUC or accuracy from its own calibrated scores, no labels required, under assumptions (mainly covariate shift without concept drift) that you must state honestly. Finally, engineer the label pipeline itself: when labels do arrive, backfill them against stored predictions automatically so delayed evaluation is continuous rather than an ad-hoc quarterly study, and consider paying for a small stream of fast labels, manual review of a sample, to shrink the blind window for your most critical segments.

Answer

Start with the honest framing: on GPUs, bit-exact reproducibility is achievable only if you pay for it, and most teams should target controlled, documented variance instead. The sources of non-determinism stack up. Framework level: many fast CUDA kernels (notably atomics-based reductions and some cuDNN convolution algorithms) are non-deterministic; PyTorch offers torch.use_deterministic_algorithms(True), which switches to deterministic implementations or raises when none exists, at a measurable speed cost, and cuDNN benchmarking mode must be off because it picks algorithms by timing races.

Data level: shuffling, augmentation, and dataloader workers each carry their own RNG state; seed Python, NumPy, and torch globally, seed each worker via worker_init_fn, and pass a seeded generator to the DataLoader so shuffle order is fixed. Distributed level: collective operations like all-reduce sum floating-point numbers in an order that can vary across runs and topologies, and floating-point addition is not associative, so changing world size, gradient accumulation steps, or even NCCL's internal algorithms changes results slightly; reproducibility claims are therefore only valid for a fixed world size and hardware configuration. Environment level: pin the container digest, CUDA, cuDNN, NCCL, and framework versions, because kernel implementations change between releases.

Given all this, the pragmatic contract most teams adopt: identical seeds plus identical environment plus identical topology must reproduce metrics within a stated tolerance, and a periodic CI job verifies it; full determinism flags are reserved for debugging sessions (bisecting a training divergence) and regulated audits. Saying that checkpoint-resume must also restore RNG and optimizer state, not just weights, otherwise a resumed run silently diverges from an uninterrupted one, is a detail that lands well.

import os, random
import numpy as np
import torch
from torch.utils.data import DataLoader

def seed_everything(seed: int = 42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # exact kernels or raise; costs speed, use when it matters
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.benchmark = False   # no timing races
    os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'

def worker_init(worker_id: int):
    s = torch.initial_seed() % 2**32     # per-worker derived seed
    np.random.seed(s)
    random.seed(s)

loader = DataLoader(
    dataset,
    batch_size=256,
    shuffle=True,
    num_workers=4,
    worker_init_fn=worker_init,
    generator=torch.Generator().manual_seed(42),  # fixed shuffle order
)

Answer

The architecture chains systems you should already have: monitoring detects, orchestration retrains, gates evaluate, the registry promotes, serving refreshes. Concretely: the drift detector (comparing live feature and prediction windows against references) emits an event; an event bus or webhook triggers the training pipeline with the drift report as input; the pipeline snapshots fresh training data as-of a cutoff, runs data validation, trains the candidate, and evaluates it against the champion on a held-out recent window including segment slices; if the candidate wins by the configured margin, it gains the challenger alias, rolls through an automated canary with guardrail metrics, and on a clean canary the champion alias flips; serving resolves the alias and the loop closes with an audit event documenting the whole chain. The engineering that makes this safe rather than reckless: idempotency and a cooldown so a flapping drift signal cannot trigger overlapping retrains; a compute budget per model per week; every automated decision logged with its evidence; and an automatic halt that pages a human whenever any gate fails twice consecutively, because repeated failure means the world changed in a way the recipe cannot fix.

Where humans deliberately stay: first, threshold and recipe changes, the automation may execute the policy but never edit it; second, high-stakes model families (credit, pricing, anything regulator-visible) keep a human approval step at promotion even when everything upstream is automated, with the evidence bundle prepared by the pipeline so approval takes minutes; third, post-incident review of any automated rollback. The interview framing that lands: automate the loop, but design it so the human moments are few, fast, and well-informed rather than absent.

Answer

A prompt change alters production behaviour exactly like a code change, but teams routinely ship prompts through a dashboard textbox with no review, no history, and no tests. The fix is treating prompts as versioned, evaluated artifacts. Versioning: prompts live in the repository (or a prompt registry such as Langfuse, MLflow prompt registry, or LangSmith) as structured files with an identifier, version, model parameters, and template variables.

Every production LLM call logs which prompt version produced its output, which is the lineage link that makes incidents debuggable. Changes arrive as pull requests with diffs and review, and rollback is redeploying the previous version. Evaluation gates: the CI pipeline runs every changed prompt against a frozen, versioned evaluation set, real cases collected from production plus adversarial and edge cases, curated and expanded whenever an incident reveals a gap.

Scoring combines exact checks where outputs are structured (JSON validity, required fields, correct labels on classification-style tasks), rubric-based LLM-as-judge scores for free-form quality, and safety checks (refusal correctness, injection resistance, banned content). The gate compares the candidate prompt against the current version's scores and blocks the merge on regression beyond tolerance, exactly champion-challenger logic applied to prompts. Two failure modes to name: judge instability, mitigate with pinned judge model versions, low temperature, and periodic human calibration of judge scores against expert labels; and eval-set overfitting, mitigate by rotating in fresh production samples and keeping a held-out set that never appears in development. Finally, provider model updates change behaviour under a frozen prompt, so rerun the eval suite when the underlying model version changes, not only when your prompt does.

# prompts/support_triage.yaml (reviewed via PR like any code)
# id: support_triage, version: 14, model: fixed, temperature: 0

# ci/eval_gate.py: blocks merge on regression
import json, sys

EVALS = json.load(open('evals/support_triage_v3.json'))  # frozen set
FLOORS = {'schema_valid': 1.0, 'label_accuracy': 0.87,
          'injection_resistance': 1.0}

def score_prompt(prompt_version):
    results = []
    for case in EVALS:
        out = call_llm(prompt_version, case['input'], temperature=0)
        results.append({
            'schema_valid': is_valid_json(out),
            'label_accuracy': out_label(out) == case['expected'],
            'injection_resistance': not leaked(out, case),
        })
    return aggregate(results)

candidate = score_prompt(sys.argv[1])
incumbent = score_prompt('production')

for metric, floor in FLOORS.items():
    if candidate[metric] < max(floor, incumbent[metric] - 0.01):
        sys.exit(f'GATE FAILED: {metric}={candidate[metric]:.3f}')
print('gate passed')

Answer

Token cost is a production metric and deserves the same treatment as latency: instrumented per request, budgeted, alerted, and optimised. Instrumentation first, because you cannot control what you cannot attribute: every LLM call logs input tokens, output tokens, model, prompt version, feature or endpoint, and tenant or user segment, either through a gateway (LiteLLM-style proxies, or observability layers like Langfuse and Helicone) or in your own client wrapper. Dashboards then answer the diagnostic questions in minutes: which feature grew, which segment, input or output tokens, volume or per-request size.

The common culprits are unglamorous: context stuffing (RAG pipelines appending ever more chunks, chat histories growing unboundedly), retry storms silently multiplying calls, a prompt edit that ballooned the template, agent loops making many hidden calls per user action, and traffic simply growing. Controls, in rough order of return: caching, exact and semantic response caches for repeated queries, and prompt caching (supported by major providers in 2026) which discounts the repeated static prefix of your prompts dramatically when structured correctly, static instructions first, variable content last. Model routing: most requests do not need the frontier model; classify or route simple cases to a much cheaper small model and escalate the hard ones, which typically cuts blended cost severalfold.

Context discipline: cap history length with summarisation, retrieve fewer but better chunks, trim boilerplate from templates. Output limits: bounded max output tokens, structured outputs instead of prose where possible. Governance closes the loop: per-feature monthly budgets with alerts at thresholds, hard rate limits per tenant so one abusive integration cannot burn the budget, and cost-per-request as a release gate metric so a prompt or routing change that doubles unit cost is caught in CI, not on the invoice.

# Gateway middleware: attribute every call, enforce budgets
async def llm_call(feature, tenant, prompt_version, messages, model):
    if monthly_spend(feature) > BUDGETS[feature]:
        model = FALLBACK_MODEL[feature]      # degrade, do not die

    resp = await client.chat(model=model, messages=messages)

    usage = resp.usage
    cost = (usage.input_tokens * PRICE[model]['in'] +
            usage.output_tokens * PRICE[model]['out'])

    metrics.emit('llm.cost', cost, tags={
        'feature': feature, 'model': model,
        'prompt_version': prompt_version, 'tenant': tenant,
        'cache_hit': resp.cache_read_tokens > 0,
    })
    return resp

# Alert rules (Prometheus-style pseudocode):
#  sum(llm.cost) by (feature) > 0.8 * budget  -> warn
#  rate(llm.calls[5m]) spikes 5x baseline     -> retry-storm page
#  avg input_tokens per call up 50% week/week -> context-bloat ticket

Answer

A canary answers an operational question, is the new version safe to expose, using guardrails and small asymmetric traffic. An A/B test answers a causal business question, does the new model improve the outcomes we care about, and that demands experimental rigour a canary does not have. Design elements interviewers probe: randomisation must be at the right unit, usually the user, not the request, both to avoid one person seeing inconsistent decisions and because request-level splits violate independence when a user makes many requests; assignment must be sticky (hash of user ID and experiment salt) and logged with every prediction.

The primary metric is a business outcome (conversion, approval rate, default rate, revenue per session), declared before the test, with guardrail metrics (latency, complaint rate, fairness slices) that can stop the test regardless of the primary. Sample size and duration come from a power calculation on the expected effect size, and the test runs through at least one full weekly cycle to absorb seasonality; stopping the moment significance first appears is the classic peeking error that inflates false positives, so use fixed horizons or sequential testing methods designed for continuous monitoring. ML-specific complications: feedback loops, the model's own decisions change future data (a lending model's approvals determine whose repayment you observe), which contaminates naive comparisons; interference, when both arms draw from shared inventory or budgets (recommendations competing for the same stock) arm outcomes are not independent, motivating switchback or cluster designs; and delayed outcomes, requiring either patience or validated early surrogates. Operationally the sequence is shadow, then canary for safety, then A/B for value, and the decision memo should report effect sizes with confidence intervals, not a bare significant/not-significant verdict.

Answer

Start by splitting the problem: sharing individual GPUs (utilisation) and arbitrating between teams (fairness). For sharing, Kubernetes offers two main mechanisms with very different guarantees. Time-slicing configures the NVIDIA device plugin to advertise one physical GPU as several schedulable replicas; workloads take turns on the whole GPU with no memory or fault isolation, one tenant can exhaust memory and crash its neighbours.

It suits trusted, bursty, light workloads: notebooks, dev experiments, low-QPS inference. MIG (Multi-Instance GPU, on A100/H100-class hardware) partitions a GPU at the hardware level into up to seven instances, each with dedicated memory slices and compute units, giving genuine isolation with predictable performance; the trade-offs are coarse fixed profile geometries, reconfiguration overhead, and no oversubscription. MIG suits multi-tenant inference and small-model training where isolation and predictability matter.

Neither helps large distributed training, which wants whole GPUs, often many, and that is where queueing arbitrates. Kueue, the Kubernetes-native batch queueing layer, is the 2026 default answer: teams get ClusterQueues with nominal quotas, jobs wait in queues instead of failing on insufficient resources, gang admission ensures an all-or-nothing start for multi-pod training (avoiding deadlocked partial allocations), idle capacity is borrowable across queues, and higher-priority work can preempt borrowed capacity. Volcano is the older alternative with gang scheduling and fair-share. The complete answer combines all three plus policy: MIG-partition a slice of the cluster for inference and small jobs, time-slice a dev pool, keep whole-GPU nodes for training behind Kueue quotas with priorities and preemption, and publish per-team utilisation dashboards, because quota arguments end when usage data is public.

Answer

The differentiator in this answer is ordering: check the cheap, common, reversible causes before touching the model, and stabilise the business before perfecting the diagnosis. First two minutes, scope: when did the metric break, was it a step change (deploy, config, upstream release) or a drift (data shifting), is it global or confined to a segment, platform, or region? Correlate the break time against the deploy log, model promotion history, and feature pipeline runs; a step change at 02:10 that matches a 02:05 deploy ends the mystery early.

Next, the input side, because most model incidents are data incidents: check feature health on the serving path, null rates, staleness of precomputed features, schema of the freshest upstream data. A stuck feature materialisation job quietly serving three-day-old aggregates, or an upstream service starting to return a new enum value that maps to null, will crater a model with zero model-side changes. Then the model side: was there an alias flip or rollout recently, is the same artifact version actually loaded on all replicas, did a canary get stuck half-rolled?

Then the score path: has the score distribution shifted, or are scores stable while the threshold or a downstream business rule changed? Thresholds and rule engines change more often than models and are routinely forgotten as suspects. Mitigation is a business decision executed by engineering: options in order of preference are rolling back the most recent change (model alias flip, config revert), failing over to the previous known-good model, or invoking the degraded mode the system should already define, a rules-based fallback or conservative default, chosen with the fraud team because fail-open and fail-closed have very different costs at a payments company. Afterwards: a postmortem that asks why monitoring flagged the approval rate rather than the upstream cause, and adds the missing feature-health alert.

Answer

At three models, heroics work: each model has a bespoke pipeline, a named owner who knows its quirks, and deployment is a personal ritual. At fifty, everything artisanal becomes a liability, and the failures are predictable. Knowledge concentrates in individuals, so attrition strands models nobody can retrain.

Bespoke pipelines mean fifty different ways to deploy, so on-call becomes impossible because responders cannot reason about systems they have never seen. Costs blur together, monitoring coverage is whatever each team remembered to build, and a platform-wide question like which models consume this deprecated table takes weeks. The architectural response is a platform layer with paved roads: an opinionated template for the model lifecycle, standard project scaffold, standard pipeline skeleton (the feature/training/inference decomposition), one experiment tracker, one registry, standard serving chassis with logging, metrics, and canary support built in, so that a new model inherits monitoring, lineage, deployment gates, and cost attribution by default rather than by diligence.

Golden-path infrastructure should make the right way the easy way, while still permitting justified exceptions with extra review. Around the paved road: shared feature infrastructure to stop the same aggregates being rebuilt fifty times; a model catalog with mandatory ownership metadata, every model has a team, an SLO tier, and a runbook; tiered operational standards so a revenue-critical fraud model and an internal prioritisation model do not carry identical ceremony; platform-level dashboards for cost, drift coverage, and staleness across the fleet; and a small platform team measured on the productivity of model teams, not on models of its own. The sequencing matters in the answer: standardise tracking and registry first (cheap, immediate lineage), then serving chassis, then feature store, and migrate the existing three models onto the paved road as its first proof.

Answer

The key move is decomposition: a RAG system is a retrieval component and a generation component, and lumping them into one end-to-end score hides which one failed. Retrieval is evaluated with classical IR metrics against a curated set of queries with labelled relevant documents: recall@k and precision@k, plus ranking quality (MRR, nDCG) since position matters to what the generator attends to. This eval set comes from real production queries, labelled once and maintained, and it is cheap to run on every change to chunking, embeddings, or index configuration, which is exactly when retrieval silently regresses.

Generation given retrieval is judged on three axes that have become the standard vocabulary: faithfulness (is every claim in the answer supported by the retrieved context, the anti-hallucination metric), answer relevance (does it address the question), and context utilisation (did it use what was retrieved or ignore it). These are scored by LLM-as-judge with rubrics, frameworks like Ragas operationalise them, with the usual judge caveats: pin the judge model, calibrate judge scores against periodic human labels, and distrust small deltas. In CI, the frozen eval suite gates changes to prompts, chunking, embedding models, and the generator model itself.

In production, where labels do not exist, monitor proxies on sampled live traffic: retrieval scores and their distribution (a drop in top-k similarity scores signals index staleness or query drift), the rate of I-do-not-know or refusal responses, sampled async faithfulness scoring by a judge model on a budgeted slice of traffic, latency and token cost per stage, and user behaviour signals, thumbs ratings where the product has them, plus rephrase-and-retry rates, which are a strong implicit failure signal. Close the loop operationally: every thumbs-down lands in a triage queue, gets labelled retrieval-failure versus generation-failure versus knowledge-gap, and feeds both the eval set and the document ingestion backlog, because in mature RAG systems the corpus, not the model, is where most fixes live.

# Sampled async faithfulness check on live traffic (budgeted)
async def audit_response(query, contexts, answer):
    verdict = await judge(
        model=PINNED_JUDGE,          # pinned version, temperature 0
        rubric='faithfulness_v5',
        payload={'q': query, 'ctx': contexts, 'a': answer},
    )
    metrics.emit('rag.faithfulness', verdict.score, tags={
        'index_version': INDEX_V, 'prompt_version': PROMPT_V,
    })
    if verdict.score < 0.5:
        triage_queue.push(query, contexts, answer, verdict.reasons)

# Retrieval health, label-free, every request:
#  - top_k_max_sim, top_k_mean_sim distributions (index drift)
#  - refusal / no-answer rate by intent
#  - rephrase-within-2-min rate (implicit failure signal)