Kubernetes Interview Questions and Answers

Answer

Because Kubernetes needed an abstraction for containers that must live and die together while sharing resources. All containers in a pod share one network namespace (they reach each other on localhost and share one IP), can share volumes, and are always scheduled onto the same node as a unit. Technically this is implemented with an infrastructure container (the pause container) that holds the network namespace open; application containers join it.

This design enables the classic multi-container patterns: a log shipper tailing files the app writes to a shared emptyDir, an adapter translating a legacy protocol, or a proxy like istio-proxy handling mTLS. The scheduler only ever thinks in pods: resource requests are summed across containers and the whole pod is bound to one node. Interviewers usually follow up with two checks.

First, do you know pods are ephemeral: they are never rescheduled or healed in place, controllers like Deployments replace them with new pods that get new IPs, which is exactly why Services exist. Second, do you know when NOT to co-locate: two containers belong in one pod only when they are tightly coupled and scale together; putting a web app and its database in one pod is the canonical wrong answer because they have different scaling and lifecycle needs.

apiVersion: v1
kind: Pod
metadata:
  name: web-with-logger
spec:
  containers:
    - name: app
      image: nginx:1.27
      volumeMounts:
        - name: logs
          mountPath: /var/log/nginx
    - name: log-tailer
      image: busybox:1.36
      command: ['sh', '-c', 'tail -F /logs/access.log']
      volumeMounts:
        - name: logs
          mountPath: /logs
  volumes:
    - name: logs
      emptyDir: {}

Answer

This is the single most common architecture question because it touches every control plane component. kubectl sends the manifest to the kube-apiserver, which authenticates you (certs, OIDC token, or service account), authorizes the verb via RBAC, runs admission (mutating webhooks, then validation, then validating webhooks and ValidatingAdmissionPolicies), and persists the Deployment object to etcd. Nothing has been scheduled yet; the API server only stores desired state. The kube-controller-manager runs the deployment controller, which notices a Deployment without a matching ReplicaSet and creates one; the replicaset controller then creates Pod objects with an empty nodeName.

The kube-scheduler watches for unscheduled pods, filters nodes (resource fit, taints, affinity, volume topology), scores the survivors, and writes a Binding, setting nodeName. Now the kubelet on that node sees a pod assigned to it, calls the container runtime through the CRI (containerd or CRI-O; dockershim was removed in 1.24), which pulls the image and starts containers. The CNI plugin wires up the pod network and assigns the IP, kube-proxy programs Services rules, and the kubelet starts running probes and reports status back to the API server. The key insight interviewers want: every step is an independent controller watching the API server and reconciling; no component talks directly to another, and etcd is only ever touched by the API server.

Answer

A ReplicaSet does exactly one thing: keep N pods matching a selector running. It has no concept of change management; if you edit its pod template, existing pods are untouched and only future replacements use the new template. A Deployment wraps ReplicaSets with rollout semantics: when you change the pod template, the deployment controller creates a NEW ReplicaSet with the updated template and progressively scales it up while scaling the old one down, honouring maxSurge and maxUnavailable.

Each template change becomes a revision, which is what makes `kubectl rollout undo` and `kubectl rollout history` possible; old ReplicaSets are retained at zero replicas up to revisionHistoryLimit (default 10). The pod-template-hash label the controller stamps on each ReplicaSet is how pods are tied to their revision. In practice you almost never create ReplicaSets directly; the honest answer to 'when would you touch one' is: during debugging.

You inspect ReplicaSets to see why a rollout is stuck (`kubectl describe rs` shows FailedCreate events from quota or admission rejections), to identify which revision is live, or in rare emergencies to scale an old ReplicaSet manually. Interviewers also like the trap question 'what happens if you delete just the Deployment?': with default foreground/background cascading deletion the ReplicaSets and pods go too, but `kubectl delete --cascade=orphan` leaves them running, which is occasionally used for migrations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 4
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: app
          image: registry.example.com/checkout:v42
          ports:
            - containerPort: 8080

Answer

ClusterIP (the default) allocates a virtual IP reachable only inside the cluster; kube-proxy translates connections to that IP into connections to ready pod endpoints. This is what services use to talk to each other, always via the DNS name, never the IP itself. NodePort builds on ClusterIP by additionally opening the same port (default range 30000-32767) on every node; traffic to any node on that port is forwarded to the service.

It is mostly a building block or a quick way to expose something on bare metal without a load balancer; exposing NodePorts directly to the internet is a review flag. LoadBalancer builds on NodePort and asks the cloud provider (via the cloud controller manager) to provision an external load balancer, an ELB/NLB on AWS or a GCLB on GCP, pointing at the node ports. It is the standard way to expose L4 traffic, but one LB per service gets expensive, which is why teams put an Ingress controller or Gateway behind a single LoadBalancer and route HTTP by host and path.

ExternalName is the odd one out: no proxying at all, just a DNS CNAME to an external hostname, used to give an in-cluster name to an external dependency like an RDS endpoint. Worth volunteering: a headless service (clusterIP: None) is not a fifth type but a ClusterIP variant that skips the virtual IP and returns pod IPs directly in DNS, which StatefulSets and gRPC client-side load balancing depend on.

apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  type: ClusterIP
  selector:
    app: checkout
  ports:
    - name: http
      port: 80        # service port
      targetPort: 8080 # container port
---
apiVersion: v1
kind: Service
metadata:
  name: checkout-lb
spec:
  type: LoadBalancer
  selector:
    app: checkout
  ports:
    - port: 443
      targetPort: 8443

Answer

Labels are key-value pairs meant for identification and grouping; selectors query them. Every controller-to-pod relationship in Kubernetes is label-based: a Service routes to pods whose labels match its selector, a Deployment owns pods via spec.selector.matchLabels, a NetworkPolicy targets pods with podSelector, and PodDisruptionBudgets, topology spread and affinity rules all speak in labels. Because selectors are the glue, label mistakes cause classic outages: change a pod template label without updating the Service selector and traffic silently stops; make two Deployments' selectors overlap and they fight over each other's pods.

Annotations carry non-identifying metadata for humans and tooling: you cannot select on them. Real-world examples include Argo CD sync waves (argocd.argoproj.io/sync-wave), Prometheus scrape hints (prometheus.io/scrape), controller configuration on Services (cloud LB settings), and kubectl's own kubectl.kubernetes.io/last-applied-configuration. Annotations can hold larger values than labels (labels are capped at 63 characters per value and must be alphanumeric-ish; annotations can hold blobs up to the object size limit).

Operationally you should know the recommended label set (app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version) because tools like Helm stamp them, and the kubectl flags: `kubectl get pods -l app=checkout,tier!=canary` for selection and `kubectl label pod web-1 debug=true` to tag a live object. Interviewers use this topic to check whether you have actually operated a cluster or only read about one.

# select with labels
kubectl get pods -l 'app=checkout,env in (prod,staging)'

# label and annotate live objects
kubectl label deployment checkout team=payments
kubectl annotate ingress shop nginx.ingress.kubernetes.io/proxy-body-size=10m

# a Service is just a label query
apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  selector:
    app: checkout   # must match pod labels EXACTLY
  ports:
    - port: 80
      targetPort: 8080

Answer

Namespaces isolate names, policy scope, and accounting; they do not isolate the network, nodes, or the kernel. Inside a namespace, object names must be unique; across namespaces they can repeat, which is what lets every team have a service called 'api'. RBAC Roles and RoleBindings are namespace-scoped, so namespaces are the natural unit for handing a team access to only their own stuff.

ResourceQuota and LimitRange apply per namespace, making them the unit of capacity allocation in shared clusters. What namespaces do NOT do surprises people: by default any pod can reach any pod in any other namespace; cross-namespace traffic flows freely until you write NetworkPolicies. DNS even makes it convenient: 'api' resolves within your namespace, and 'api.payments.svc.cluster.local' reaches another team's service.

Pods from different namespaces also land on the same nodes and share the same kernel, so namespaces are a soft multi-tenancy boundary, not a security boundary; hostile-tenant isolation needs separate clusters or sandboxing layers. Also know which resources are cluster-scoped and live outside namespaces entirely: Nodes, PersistentVolumes, StorageClasses, ClusterRoles, CRDs, and namespaces themselves (`kubectl api-resources --namespaced=false` lists them). One operational gotcha worth mentioning: deleting a namespace deletes everything in it, and namespaces stuck in Terminating (usually a finalizer on some resource whose controller is gone) are a common on-call ticket; the fix is finding and clearing the stuck finalizer, not force-deleting blindly.

Answer

The two resources fail completely differently, and this distinction drives real production behaviour. CPU is compressible: when a container tries to use more CPU than its limit, the kernel's CFS quota mechanism throttles it. By default the kernel enforces the quota over 100ms periods, so a container with a 500m limit gets 50ms of CPU time per 100ms window and then sits in throttled state until the next period.

Nothing is killed; the app just gets slower, which shows up as mysterious latency spikes (p99 blowups on JVM apps during GC are the classic case). Memory is incompressible: exceed the memory limit and the kernel OOM killer terminates the process; the container shows OOMKilled with exit code 137, and the kubelet restarts it per restartPolicy, often producing a CrashLoopBackOff with no application log explaining why. Requests are different from limits: requests are what the scheduler uses to place pods and what your QoS class is derived from; limits are the runtime ceiling.

The 2026 consensus you should be able to defend: always set memory request equal to memory limit (memory overcommit gets nodes OOMing unpredictably), always set CPU requests, and think carefully before setting CPU limits at all, since throttling hurts latency and the request already guarantees fair scheduling. Check throttling with the container_cpu_cfs_throttled_periods_total metric, and OOM kills with `kubectl describe pod` (Last State: OOMKilled).

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
    - name: app
      image: registry.example.com/api:v7
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
        limits:
          memory: 512Mi   # memory request == limit
          # no cpu limit: avoid CFS throttling latency

Answer

Liveness probes answer 'is this process beyond saving?': on failure the kubelet kills and restarts the container. Readiness probes answer 'should this pod receive traffic?': on failure the pod is removed from Service endpoints but keeps running. Startup probes gate the other two during slow boots: until the startup probe succeeds, liveness and readiness are suspended, which is how you accommodate a JVM that takes 90 seconds to warm up without inflating initialDelaySeconds everywhere.

Each has a signature outage. A liveness probe that checks a dependency (like /health hitting the database) turns a database blip into a cluster-wide restart storm: every pod restarts simultaneously, dropping all in-flight traffic and hammering the recovering database with cold-start connection floods. Liveness checks must test only the process itself.

A readiness probe that also checks a shared dependency takes every pod out of the endpoints at once, producing a total outage from a partial degradation; sometimes serving degraded responses beats serving nothing. Missing startup probes on slow apps cause boot loops: liveness fires before the app finishes initializing, the kubelet restarts it, and it never escapes. The tuning knobs are periodSeconds, timeoutSeconds, failureThreshold and successThreshold; effective detection time is roughly periodSeconds times failureThreshold. Also know the three mechanisms: httpGet, tcpSocket, exec (plus grpc, which is stable and avoids packing grpc_health_probe binaries into images).

containers:
  - name: app
    image: registry.example.com/api:v7
    startupProbe:            # tolerate up to 5 min of boot
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 5
      failureThreshold: 60
    livenessProbe:           # process-only check, no dependencies
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:          # may consider critical dependencies
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5
      failureThreshold: 2

Answer

Both hold configuration decoupled from images; the difference is intent and handling, not strength. A Secret's data values are base64-encoded purely so binary content survives JSON transport; base64 is trivially reversible and provides zero confidentiality. What actually distinguishes Secrets: they can be encrypted at rest in etcd (only if the cluster enables an EncryptionConfiguration; by default they sit in etcd in plain base64), kubelets fetch them only for pods that mount them and hold them in tmpfs rather than on disk, and RBAC lets you grant ConfigMap access without Secret access.

Consumption is identical for both: as environment variables (env with valueFrom, or envFrom to inject all keys) or as files via a volume mount. The operationally important difference between those two paths: volume-mounted ConfigMaps and Secrets are updated in the running pod when the object changes (the kubelet syncs them periodically, typically within about a minute), while environment variables are frozen at container start and require a restart to pick up changes. Many teams therefore mount config as files and have the app watch them, or roll pods deliberately with a checksum annotation on the pod template (the standard Helm trick) so config changes trigger a rollout.

Also know immutable: true, which marks a ConfigMap or Secret unchangeable, protects against accidental edits, and reduces kubelet watch load in large clusters. In interviews, the phrase they are listening for is 'base64 is encoding, not encryption' followed by how you actually protect Secrets: etcd encryption, RBAC, and an external manager.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
immutable: true
data:
  LOG_LEVEL: info
  feature-flags.yaml: |
    newCheckout: true
---
apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
    - name: app
      image: registry.example.com/api:v7
      envFrom:
        - configMapRef: { name: api-config }
      volumeMounts:
        - name: flags
          mountPath: /etc/flags
          readOnly: true
  volumes:
    - name: flags
      configMap: { name: api-config }

Answer

kubectl create is imperative: it POSTs the object and fails with AlreadyExists if it is present. kubectl replace is a full PUT: it overwrites the entire live object with your file, requires the object to exist, and will happily wipe out fields another controller set (like an HPA-managed replicas count) because whatever is not in your file is gone. kubectl apply is declarative: it computes a patch so that fields you do not mention are left alone, which is why it coexists with HPAs and admission mutations. Classic client-side apply does a three-way merge between your file, the live object, and the previous applied state stored in the kubectl.kubernetes.io/last-applied-configuration annotation. Server-side apply (kubectl apply --server-side) moves the merge into the API server and replaces the annotation with managedFields, which track which manager owns each field; conflicts between managers become explicit errors instead of silent overwrites, which matters when kubectl, a GitOps controller and an operator all touch the same object.

Supporting commands you should name: kubectl diff shows what apply would change before you run it, and --dry-run=server validates against real admission logic (schema, webhooks, quotas) without persisting, strictly better than --dry-run=client which only checks locally. The interview trap: mixing create/edit with apply causes drift because last-applied-configuration goes stale; pick declarative apply (usually via GitOps) and stick to it. Deleting with `kubectl delete -f` plus re-apply is not equivalent to replace: delete kills the pods, replace does not.

# preview then apply declaratively
kubectl diff -f deploy.yaml
kubectl apply -f deploy.yaml

# validate against real admission chain without persisting
kubectl apply -f deploy.yaml --dry-run=server

# server-side apply with explicit field ownership
kubectl apply --server-side --field-manager=ci-pipeline -f deploy.yaml

# see who owns which fields
kubectl get deploy checkout -o yaml --show-managed-fields | less

Answer

The kubelet is the node agent, and it is the only Kubernetes component that actually starts containers. Its responsibilities: watch the API server for pods bound to its node, drive the container runtime through the CRI gRPC interface (containerd or CRI-O) to pull images and manage container lifecycles, mount volumes (delegating to CSI drivers), execute liveness/readiness/startup probes, enforce the pod's cgroup limits, collect container stats through its embedded cAdvisor, run the node-pressure eviction logic when memory or disk runs low, and report pod and node status back to the API server, including the heartbeat via a Lease object in the kube-node-lease namespace. What the kubelet does not do is equally examinable: it does not schedule (it only runs what is bound to it), and it does not program Service routing (kube-proxy) or pod networking (the CNI plugin, though kubelet invokes it).

Static pods are the bootstrap trick: the kubelet can run pods defined as manifest files in a directory (conventionally /etc/kubernetes/manifests, set by staticPodPath in the kubelet config) with no API server involved at all. The kubelet then creates read-only mirror pods in the API server so the static pods are visible to kubectl; deleting a mirror pod does nothing because the file is the source of truth. This is exactly how kubeadm runs the control plane itself: kube-apiserver, controller-manager, scheduler and etcd are static pods, solving the chicken-and-egg problem of who runs the components that run everything else. Editing those manifest files live-restarts the control plane, which is both the upgrade mechanism and a classic way to break a cluster.

Answer

CoreDNS (the default DNS server since 1.13) runs as a Deployment behind the kube-dns Service, and every pod's /etc/resolv.conf is pointed at that Service IP by the kubelet. Services get A/AAAA records of the form service.namespace.svc.cluster.local; pods in the same namespace can use the short name 'api', and cross-namespace calls use 'api.payments' or the FQDN. This works through search domains: resolv.conf lists namespace.svc.cluster.local, svc.cluster.local, and cluster.local, so short names are expanded until something resolves.

The ndots:5 problem falls out of this: resolv.conf sets ndots:5, meaning any name with fewer than five dots is treated as relative and tried against every search domain BEFORE being tried as an absolute name. So a pod resolving api.stripe.com (two dots) first queries api.stripe.com.payments.svc.cluster.local, api.stripe.com.svc.cluster.local, and api.stripe.com.cluster.local, all guaranteed NXDOMAIN, quadrupling DNS load and adding latency to every external call. Fixes: use a trailing dot (api.stripe.com.) to force absolute resolution, or set dnsConfig options ndots to 1 or 2 on latency-sensitive pods.

At scale, add NodeLocal DNSCache, a DaemonSet that runs a caching resolver on each node, cutting CoreDNS load and the conntrack races that cause intermittent 5-second DNS timeouts (a famous production issue). Debugging toolkit: `kubectl exec -it pod -- cat /etc/resolv.conf`, a dnsutils pod for dig/nslookup, and checking CoreDNS logs after enabling the log plugin in its Corefile ConfigMap.

apiVersion: v1
kind: Pod
metadata:
  name: external-caller
spec:
  dnsConfig:
    options:
      - name: ndots
        value: '1'   # treat dotted names as absolute
  containers:
    - name: app
      image: registry.example.com/worker:v3

# debug DNS from inside the cluster
# kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/agnhost:2.47 -- sleep infinity
# kubectl exec -it dnsutils -- dig api.payments.svc.cluster.local

Answer

When the pod template changes, the deployment controller creates a new ReplicaSet and walks two counters toward the target: maxSurge caps how many pods can exist ABOVE the desired replica count during the transition (absolute number or percentage, default 25%), and maxUnavailable caps how many can be missing from the desired count (default 25%). With replicas=4, surge=1, unavailable=0, the controller creates one new pod, waits for it to become Ready (this is where readiness probes gate the rollout), then scales the old ReplicaSet down by one, repeating until done: a strictly capacity-safe rollout at the price of speed and one pod of headroom. maxUnavailable=0 with maxSurge=0 is rejected because progress would be impossible. Readiness is the safety interlock: if new pods never pass their readiness probe, the rollout stalls rather than replacing healthy pods, and after progressDeadlineSeconds (default 600) the Deployment reports ProgressDeadlineExceeded in its conditions; nothing is rolled back automatically, it just stops and waits for a human or a tool like Argo Rollouts.

Also know minReadySeconds: a pod must stay Ready that many seconds before counting as available, a cheap guard against pods that pass one probe then crash. The command set: `kubectl rollout status deployment/checkout` to block until done, `kubectl rollout history` to list revisions with their change-cause, `kubectl rollout undo --to-revision=3` to roll back (which is itself just another rolling update to the old template), and `kubectl rollout restart` to bounce pods without a template change (it stamps a restartedAt annotation). The Recreate strategy, which kills all pods before starting new ones, remains correct for singleton workloads holding exclusive locks or RWO volumes.

spec:
  replicas: 4
  minReadySeconds: 10
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
---
# drive and inspect the rollout
# kubectl set image deploy/checkout app=registry.example.com/checkout:v43
# kubectl rollout status deploy/checkout --timeout=5m
# kubectl rollout history deploy/checkout
# kubectl rollout undo deploy/checkout --to-revision=6

Answer

CrashLoopBackOff means the container starts, exits, and the kubelet is delaying the next restart with exponential backoff (10s doubling up to a 5-minute cap, reset after 10 minutes of stable running). The sequence: first `kubectl logs pod-name --previous`, because the current container may not have logged anything yet; --previous shows the terminated instance's output, which answers most cases (stack trace, missing env var, config parse error). Second, `kubectl describe pod pod-name`: read Last State and its exit code, and the Events at the bottom.

Exit code 137 is SIGKILL, which combined with Reason: OOMKilled means the memory limit is too low (and with no OOMKilled reason often means the liveness probe is killing it, visible as 'Liveness probe failed' events). Exit code 1 is an application error; 126/127 mean the command is not executable or not found, pointing at a bad command/args override or an ENTRYPOINT mismatch; 143 is SIGTERM, usually a probe or eviction. Third, check whether the crash is config-driven: `kubectl get pod -o yaml` to verify env, mounted ConfigMaps/Secrets exist (a missing Secret key shows as CreateContainerConfigError instead, a sibling failure mode worth naming).

Fourth, if logs are empty and the container dies instantly, override the entrypoint to keep it alive and poke around: `kubectl debug pod-name --copy-to=debug-pod --container=app -- sleep 1d`, then exec in and run the binary by hand. Interviewers score you on ordering (logs --previous first) and on mapping exit codes to causes without looking them up; volunteering the liveness-probe-kill case usually seals it.

kubectl logs api-7f9c-x2v --previous --timestamps
kubectl describe pod api-7f9c-x2v | sed -n '/Last State/,/Events/p'
kubectl get events --field-selector involvedObject.name=api-7f9c-x2v \
  --sort-by=.lastTimestamp

# instant-death case: clone the pod with a sleep entrypoint
kubectl debug api-7f9c-x2v --copy-to=api-debug \
  --container=app -- sleep infinity
kubectl exec -it api-debug -- /app/server --validate-config

Answer

ImagePullBackOff is the backoff state after repeated ErrImagePull failures, and the causes fall into four buckets. Auth: the registry is private and the pod has no valid credentials; the fix is a docker-registry Secret referenced via imagePullSecrets on the pod spec or attached to the ServiceAccount (the cleaner pattern, since every pod using that SA inherits it); on EKS/GKE, node IAM roles or Workload Identity often replace this entirely for the cloud registry. Existence: the tag is wrong, the image was never pushed, or the digest was garbage-collected; `kubectl describe pod` shows 'manifest unknown' or 'not found' in events.

Rate limits: Docker Hub's anonymous pull limits regularly break clusters that reference docker.io images without credentials, which is why production clusters mirror through a pull-through cache or use their own registry (ECR, GCR, Harbor). Platform: an amd64-only image on arm64 nodes (Graviton is common in Indian cost-optimised setups) fails with 'no match for platform'; fix with multi-arch manifests via buildx. imagePullPolicy interacts subtly: Always re-checks the registry on every pod start (though layers are cached, only the manifest check hits the network), IfNotPresent uses the node's cached image, and Never requires it pre-loaded. Defaulting is the trap: tags of :latest default to Always, while any other tag defaults to IfNotPresent, so a re-pushed mutable tag like :staging silently keeps running the OLD cached image on nodes that have it, causing per-node version skew. The clean answer is immutable tags or digests (image@sha256:...), which also closes the supply-chain hole of a tag being repointed.

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=ci-bot --docker-password=$TOKEN

# prefer attaching to the ServiceAccount over per-pod
kubectl patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

# pin by digest to make pulls reproducible
spec:
  containers:
    - name: app
      image: registry.example.com/api@sha256:9f8a...
      imagePullPolicy: IfNotPresent

Answer

Init containers run BEFORE app containers, strictly sequentially, and each must exit successfully before the next starts; if one fails, the kubelet retries it according to the pod's restartPolicy (with Never, the whole pod fails). App containers, by contrast, all start together and run concurrently. Because init containers finish and exit, they do not have readiness probes, and their resource requests are accounted differently: the pod's effective request is the maximum of (highest single init container request, sum of app container requests), since inits run one at a time and never alongside the apps.

Canonical uses: waiting for a dependency to be reachable before the app boots (a loop around nc or a real healthcheck, which keeps retry logic out of app code), performing privileged one-time setup like chown-ing a volume or setting a sysctl so the app container itself can stay unprivileged, cloning config or assets into a shared emptyDir, and registering with an external system. Database schema migrations in an init container is a debated pattern worth discussing honestly: it works for single-replica apps but races when multiple replicas start simultaneously, so a Job with a rollout dependency (or a migration tool with advisory locks) is safer. The 2026 twist you must know: an init container with restartPolicy: Always is no longer an init container in the old sense; it becomes a native sidecar that starts before the apps and keeps running alongside them, so the initContainers list is now the home of both run-once setup and long-running sidecars, distinguished only by that field.

spec:
  initContainers:
    - name: wait-for-db
      image: busybox:1.36
      command: ['sh', '-c', 'until nc -z postgres 5432; do sleep 2; done']
    - name: fix-perms
      image: busybox:1.36
      command: ['sh', '-c', 'chown -R 1000:1000 /data']
      securityContext: { runAsUser: 0 }
      volumeMounts:
        - { name: data, mountPath: /data }
  containers:
    - name: app
      image: registry.example.com/api:v7
      securityContext: { runAsUser: 1000 }
      volumeMounts:
        - { name: data, mountPath: /data }
  volumes:
    - { name: data, persistentVolumeClaim: { claimName: api-data } }

Answer

emptyDir is scratch space created when the pod is assigned to a node and deleted when the pod is removed (it survives container restarts within the pod, which people forget). Legitimate uses: shared files between containers in a pod (app writes logs, sidecar ships them), checkpoints, caches. Two flags matter: medium: Memory backs it with tmpfs, which is fast but counts against the container memory limit, so a runaway cache can OOMKill your pod through its own volume; sizeLimit caps usage, and exceeding it gets the pod evicted. hostPath mounts a path from the node's filesystem into the pod.

Its legitimate uses are almost exclusively DaemonSets that genuinely need the node: log collectors reading /var/log, CNI and CSI plugins, node monitoring agents reading /proc. For ordinary workloads hostPath is a red flag: it breaks scheduling portability (data lives on one node), and write access to host paths is a container-escape vector, which is why the restricted Pod Security profile forbids it. PVC-backed volumes are the durable option: the pod references a PersistentVolumeClaim, which binds to a PersistentVolume provisioned by a CSI driver (EBS, PD, Azure Disk, NFS, Ceph), so data survives pod deletion and rescheduling.

Use for databases, queues, anything stateful. The follow-up interviewers reach for: what happens to an RWO (ReadWriteOnce) EBS-style volume when the pod moves nodes: the volume must detach and re-attach, which adds failover latency and is why RWO volumes plus multi-replica Deployments do not mix (two pods on different nodes cannot mount it; you use a StatefulSet with per-replica PVCs instead).

Answer

A DaemonSet runs exactly one copy of a pod on every node matching its criteria, and automatically adds pods to new nodes as they join. It is the deployment vehicle for per-node infrastructure: CNI agents (calico-node, cilium), kube-proxy itself, log collectors (fluent-bit, vector), node monitoring (node-exporter, Datadog agent), and CSI node plugins. DaemonSet pods are scheduled by the default scheduler like everything else these days, but the controller adds automatic tolerations for node.kubernetes.io/not-ready and unreachable so daemons stay up on troubled nodes.

To target a subset of nodes, use spec.template.spec.nodeSelector or nodeAffinity, matching node labels like a pool label or kubernetes.io/os. Tainted nodes are the part candidates fumble: taints repel pods without a matching toleration, and control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule, so a monitoring DaemonSet silently skips them unless you add the toleration explicitly; the blunt hammer is tolerating everything with an empty key and operator: Exists, which is exactly what CNI daemonsets do because networking must run everywhere. Updates: DaemonSets support RollingUpdate (with maxUnavailable, default 1, and maxSurge) and OnDelete strategies; rolling a log agent across a 500-node cluster one node at a time is slow, so bumping maxUnavailable or using a percentage is a real operational decision. Also worth naming: since there is one pod per node, there is no replicas field, and kubectl rollout status/undo work on DaemonSets just as they do on Deployments.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels: { app: node-exporter }
  updateStrategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 10% }
  template:
    metadata:
      labels: { app: node-exporter }
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: exporter
          image: quay.io/prometheus/node-exporter:v1.8.2
          volumeMounts:
            - { name: proc, mountPath: /host/proc, readOnly: true }
      volumes:
        - { name: proc, hostPath: { path: /proc } }

Answer

A Job runs pods to completion. Its core knobs: completions (how many successful pods are needed), parallelism (how many run at once), backoffLimit (retry budget before the Job is marked Failed, default 6 with exponential delay), and activeDeadlineSeconds (wall-clock cap on the whole Job, which overrides backoffLimit and kills active pods when exceeded). The pod template's restartPolicy must be Never or OnFailure; with OnFailure the same pod restarts in place (logs of failed attempts are easy to lose), with Never each retry is a fresh pod, which is friendlier for debugging and the common production choice.

Finished Jobs linger and clutter the API unless you set ttlSecondsAfterFinished, which garbage-collects the Job and its pods after the given delay. podFailurePolicy gives finer control, letting you say certain exit codes fail the Job immediately without burning retries while pod disruptions (evictions) do not count against backoffLimit at all, which fixed a long-standing problem of node drains failing batch pipelines. CronJob wraps Job with a schedule in cron syntax (evaluated in the cluster's or the object's configured time zone via the timeZone field, so an IST schedule is spec.timeZone: Asia/Kolkata rather than offset arithmetic). Overlap behaviour is concurrencyPolicy: Allow (default) lets runs overlap, Forbid skips the new run if the previous is still going, Replace kills the old run and starts fresh. startingDeadlineSeconds bounds how late a missed run may start; and if more than 100 scheduled runs are missed with no deadline set, the controller stops scheduling and reports an error, a genuinely obscure fact that impresses interviewers because it bites real clusters after long controller outages.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: settlement-report
spec:
  schedule: '30 1 * * *'
  timeZone: Asia/Kolkata
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 1800
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report
              image: registry.example.com/settlement:v12
              args: ['--date=yesterday']

Answer

The core loop is describe, logs, events, exec, and top. `kubectl describe pod X` is the first stop: it assembles the spec, container states with last termination reasons and exit codes, conditions, and the event stream for that object; most scheduling failures (Insufficient cpu, node affinity mismatches, untolerated taints) are spelled out verbatim in its Events section. `kubectl logs X` with the flags that matter: -c for a specific container, --previous for the crashed instance, -f to follow, --since=10m and --timestamps for incident correlation, and -l app=checkout --prefix to tail a whole deployment's pods at once. `kubectl get events --sort-by=.lastTimestamp -A` gives the cluster-wide narrative; events are retained only about an hour by default, so capture them early in an incident. `kubectl exec -it X -- sh` for in-container inspection, and `kubectl debug` when there is no shell. `kubectl top pods --containers` and `kubectl top nodes` (backed by metrics-server) reveal actual usage versus requests, the fastest way to spot an approaching OOMKill or CPU saturation. Rounding out the kit: `kubectl get X -o yaml` to see the full live object including status and managedFields; `kubectl port-forward svc/api 8080:80` to hit a service from your laptop without exposing it; `kubectl rollout status` for deploy watching; `kubectl auth can-i create pods --as=system:serviceaccount:ci:deployer` for RBAC checks; and `kubectl api-resources` plus `kubectl explain pod.spec.affinity --recursive` as the built-in schema documentation. Senior interviewers often just ask you to narrate debugging a broken service end-to-end and listen for whether these commands come out in a sensible order.

kubectl describe pod api-7f9c-x2v
kubectl logs -l app=api --prefix --since=15m --timestamps
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl top pods --containers -n payments
kubectl port-forward svc/api 8080:80 -n payments
kubectl auth can-i delete pods -n payments \
  --as=system:serviceaccount:payments:ci-deployer
kubectl explain deployment.spec.strategy --recursive

Answer

Three guarantees. Stable identity: pods are named ordinally (kafka-0, kafka-1, kafka-2) and keep those names across rescheduling, unlike Deployment pods' random hashes. Combined with a headless service (mandatory, referenced via spec.serviceName), each pod gets a stable DNS record, kafka-0.kafka-headless.namespace.svc.cluster.local, so peers can address each other by name; this is what makes quorum systems (Kafka controllers, etcd, MongoDB replica sets) configurable at all.

Stable storage: volumeClaimTemplates create a dedicated PVC per replica (data-kafka-0, data-kafka-1), and each pod always remounts ITS claim; kafka-0 rescheduled to another node reattaches the same disk. Deployments can only share one PVC across replicas (RWX) or have none. Ordered operations: by default (podManagementPolicy: OrderedReady) pods are created 0,1,2 with each waiting for the previous to be Ready, scaled down in reverse, and rolled newest-ordinal-first during updates; Parallel relaxes creation ordering when the app does not need it.

The exam-grade caveats: deleting a StatefulSet or scaling it down does NOT delete the PVCs (deliberate data protection; the persistentVolumeClaimRetentionPolicy field, stable in recent releases, lets you opt into deletion on whenDeleted or whenScaled). A pod on a dead node is not replaced automatically until the node object is deleted or the pod is force-deleted, because the controller cannot risk two kafka-0s writing the same disk: at-most-one semantics beat availability. And a failed pod mid-rollout blocks the ordered update entirely, so a bad image can wedge the whole set until you intervene.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  podManagementPolicy: OrderedReady
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  selector:
    matchLabels: { app: kafka }
  template:
    metadata:
      labels: { app: kafka }
    spec:
      containers:
        - name: kafka
          image: apache/kafka:3.8.0
          volumeMounts:
            - { name: data, mountPath: /var/lib/kafka }
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources: { requests: { storage: 100Gi } }

Answer

They answer opposite questions. nodeSelector and node affinity are pod-side attraction: 'this pod may only land on nodes like X'. Taints are node-side repulsion: 'no pod may land here unless it explicitly tolerates this taint'. A toleration does not attract a pod to the tainted node; it merely removes the barrier, which is the distinction interviewers test.

Taints have three effects: NoSchedule (hard bar for new pods), PreferNoSchedule (soft), and NoExecute (also EVICTS already-running pods without the toleration; tolerationSeconds on the toleration bounds how long a pod may stay after the taint appears, which is exactly how node-failure eviction works via the node.kubernetes.io/not-ready and unreachable taints). The canonical combined pattern is a dedicated node pool, say GPU nodes: taint them (gpu=true:NoSchedule) so general workloads stay off, AND put a nodeSelector or affinity on GPU workloads so they land there rather than anywhere. Taint alone lets GPU pods still schedule on cheap CPU nodes; selector alone lets random pods squat on expensive GPU nodes; you need both fences.

Node affinity adds expressiveness over nodeSelector: requiredDuringSchedulingIgnoredDuringExecution with matchExpressions (In, NotIn, Exists, Gt, Lt) and preferredDuringSchedulingIgnoredDuringExecution with weights for soft preferences (prefer spot nodes, fall back to on-demand). The IgnoredDuringExecution suffix is meaningful: existing pods are not evicted when labels change later. Kubernetes itself runs on this machinery: control-plane isolation, the cluster autoscaler's scale-down process, and kubectl cordon (which sets node.spec.unschedulable rather than a taint, a nuance worth knowing) all manipulate these primitives.

# dedicate a GPU pool: fence both directions
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

# pod that belongs there
spec:
  tolerations:
    - key: workload
      operator: Equal
      value: gpu
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: nvidia.com/gpu.present
                operator: In
                values: ['true']
  containers:
    - name: trainer
      image: registry.example.com/trainer:v2
      resources:
        limits: { nvidia.com/gpu: 1 }

Answer

Raw YAML fails at three things Helm addresses: parameterisation (the same app deployed to dev/staging/prod with different replica counts, resources and hostnames means copy-pasted manifests drifting apart), packaging (distributing a complex app like Prometheus with its 40 manifests, CRDs and sane defaults), and lifecycle (upgrading a multi-manifest app atomically and rolling it back when it breaks). A chart is a directory of Go-templated manifests plus a values.yaml of defaults; consumers override values per environment (-f values-prod.yaml or --set). helm install renders templates server-side of your workflow but client-side of the cluster, applies the result, and records the release. Helm 3's architecture answer: Tiller, the in-cluster privileged component of Helm 2, is gone; the helm CLI talks to the API server directly with YOUR kubeconfig credentials, so RBAC applies to the human, closing Helm 2's giant privilege-escalation hole.

Release state (the rendered manifests, values, and revision history) is stored in Secrets named sh.helm.release.v1.<name>.v<revision> in the release namespace, which is why `helm list` needs Secret read access and why you can archaeologically inspect what revision 3 deployed with `helm get manifest myapp --revision 3`. The commands that matter operationally: `helm upgrade --install` (idempotent CI deploys), `--atomic` (auto-rollback on failed upgrade), `--wait` (block until resources are ready), `helm rollback myapp 3`, `helm template` (render locally for review or GitOps pipelines), and `helm diff upgrade` via the diff plugin, the single best pre-deploy safety check. Know also that chart dependencies live in Chart.yaml and land in charts/ via `helm dependency update`.

# idempotent environment deploy
helm upgrade --install checkout ./charts/checkout \
  -n payments --create-namespace \
  -f values.yaml -f values-prod.yaml \
  --set image.tag=v43 \
  --atomic --wait --timeout 5m

# inspect state and history
helm history checkout -n payments
helm get values checkout -n payments --revision 6
helm rollback checkout 6 -n payments

# render without touching the cluster (GitOps / review)
helm template checkout ./charts/checkout -f values-prod.yaml

Answer

Every object has the same skeleton. apiVersion is really group/version: core objects (Pod, Service, ConfigMap, Namespace) live in the legacy core group written as just v1; workloads live in apps/v1 (Deployment, StatefulSet, DaemonSet, ReplicaSet); batch/v1 holds Job and CronJob; networking.k8s.io/v1 holds Ingress and NetworkPolicy; rbac.authorization.k8s.io/v1 holds Roles and bindings; autoscaling/v2 holds the HPA. Version suffixes encode maturity: v1alpha1, v1beta1, v1; beta APIs can and do get removed (the Ingress extensions/v1beta1 removal broke countless clusters in 1.22, the standard cautionary tale for pinning old apiVersions). kind names the type; together with group/version it fully identifies the schema, which `kubectl explain deployment.spec --recursive` documents offline and `kubectl api-resources` enumerates, including short names (deploy, svc, sts) and whether the resource is namespaced. metadata holds name, namespace, labels, annotations, plus system-managed fields: uid, resourceVersion (the optimistic-concurrency token; a stale resourceVersion causes the Conflict errors controllers retry on), generation, ownerReferences (the garbage-collection chain: delete a Deployment and its ReplicaSets and pods go because they carry owner refs), and finalizers (delete blockers; a stuck deletion almost always means a finalizer whose controller is gone). spec is desired state, written by you; status is observed state, written only by controllers through the /status subresource, which is why your applies never fight over it. This spec/status split IS the Kubernetes model: every controller is a loop making status converge to spec, and CRDs extend the system by adding new kinds that follow exactly the same contract.

# explore the API surface without leaving the terminal
kubectl api-resources --namespaced=true | head -20
kubectl api-versions | sort | head
kubectl explain cronjob.spec.jobTemplate --recursive | less

# watch the spec/status split live
kubectl get deployment checkout -o jsonpath='{.metadata.generation}'
kubectl get deployment checkout \
  -o jsonpath='{.status.observedGeneration} {.status.readyReplicas}'

# find what is blocking a deletion
kubectl get namespace stuck-ns -o jsonpath='{.spec.finalizers}'
kubectl get pod stuck-pod -o jsonpath='{.metadata.finalizers}'

Answer

A ClusterIP is a virtual IP: no interface anywhere owns it, no pod listens on it, and you cannot ping it. kube-proxy watches Services and EndpointSlices and programs the node's packet-processing layer so that connections TO the virtual IP are destination-NATed to a real pod IP and port before routing happens. In iptables mode (the long-time default), each service becomes a chain of rules; backend selection is random with per-rule probability (for 3 backends: first rule matches with probability 1/3, second with 1/2 of the remainder, and so on), and conntrack pins the chosen backend for the connection's lifetime. The scaling problem: rules are a flat list evaluated sequentially and updated wholesale, so clusters with tens of thousands of services suffer slow rule syncs and packet-processing overhead.

IPVS mode replaces the per-service chains with kernel IPVS virtual servers using hash tables, giving O(1) lookup and real scheduling algorithms (round-robin, least-connection, source-hash) instead of random choice; it still uses some iptables for auxiliary marking. nftables mode, stable in recent releases, is the modern replacement for the iptables backend with faster incremental rule updates and better performance on large rule sets, and is the direction the project is pushing as distros deprecate legacy iptables. Two behaviours this machinery explains, and interviewers love: long-lived connections (gRPC, HTTP/2, database pools) are balanced only ONCE at connect time, so one backend can end up with all the load, which is why gRPC needs headless services with client-side balancing or a mesh; and externalTrafficPolicy: Local preserves client source IPs by only routing to local pods at the cost of imbalance. Also note eBPF-based CNIs like Cilium can replace kube-proxy entirely.

Answer

Setting clusterIP: None turns off the virtual-IP machinery entirely: kube-proxy programs nothing, and CoreDNS changes behaviour. A normal service's DNS name resolves to its single ClusterIP; a headless service's name resolves to the full set of READY pod IPs directly (multiple A/AAAA records), and clients receive them all and pick. Additionally, when the headless service governs a StatefulSet (via spec.serviceName), each pod gets its own stable per-pod record: kafka-0.kafka-headless.ns.svc.cluster.local resolves to whatever IP kafka-0 currently has.

That indirection is the entire trick: pod IPs change on every reschedule, but the DNS name is stable, so quorum configs can reference peers by name. For gRPC and HTTP/2 the motivation is different: those protocols multiplex everything over one long-lived connection, and kube-proxy balances per-connection, so a normal ClusterIP sends ALL of a client's requests to one backend forever. Pointing the gRPC client at a headless service with a dns:/// target lets its round_robin policy open a connection per backend and spread requests properly. The gotchas that separate practitioners: DNS caching means clients may hold stale pod IPs after a rollout (tune the client's re-resolution; gRPC re-resolves on connection failure); readiness still gates DNS inclusion, but publishNotReadyAddresses: true forces even unready pods into DNS, which bootstrapping quorum systems need (a Kafka pod cannot become ready before discovering peers who are also not ready, a chicken-and-egg problem this field exists to solve); and headless services return EndpointSlice-backed records, so extremely large backend sets can hit DNS response-size limits, pushing you toward client-side EndpointSlice watching instead.

apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
spec:
  clusterIP: None
  publishNotReadyAddresses: true   # peers resolvable during bootstrap
  selector:
    app: kafka
  ports:
    - { name: broker, port: 9092 }
---
# gRPC client target for proper round-robin
# grpc.Dial('dns:///orders-headless.payments.svc.cluster.local:50051',
#   grpc.WithDefaultServiceConfig('{"loadBalancingConfig":[{"round_robin":{}}]}'))

Answer

Ingress (networking.k8s.io/v1) gives you host and path routing to services, but its schema is frozen: anything beyond basic routing (timeouts, rewrites, canary weights, TLS options, rate limits) lives in controller-specific annotations, which is how production Ingress objects ended up with fifteen nginx.ingress.kubernetes.io/* annotations and zero portability between controllers. The Gateway API (gateway.networking.k8s.io/v1, GA since its v1.0 release) is the successor, designed around role separation: GatewayClass (which implementation, owned by the platform), Gateway (a listener deployment: ports, TLS certs, allowed namespaces, owned by cluster operators), and HTTPRoute/GRPCRoute/TCPRoute (routing rules, owned by app teams, attached to Gateways across namespaces via ReferenceGrant). Features that were annotations become typed fields: header matching, traffic weighting for canaries, filters for header manipulation and redirects, all portable across implementations (Envoy Gateway, Cilium, Istio, NGINX Gateway Fabric, and the managed cloud gateways).

The 2026 fact that reframed this debate: the Kubernetes project announced retirement of ingress-nginx, the most-deployed Ingress controller, with maintenance ending in March 2026, forcing the massive installed base to migrate; that announcement turned Gateway API adoption from gradual to urgent, and interviewers now ask about migration plans specifically. A defensible answer: new clusters standardise on Gateway API with an Envoy-based implementation; existing ingress-nginx estates migrate route-by-route (the ingress2gateway tool automates first drafts), prioritising anything internet-facing since unmaintained proxies accumulate CVEs. Mentioning that service meshes are converging on Gateway API for east-west traffic too (GAMMA initiative) signals current knowledge.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-route
  namespace: payments
spec:
  parentRefs:
    - name: shared-gateway
      namespace: infra
  hostnames: ['pay.example.com']
  rules:
    - matches:
        - path: { type: PathPrefix, value: /checkout }
      backendRefs:
        - { name: checkout, port: 80, weight: 90 }
        - { name: checkout-canary, port: 80, weight: 10 }
    - matches:
        - path: { type: PathPrefix, value: /legacy }
      filters:
        - type: RequestRedirect
          requestRedirect: { path: { type: ReplacePrefixMatch, replacePrefixMatch: /v2 } }
      backendRefs:
        - { name: checkout, port: 80 }

Answer

Start from the semantics: NetworkPolicies are allow-lists that activate by selection. A pod selected by no policy allows all traffic; the moment any policy selects it for a direction (ingress or egress), that direction becomes default-deny except for what policies allow. Policies are additive (union of allows), there is no deny rule, and ordering does not exist.

The standard design: first a default-deny policy with an empty podSelector matching every pod in the namespace and policyTypes [Ingress, Egress], flipping the namespace to zero-trust. Then targeted allows: a policy selecting the db pods allowing ingress from pods labelled app=api on port 5432, and a policy selecting api pods allowing egress to the db plus, critically, egress to CoreDNS on UDP and TCP 53, because default-deny egress breaks DNS and every service call with it: the single most common NetworkPolicy self-own. Cross-namespace sources combine namespaceSelector with podSelector in ONE from-entry (two separate entries mean OR, one combined entry means AND, a subtle YAML shape that interviewers deliberately probe).

Selectors match labels, never service names or IPs of services; ipBlock is for true externals only. The enforcement caveat: the API server accepts NetworkPolicy objects regardless, but enforcement is the CNI's job: Calico and Cilium enforce them, flannel alone does not, so policies can be silently decorative, worth verifying with a connectivity test (a wget from a debug pod that SHOULD be blocked). For requirements beyond the model (L7 rules, FQDN egress allow-lists like 'only api.razorpay.com'), name CiliumNetworkPolicy as the extension, since vanilla NetworkPolicy is L3/L4 only.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-to-db
  namespace: payments
spec:
  podSelector:
    matchLabels: { app: postgres }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: api }
      ports:
        - { protocol: TCP, port: 5432 }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Answer

The autoscaling/v2 HPA runs a control loop (every 15 seconds by default) computing: desiredReplicas = ceil(currentReplicas x currentMetric / targetMetric). With 4 replicas at 80% average CPU utilisation against a 50% target, it computes ceil(4 x 80/50) = 7. Details that distinguish real answers: for resource metrics, utilisation is measured against REQUESTS, not limits, so wrong requests silently mean wrong autoscaling; pods without requests break CPU-based HPA entirely; unready pods and pods missing metrics are handled conservatively (scale-up assumes they use 0%, scale-down assumes 100%, biasing against both overreaction and premature shrink); and a tolerance band (about 10%) around the ratio suppresses changes for small deviations.

Metrics arrive via three APIs: metrics.k8s.io from metrics-server (CPU/memory), custom.metrics.k8s.io (in-cluster metrics like requests-per-second via prometheus-adapter), and external.metrics.k8s.io (queue depths from cloud services). CPU is a lagging, often misleading signal for request-driven services; scaling on RPS or queue lag tracks demand more directly, a point worth volunteering. Flapping control lives in spec.behavior: scaleDown.stabilizationWindowSeconds (default 300) makes scale-down use the HIGHEST desired count over the window, so brief dips do not shed pods; policies cap the rate (at most N pods or X% per period) for both directions; and selectPolicy picks between competing policies.

A burst-tolerant profile: aggressive scale-up (100% per 15s, no stabilisation) with conservative scale-down (10% per minute, 600s window). Finally, never set spec.replicas on a Deployment managed by an HPA; every apply would fight the autoscaler, which is exactly the field-ownership problem server-side apply exposes.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 3
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }
    - type: Pods
      pods:
        metric: { name: http_requests_per_second }
        target: { type: AverageValue, averageValue: '150' }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - { type: Percent, value: 100, periodSeconds: 15 }
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - { type: Percent, value: 10, periodSeconds: 60 }

Answer

KEDA (Kubernetes Event-Driven Autoscaling, a CNCF graduated project) solves two things the HPA cannot: scaling on external event sources without building a custom metrics adapter, and scaling to zero. You declare a ScaledObject targeting your Deployment with one or more triggers from KEDA's scaler catalogue: Kafka consumer lag, RabbitMQ or SQS queue depth, Prometheus query results, cron schedules, Postgres query results, Azure Service Bus, and dozens more. Under the hood KEDA does not replace the HPA; it manufactures one.

The KEDA operator registers itself as the external metrics API server, translates each trigger into an external metric, and creates an HPA against it, so all the HPA behavior machinery still applies. The special part is the zero boundary, which the HPA cannot cross (minReplicas must be at least 1 for metric-based scaling to function, since zero pods produce zero metrics): KEDA itself handles 0-to-1 activation by polling the event source directly (activationThreshold decides when), then hands 1-to-N scaling to the generated HPA; on idle (cooldownPeriod, default 300s, with no events) KEDA scales back to zero. Use cases where this is the right tool: queue consumers that should track backlog rather than CPU (scale workers so lag stays near zero), scheduled batch capacity (cron trigger pre-warms workers before a known daily spike, useful for IST-business-hours patterns), and dev environments scaled to zero overnight for cost.

What KEDA does not do: HTTP request-based scale-to-zero for synchronous traffic needs the separate KEDA HTTP add-on or Knative, because something must buffer requests while the pod cold-starts; and it does not size pods (that is VPA) or nodes (autoscaler/Karpenter). Sane pairing: KEDA scales pods on lag, Karpenter scales nodes underneath.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: settlement-worker
  namespace: payments
spec:
  scaleTargetRef:
    name: settlement-worker
  minReplicaCount: 0
  maxReplicaCount: 50
  cooldownPeriod: 120
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-headless.kafka:9092
        consumerGroup: settlement
        topic: transactions
        lagThreshold: '500'
        activationLagThreshold: '10'
    - type: cron   # pre-warm before the 9 AM IST spike
      metadata:
        timezone: Asia/Kolkata
        start: 45 8 * * 1-5
        end: 30 21 * * 1-5
        desiredReplicas: '5'

Answer

Both add nodes when pods are Pending for lack of capacity and remove underused nodes, but their models differ fundamentally. The Cluster Autoscaler (CA) works through node GROUPS (ASGs on AWS, MIGs on GCP, node pools on managed offerings): every node in a group is identical, and CA simulates the scheduler against each group's template to decide which group to grow. Consequences: you pre-define the instance shapes, a pod that fits no group's template stays Pending forever, and heterogeneous needs mean maintaining many groups.

Scale-down is conservative: a node must sit below a utilisation threshold with all its pods relocatable before removal. CA is mature, cloud-neutral, and the default on GKE and AKS. Karpenter (originated at AWS, now a CNCF project, with the v1 NodePool/EC2NodeClass APIs) discards groups entirely: it watches Pending pods, computes the cheapest instance types satisfying their aggregate requirements from potentially hundreds of types, and provisions nodes directly, typically much faster than ASG round-trips.

NodePools declare constraints (architectures, capacity types, size ranges) rather than fixed shapes, so one pool serves diverse workloads: bin-packing improves and spot diversification is automatic (Karpenter picks from deep spot pools and handles interruption notices). Its consolidation feature actively replaces underutilised or pricier nodes with cheaper arrangements, deliberately churning nodes for cost, so PDBs and graceful shutdown hygiene become mandatory. When each wins: Karpenter on EKS for cost-sensitive, bursty, heterogeneous workloads (the typical Indian startup profile, where spot plus consolidation routinely cuts compute spend dramatically); CA where you need its maturity on GKE/AKS, want strictly predictable node shapes for compliance, or run on-prem where Karpenter lacks a provider. Interviewers often follow with 'what breaks with aggressive consolidation': stateful pods on RWO volumes and long-lived connections, mitigated with do-not-disrupt annotations and consolidation policies.

Answer

Four kinds compose the model: Role (namespaced rules), ClusterRole (cluster-scoped rules), RoleBinding (grants within one namespace), ClusterRoleBinding (grants everywhere). Rules are triples of apiGroups, resources, verbs (get, list, watch, create, update, patch, delete, plus deletecollection); there are no deny rules, everything is additive allow, so 'revoking' means not granting. For the scenario: create a Role in team-payments granting write on the workload surface (deployments, statefulsets, services, configmaps, plus pods for debugging with pods/log and pods/exec subresources listed explicitly, since subresources need their own entries) and bind it to the team's group via RoleBinding.

For cluster-wide read, bind the built-in view ClusterRole... but that is namespaced-read; to also read cluster-scoped objects like nodes, create a small ClusterRole (get/list/watch on nodes, namespaces, persistentvolumes) and a ClusterRoleBinding. The trick worth naming: a ClusterRole bound by a RoleBinding grants its rules only within that binding's namespace, the standard pattern for defining a permission set once and granting it per-namespace without duplicating Roles. Details that show production experience: bind to groups from your IdP (OIDC claims) rather than individual users; be stingy with pods/exec (it is effectively code execution in every container the binding covers); never grant escalate, bind, or impersonate casually (each is a privilege-escalation primitive); remember Secrets read is root-equivalent in most clusters (service account tokens live there); and the aggregationRule on ClusterRoles lets operators extend built-in roles like view with CRD permissions via labels. Verification commands: `kubectl auth can-i create deployments -n team-payments --as-group=payments-devs --as=anyone` and `kubectl auth can-i --list -n team-payments` to dump an identity's effective permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-deployer
  namespace: team-payments
rules:
  - apiGroups: ['apps']
    resources: ['deployments', 'statefulsets']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch']
  - apiGroups: ['']
    resources: ['services', 'configmaps', 'pods', 'pods/log']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch']
  - apiGroups: ['']
    resources: ['pods/exec']
    verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-devs-deploy
  namespace: team-payments
subjects:
  - kind: Group
    name: payments-devs      # OIDC group claim
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: app-deployer
  apiGroup: rbac.authorization.k8s.io

Answer

Before 1.24, creating a ServiceAccount auto-generated a Secret containing a non-expiring token, and that Secret was mounted into pods: long-lived, never rotated, valid until manually deleted, and a beloved target in every cluster compromise. Since 1.24, no Secret is created. Pods instead receive a projected token via the TokenRequest API: bound to the specific pod (invalid once the pod is gone), audience-scoped, expiring (kubelet requests roughly hour-long tokens and refreshes them transparently), and mounted at the same well-known path, so most in-cluster clients (client-go, kubectl in pods) noticed nothing.

Applications must re-read the token file periodically instead of caching it forever, the one migration gotcha. When you genuinely need a long-lived token (legacy external system authenticating to the cluster), you explicitly create a Secret of type kubernetes.io/service-account-token with the right annotation, or better, mint a bounded one with `kubectl create token my-sa --duration=24h`. Workload identity answers the next question: how do pods authenticate to CLOUD APIs without stuffing cloud keys into Secrets?

The mechanism is OIDC federation: the cluster's API server acts as an OIDC issuer for service account tokens, the cloud IAM is configured to trust that issuer, and a pod exchanges its projected token (with the cloud's audience) for cloud credentials. On EKS this is IRSA (an annotation on the ServiceAccount naming an IAM role, with EKS Pod Identity as the newer variant), on GKE it is Workload Identity Federation binding KSAs to Google service accounts, on AKS it is Workload Identity via federated credentials. The interview payoff line: no static cloud keys exist anywhere, tokens are short-lived and pod-bound, and permissions are scoped per-ServiceAccount rather than per-node, killing both the leaked-key and the every-pod-inherits-node-role failure modes.

Answer

Layer one: encryption at rest. The API server's EncryptionConfiguration file specifies providers per resource type; configuring aescbc or, properly, kms encrypts Secrets before they hit etcd. KMS v2 (stable in recent releases) envelope-encrypts with a cloud KMS or HSM-backed key, so a stolen etcd snapshot yields ciphertext.

After enabling it, existing Secrets must be rewritten to pick up encryption: `kubectl get secrets -A -o json | kubectl replace -f -`. Managed control planes (EKS, GKE) offer this as a checkbox, but it defaults OFF on self-managed clusters, a routine audit finding. Layer two: keep secrets out of Git.

Plain Secret manifests in a repo are the most common leak vector in GitOps shops. Options: Sealed Secrets (Bitnami controller; you commit SealedSecret objects encrypted to the controller's public key, only the in-cluster controller can decrypt, simple but key rotation and cluster recovery need care) or, the 2026 default, the External Secrets Operator: Git holds an ExternalSecret object that only REFERENCES paths in an external manager (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault), and the operator syncs real Secrets into the cluster, refreshing on an interval so rotation propagates automatically. Authentication to the store uses workload identity, closing the loop without static credentials.

Layer three: reduce blast radius: RBAC that treats Secret read as privileged, one ServiceAccount per workload, immutable Secrets where possible, and audit logging on secret access. For the strictest cases, the Vault Agent injector or the Secrets Store CSI driver delivers secrets as in-memory files without creating Secret objects at all, at the cost of coupling pod startup to the external store's availability, a trade-off worth stating explicitly in interviews.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payment-gateway-creds
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payment-gateway-creds   # Secret the operator maintains
    creationPolicy: Owner
  data:
    - secretKey: api_key
      remoteRef:
        key: prod/payments/gateway
        property: api_key
    - secretKey: webhook_secret
      remoteRef:
        key: prod/payments/gateway
        property: webhook_secret

Answer

PodSecurityPolicy was removed in 1.25 after years of deprecation; its replacement is Pod Security Admission (PSA), a built-in admission controller configured through namespace LABELS rather than API objects. PSA evaluates pods against three fixed profiles defined by the Pod Security Standards: privileged (no restrictions), baseline (blocks known escalation paths: hostNetwork, hostPID, privileged containers, hostPath, dangerous capabilities), and restricted (baseline plus hardening requirements: runAsNonRoot, seccompProfile RuntimeDefault, allowPrivilegeEscalation false, drop ALL capabilities). Each namespace gets up to three mode labels: enforce (violations are rejected), audit (violations recorded in audit logs), and warn (violations return warnings to the client), each optionally pinned to a profile version like v1.31 so upgrades do not silently change semantics.

The migration play interviewers want to hear: never flip enforce first. Label namespaces with warn=restricted and audit=restricted while enforce stays at baseline (or unset), run for a sprint, and collect violations from client warnings and audit logs; fix the workloads (usually adding securityContext blocks, removing hostPath mounts, rebuilding images to run as non-root); then raise enforce. Exemptions for genuinely privileged infrastructure (CNI, CSI, monitoring agents) are handled by keeping their namespaces at privileged, since PSA has no per-pod exception mechanism: that granularity gap is deliberate, and it is why PSA is positioned as the floor while policy engines (Kyverno, OpenPolicyAgent/Gatekeeper, or CEL-based ValidatingAdmissionPolicies) layer custom rules on top: require image registries, forbid :latest tags, enforce labels. A crisp closing distinction: PSA validates PODS only and cannot mutate; Kyverno-class engines cover any resource and can mutate defaults into place.

# staged rollout on an existing namespace
kubectl label namespace payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version=v1.31 \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

# see what would break before enforcing restricted
kubectl label --dry-run=server --overwrite namespace payments \
  pod-security.kubernetes.io/enforce=restricted

# after fixes, raise the floor
kubectl label --overwrite namespace payments \
  pod-security.kubernetes.io/enforce=restricted

Answer

The restricted-profile-aligned baseline, field by field. runAsNonRoot: true makes the kubelet refuse to start a container whose effective user is root; container escapes and file permission abuse both get dramatically harder when the process is UID 1000, and images must be built accordingly (a USER directive; distroless and chainguard-style images ship non-root variants). runAsUser/runAsGroup pin the IDs explicitly when the image metadata is untrustworthy. allowPrivilegeEscalation: false sets the kernel no_new_privs flag, so setuid binaries like sudo cannot elevate even if present in the image. capabilities.drop: [ALL] strips every Linux capability; the only commonly legitimate add-back is NET_BIND_SERVICE for binding ports below 1024, and even that is avoidable by listening on 8080. readOnlyRootFilesystem: true mounts the container root read-only, turning malware persistence and config tampering into instant crashes; apps that need scratch space get explicit writable emptyDir mounts at /tmp or a cache path, which also documents exactly where writes happen. seccompProfile.type: RuntimeDefault applies the runtime's syscall filter blocking a few dozen exotic syscalls that underpin many kernel exploits, at effectively zero performance cost; it went from opt-in to expected-everywhere. At the pod level, fsGroup makes mounted volumes group-writable to the pod's group (the standard fix for permission-denied on PVCs with non-root users). What you leave OFF matters equally: privileged: true (full host access), hostNetwork/hostPID/hostIPC (namespace sharing with the node), and hostPath mounts are for node-level infrastructure only.

In interviews, walking this list with the attack each setting blocks, rather than reciting field names, is the difference between a hardening answer and a checklist answer. Admission policy (PSA restricted, or Kyverno mutate-to-default) is what keeps the settings universal instead of aspirational.

spec:
  securityContext:            # pod level
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: registry.example.com/api:v7
      securityContext:        # container level
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ['ALL']
      volumeMounts:
        - { name: tmp, mountPath: /tmp }
  volumes:
    - { name: tmp, emptyDir: { sizeLimit: 256Mi } }

Answer

Pod anti-affinity is binary: requiredDuringScheduling anti-affinity on kubernetes.io/hostname means no two replicas share a node, full stop. That fails ugly at scale: 6 replicas on a 5-node cluster leaves one pod permanently Pending, and the preferred variant degrades to no guarantee at all. It is also computationally expensive for the scheduler on large clusters. topologySpreadConstraints instead express a tolerated IMBALANCE: maxSkew is the maximum allowed difference between the most and least loaded topology domain, so maxSkew: 1 across zones keeps replica counts within one of each other (4/3/3 for ten replicas across three zones) while never blocking scheduling outright the way strict anti-affinity does.

Fields that matter: topologyKey picks the domain (topology.kubernetes.io/zone for AZ spread, kubernetes.io/hostname for node spread; both labels are set automatically by cloud providers); whenUnsatisfiable is DoNotSchedule (hard) or ScheduleAnyway (soft, skew minimised best-effort); labelSelector scopes which pods are counted; and minDomains forces spreading across at least N domains even when fewer currently have nodes, nudging the cluster autoscaler to open a new zone. The production combo worth reciting: zone-level spread with maxSkew 1 DoNotSchedule, plus hostname-level spread with ScheduleAnyway, plus a PodDisruptionBudget: the spread makes a zone outage lose only about a third of capacity, and the PDB stops voluntary drains from finishing the job. Gotchas that show real experience: constraints are evaluated at SCHEDULING time only, so scale-downs and node failures can skew the distribution afterwards (the descheduler project rebalances if you need it); and the counted pods include terminating ones in older versions, which briefly distorts skew during rollouts.

spec:
  replicas: 9
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          minDomains: 3
          labelSelector:
            matchLabels: { app: checkout }
        - maxSkew: 2
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: checkout }

Answer

A PriorityClass is a cluster-scoped object mapping a name to an integer; pods reference it via priorityClassName and the admission controller stamps the resolved number into spec.priority. Two effects follow. Queueing: higher-priority Pending pods are considered by the scheduler first.

Preemption: when a high-priority pod fits nowhere, the scheduler looks for a node where evicting lower-priority pods would make room, then evicts them (gracefully: preempted pods get their full terminationGracePeriodSeconds) and reserves the spot via the nominatedNodeName field. Kubernetes itself relies on this: system-cluster-critical and system-node-critical are built-in classes (values around two billion) protecting CoreDNS, CNI agents and kube-proxy from being starved out by application pods, and a cluster where someone forgot priorities on system add-ons behaves horribly under pressure. Recommended production tiering: a high class for revenue-path services, a default (via a PriorityClass with globalDefault: true) for everything else, and a low or negative class for batch and best-effort work that should yield capacity first.

The hurt scenarios interviewers dig for: preemption cascades, where a wave of high-priority scale-up evicts swathes of batch work repeatedly, wasting all its partial progress (mitigate with preemptionPolicy: Never on classes that should queue-jump without evicting, and with podFailurePolicy on Jobs so preemption does not burn retry budgets); PDB interaction, where the scheduler TRIES to respect PodDisruptionBudgets when choosing victims but will violate them if there is no alternative, so a PDB is not armour against preemption, a subtlety most candidates get wrong; and priority inversion via missing classes, where forgetting priorityClassName on a critical service leaves it at 0, evictable by anything labelled important. Also distinguish preemption (scheduler making room) from node-pressure eviction (kubelet reclaiming resources): different mechanisms, different signals, different ordering rules.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: revenue-critical
value: 100000
preemptionPolicy: PreemptLowerPriority
description: Checkout and payment path services
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-preemptible
value: -100
preemptionPolicy: Never   # may be preempted, never preempts
description: Reprocessing and analytics batch jobs
---
# in the deployment pod template
spec:
  priorityClassName: revenue-critical

Answer

Drain is two operations. First it cordons: sets node.spec.unschedulable, so nothing new lands (visible as SchedulingDisabled in kubectl get nodes). Then it evicts every evictable pod using the Eviction API (a POST to pods/<name>/eviction), which is crucially different from deletion: an eviction request is CHECKED against PodDisruptionBudgets and refused with a 429 if the disruption would violate one, whereas kubectl delete pod bypasses PDBs entirely. kubectl drain retries refused evictions until they pass or you time out.

A PDB (policy/v1) selects pods by label and declares either minAvailable or maxUnavailable (integers or percentages); with 3 replicas and minAvailable: 2, only one pod can be down voluntarily at a time, so a rolling node upgrade proceeds node by node at the pace your replicas can tolerate. The flags you end up needing: --ignore-daemonsets (DaemonSet pods cannot be evicted meaningfully since the controller would recreate them; drain skips them), --delete-emptydir-data (acknowledges scratch data loss), --force (for naked pods with no controller, which would otherwise never come back), and --timeout. The failure modes that make this an interview favourite: a PDB with minAvailable equal to the replica count makes every drain hang forever (zero disruption allowed); a PDB selecting pods that are ALREADY unhealthy blocks drains pointlessly, which unhealthyPodEvictionPolicy: AlwaysAllow on the PDB fixes by letting non-Ready pods be evicted regardless; and single-replica workloads with a PDB block drains by construction, the honest fix being two replicas, not deleting the PDB. Finally, distinguish voluntary disruptions (drains, evictions: PDB-protected) from involuntary ones (node crash, OOM kill, preemption edge cases: not PDB-protected); PDBs are a maintenance contract, not an availability guarantee.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
  namespace: payments
spec:
  minAvailable: 2
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels: { app: checkout }
---
# node maintenance flow
# kubectl cordon node-7
# kubectl drain node-7 --ignore-daemonsets \
#   --delete-emptydir-data --timeout=10m
# ...maintain, then:
# kubectl uncordon node-7

Answer

When a pod is deleted (rollout, drain, scale-down), the API server marks it Terminating and two things start IN PARALLEL: the kubelet sends the container's main process SIGTERM (running the preStop hook first, if defined), and the endpoints machinery removes the pod from EndpointSlices, which kube-proxy nodes and load balancers then act on. That parallelism is the race: for some window (typically hundreds of milliseconds to seconds, longer with slow cloud LB deregistration), traffic still arrives at a pod that has already received SIGTERM. If the app exits immediately on SIGTERM, every request in that window gets a connection refused, surfacing as 502/504 spikes on every single deploy: one of the most common production complaints attributed to 'flaky Kubernetes' that is actually a shutdown-sequence bug.

The fix has two halves. App side: on SIGTERM, stop accepting NEW connections but finish in-flight requests, then exit (most frameworks have a graceful shutdown mode; also mind that PID 1 in a container must actually receive signals: shell-form ENTRYPOINT wrapping your binary in sh eats SIGTERM, the classic gotcha fixed by exec-form or tini). Platform side: a preStop hook that simply sleeps 5-10 seconds delays SIGTERM until endpoint removal has propagated, so no new traffic arrives by the time the app begins shutdown; recent versions provide a built-in sleep action for lifecycle hooks so this no longer requires a shell in the image.

The overall clock is terminationGracePeriodSeconds (default 30, counted from deletion INCLUDING preStop time): when it expires, SIGKILL, no appeal. Long-running workloads (queue consumers mid-batch, video processing) legitimately raise it to minutes. Verification: watch for pods stuck Terminating (grace period too high plus ignored SIGTERM) and measure deploy-time 5xx before and after adding the preStop sleep; the delta is usually the entire problem.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: app
      image: registry.example.com/api:v7
      lifecycle:
        preStop:
          sleep:            # built-in action, no shell needed
            seconds: 8
      # app must: stop accepting, drain in-flight, then exit
---
# Dockerfile detail that breaks all of this:
# ENTRYPOINT /app/server        <- sh -c wraps it, SIGTERM lost
# ENTRYPOINT ["/app/server"]  <- exec form, signals delivered

Answer

Every pod is assigned one of three QoS classes derived purely from its resource spec. Guaranteed: every container has requests equal to limits for BOTH cpu and memory. Burstable: at least one container has a request or limit, but the pod does not meet the Guaranteed bar.

BestEffort: no requests or limits anywhere. Two distinct killing mechanisms use this, and separating them is what interviewers listen for. Node-pressure eviction is the kubelet acting BEFORE the kernel panics: it monitors signals like memory.available and nodefs.available against eviction thresholds (hard thresholds evict immediately; soft ones after a grace period).

When memory crosses the threshold, the kubelet ranks pods: first by whether usage exceeds requests (BestEffort pods and over-request Burstable pods rank first), then by pod priority, then by how far usage exceeds requests. Guaranteed pods and Burstable pods within their requests are last in line: they only go if system daemons need reclaiming. Evicted pods show status Evicted with a reason, and controller-managed ones reschedule elsewhere.

The kernel OOM killer is the uncoordinated fallback when memory vanishes faster than the kubelet can react: the kubelet pre-arms it by setting oom_score_adj per container (Guaranteed gets -997, nearly untouchable; BestEffort 1000, first to die; Burstable a formula between, scaled by request size), so even the kernel's emergency response roughly respects QoS. OOM kills terminate a CONTAINER (OOMKilled, exit 137, restarted in place), while evictions terminate a POD; different blast radius, different signals in kubectl. Practical consequences: run revenue-path services as Guaranteed (memory request = limit), give batch work honest requests so it lands in Burstable rather than BestEffort roulette, and treat any BestEffort pod in production as an incident waiting for a node with memory pressure.

Answer

A StorageClass names a CSI provisioner plus parameters (EBS volume type gp3 with IOPS settings, a Ceph pool, an NFS export class). When a PVC references it (storageClassName, or omits it to get the class annotated as default), the external-provisioner sidecar of that CSI driver sees the claim and calls CreateVolume, producing a PersistentVolume bound one-to-one to the claim: no admin pre-creating PVs, which is what 'dynamic' means. volumeBindingMode is the field with real operational teeth: Immediate provisions as soon as the PVC is created, BEFORE any pod is scheduled, which on multi-zone clusters can create the disk in zone A while the scheduler later wants the pod in zone B, an unresolvable conflict for zonal disks. WaitForFirstConsumer delays provisioning until a pod using the claim is scheduled, letting the scheduler consider CPU, memory AND storage topology together, then provisions in the chosen zone; it is the correct default for zonal storage, and interviewers specifically probe the zone-mismatch story. allowVolumeExpansion: true permits growing a volume by editing the PVC's requested size (never shrinking); the CSI driver resizes the backing disk and, for filesystems, the kubelet grows the FS, online for most modern drivers. reclaimPolicy decides the PV's fate when its claim is deleted: Delete (default for dynamic provisioning) destroys the backing disk, Retain keeps PV and disk in Released state for manual salvage, the sane choice for databases where a fat-fingered PVC delete should not erase data. accessModes are about NODE attachment, not pod counts: ReadWriteOnce (one node), ReadWriteMany (many nodes, needs a shared filesystem like EFS/NFS/CephFS; block EBS-style storage cannot do it), ReadOnlyMany, and ReadWriteOncePod, which restricts to a single POD for cases like databases where even two pods on the same node must be prevented. Add volumeSnapshots via the VolumeSnapshot API for backup workflows, and you have the full storage lifecycle interviews cover.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retained
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: '6000'
  throughput: '250'
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-data
spec:
  storageClassName: gp3-retained
  accessModes: [ReadWriteOncePod]
  resources:
    requests:
      storage: 200Gi   # later edit upward to expand online

Answer

Five recurring surprises. One: deleting the StatefulSet, or scaling it down, leaves the PVCs (and the cloud disks billing you) behind. That is deliberate data protection, but teams discover it as either a surprise bill or a surprise when a scale-up re-attaches an OLD disk with stale data to the new pod-3.

The persistentVolumeClaimRetentionPolicy field (stable in recent releases) makes the behaviour explicit per lifecycle event (whenDeleted / whenScaled, Retain or Delete). Two: a pod on a failed node is NOT automatically replaced. Because the controller cannot verify the kubelet is truly dead rather than partitioned, it will not risk two pods with the same identity mounting the same volume; the pod sits Terminating until the Node object is deleted (cloud controllers do this when the instance is gone) or someone force-deletes the pod, accepting split-brain risk consciously.

This at-most-once guarantee versus availability trade-off is a strong interview talking point. Three: ordered rolling updates halt on failure: if pod 2 of 5 crashloops on the new image, the rollout freezes there, and unlike Deployments there is no automatic surge capacity; recovery is rolling back the template and often deleting the stuck pod manually. Four: the updateStrategy partition field is the built-in canary: set partition: 2 and only ordinals >= 2 update, letting you bake the new version on one replica before lowering the partition to 0.

Five: OnDelete strategy (pods update only when YOU delete them) remains the right choice for databases where you want update timing under human or operator control. Practical operating posture: pair StatefulSets with an operator (CloudNativePG, Strimzi, MongoDB operator) whenever one exists, because these edge cases: identity, ordered recovery, backup orchestration, are exactly what operators encode.

Answer

Structure: one chart per service (or a shared library chart for common templates), with values.yaml holding safe defaults and per-environment overlays (values-staging.yaml, values-prod.yaml) containing ONLY the deltas: replicas, resources, hostnames, feature flags. CI deploys with `helm upgrade --install -f values.yaml -f values-prod.yaml --set image.tag=$SHA`, keeping image tags out of files entirely, or, in GitOps shops, a bot commits the tag bump and Argo CD applies it. Never fork charts per environment; the whole point is one template tested everywhere.

Safety mechanisms: --atomic turns a failed upgrade into an automatic rollback to the previous revision (it implies --wait, which blocks until Deployments are ready, probes passing, within --timeout), converting the worst Helm failure mode, a half-applied release, into a non-event. helm diff (the ubiquitous plugin) renders the incoming release against the live one and prints a diff in CI review; deploying without seeing that diff is flying blind, and many teams gate merges on it. Hooks attach Jobs to lifecycle points via annotations: helm.sh/hook: pre-upgrade with a hook-weight for ordering is the standard slot for database migrations, so schema changes complete before new pods start; hook-delete-policy: before-hook-creation,hook-succeeded keeps failed hook Jobs around for debugging while cleaning successes. Hook caveats worth volunteering: hooks are not transactional with the release (a failed pre-upgrade hook aborts the upgrade, but a succeeded migration is not rolled back by --atomic, so migrations must be backward-compatible), and hook resources are not tracked as release resources, so they escape helm uninstall. Round out with chart testing: helm lint, helm-unittest for template assertions, and kubeconform against the rendered output in CI to catch schema drift before the cluster does.

# CI deploy with full safety rails
helm diff upgrade checkout ./charts/checkout \
  -n payments -f values.yaml -f values-prod.yaml \
  --set image.tag=$GIT_SHA

helm upgrade --install checkout ./charts/checkout \
  -n payments -f values.yaml -f values-prod.yaml \
  --set image.tag=$GIT_SHA \
  --atomic --timeout 7m
---
# migration hook in templates/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: '{{ .Release.Name }}-migrate'
  annotations:
    helm.sh/hook: pre-upgrade,pre-install
    helm.sh/hook-weight: '-5'
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: 'registry.example.com/checkout:{{ .Values.image.tag }}'
          args: ['migrate', 'up']

Answer

Kustomize is template-free: you keep plain, valid YAML and layer modifications. A base/ directory holds the canonical manifests plus a kustomization.yaml listing them; each overlay (overlays/prod/) references the base and applies transformations: namespace and label injection, image tag overrides via the images field, replica counts, ConfigMap/Secret generation from files with content-hash suffixes (which automatically triggers rollouts when config changes, solving a problem Helm needs the checksum-annotation trick for), and patches. Patches come in two flavours: strategic-merge (a YAML fragment merged using the same merge keys the API server uses, so list items match by name rather than index) and JSON6902 (surgical operations: add/replace/remove at a path) for resources or fields where strategic merge is ambiguous.

It is built into kubectl (`kubectl apply -k overlays/prod`), so there is no extra binary, and being template-free means every layer is valid YAML you can lint and diff: no {{ }} soup, no whitespace-trim bugs. What it lacks is packaging and distribution: no versioned releases, no rollback state, no dependency management, no values-based public consumption. Hence the division of labour: Helm wins for DISTRIBUTING software (third-party charts: ingress controllers, Prometheus, cert-manager) and when consumers need a parameter surface; Kustomize wins for YOUR OWN apps in a GitOps repo where environments differ by small structured deltas and you want maximal reviewability.

The hybrid is standard practice and worth naming: render a Helm chart (helm template, or Argo CD and Flux both support Helm-then-Kustomize natively) and patch the output with Kustomize for the org-specific mutations the chart never parameterised: adding a sidecar, forcing a runtime class, injecting labels your cost tooling requires. Interviewers mostly want to hear that you know these are complementary, not competing, and that GitOps controllers happily drive both.

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: payments-prod
commonLabels:
  env: prod
images:
  - name: registry.example.com/checkout
    newTag: v43
replicas:
  - name: checkout
    count: 6
configMapGenerator:
  - name: checkout-config
    files:
      - config/flags.yaml   # hash suffix forces rollout on change
patches:
  - target:
      kind: Deployment
      name: checkout
    patch: |-
      - op: add
        path: /spec/template/spec/priorityClassName
        value: revenue-critical
# apply: kubectl apply -k overlays/prod

Answer

Argo CD inverts the deploy direction: instead of CI pushing manifests at the cluster, a controller inside the cluster continuously PULLS desired state from Git and reconciles. An Application CRD binds a source (repo, path or chart, target revision) to a destination (cluster, namespace); the controller renders the source (plain YAML, Helm, or Kustomize), compares it to live state, and reports Synced/OutOfSync plus Healthy/Degraded (health being resource-type-aware: a Deployment is healthy when its rollout completes). With automated sync policy, drift gets corrected without a human: selfHeal reverts manual kubectl edits (killing hotfix drift as a category, which is both the feature and the culture shock), and prune deletes live resources that vanished from Git, which is why prune is usually enabled only after teams trust the pipeline.

The ordering problem: applying forty manifests alphabetically breaks when the namespace, CRDs or the database must exist before consumers; sync waves solve it with the argocd.argoproj.io/sync-wave annotation, an integer per resource, applied in ascending waves with health checks passing between waves (CRDs at wave -2, infrastructure at -1, apps at 0, post-jobs at 1). Resource hooks (PreSync, Sync, PostSync, SyncFail) run Jobs at phase boundaries: PreSync is where database migrations live in Argo shops, the equivalent of Helm's pre-upgrade hook. App-of-apps is the bootstrap pattern: one root Application whose source directory contains OTHER Application manifests, so pointing a fresh cluster at the root app materialises the entire platform recursively; ApplicationSets generalise this with generators that stamp Applications from templates across clusters or Git directories, the standard multi-cluster answer. Round off with the operational reality: secrets never live in Git plainly (pair with External Secrets or Sealed Secrets), and RBAC on Argo projects controls which teams may deploy where.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout
  namespace: argocd
spec:
  project: payments
  source:
    repoURL: https://github.com/example/platform-config
    targetRevision: main
    path: apps/checkout/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3
      backoff: { duration: 20s, factor: 2 }
---
# ordering: migrations before app rollout
# metadata:
#   annotations:
#     argocd.argoproj.io/hook: PreSync
#     argocd.argoproj.io/sync-wave: '-1'

Answer

kubectl exec needs a shell in the image; distroless and scratch images deliberately have none (smaller attack surface, no CVE-laden userland), so the tool is ephemeral containers, stable since 1.25 via `kubectl debug`. An ephemeral container is injected into the RUNNING pod through the pod's ephemeralcontainers subresource: no restart, no spec mutation of the durable containers, and it can carry a full toolbox image while the app image stays minimal. The crucial flag is --target=<container>: with it, the debug container joins the target container's PROCESS namespace, so ps sees the app's processes and, the killer trick, you can inspect the app's filesystem through /proc/<pid>/root even though the debug container has its own separate image filesystem.

Network namespace is shared pod-wide already, so ss, netstat and curl against localhost observe the app's actual sockets. Limits: ephemeral containers cannot have ports or probes, cannot be removed once added (they stay until the pod dies), and process-namespace targeting needs the runtime's cooperation. Second mode: `kubectl debug --copy-to=X` clones the pod with modifications, the standard play for crash-at-startup cases: clone with --container app and a sleep override for the command, then exec in and run the real binary by hand; --set-image swaps images in the copy, and the copy drops labels so Services do not route to it.

Third mode: node debugging: `kubectl debug node/<name> -it --image=busybox` runs a privileged pod with the node filesystem at /host, effectively SSH-less node access for inspecting kubelet logs (chroot /host journalctl -u kubelet) or containerd state: how you debug nodes in SSH-locked managed clusters. Purpose-built debug images to name: registry.k8s.io/e2e-test-images/agnhost for network probing, nicolaka/netshoot as the swiss-army networking image.

# live debug: join the app container's process namespace
kubectl debug -it api-7f9c-x2v \
  --image=nicolaka/netshoot \
  --target=app -- bash
# inside: ps aux; ss -tlnp;
# cat /proc/1/root/etc/app/config.yaml   <- app's filesystem

# crash-at-startup: clone with a sleep entrypoint
kubectl debug api-7f9c-x2v --copy-to=api-debug \
  --container=app -- sleep infinity
kubectl exec -it api-debug -- /app/server --config /etc/app/config.yaml

# node access without SSH
kubectl debug node/ip-10-0-3-17 -it --image=busybox
# inside: chroot /host sh; journalctl -u kubelet --since '10 min ago'

Answer

ResourceQuota caps a NAMESPACE's totals; LimitRange constrains and defaults INDIVIDUAL pods and containers. A quota can cap compute (requests.cpu, requests.memory, limits.cpu, limits.memory summed across all pods), object counts (pods, services, secrets, count/deployments.apps), storage (requests.storage total, per-StorageClass caps, persistentvolumeclaims), and even scoped subsets (quota only for a given priorityClass via scopeSelector, the mechanism for capping how much high-priority capacity a team may claim). Enforcement is at admission: a pod pushing any tracked total over the cap is rejected with a 403 and an explicit exceeded-quota message; running pods are never killed by quota changes, only future admissions blocked.

The interlock that catches people: once a quota tracks requests.cpu or memory, EVERY pod in the namespace must declare those requests, or it is rejected outright: quota needs numbers to add up. That is what LimitRange's defaulting exists for: defaultRequest and default (for limits) are injected into containers that did not specify them, so developer manifests without resources blocks keep admitting instead of erroring. LimitRange also enforces per-container min and max (nobody requests 500 CPUs), maxLimitRequestRatio (bounds overcommit by capping limit divided by request, forcing honest requests), and PVC size bounds via type PersistentVolumeClaim.

What breaks with only one: quota without LimitRange rejects every unannotated pod (developer experience disaster, teams 'fix' it by copy-pasting inflated requests); LimitRange without quota means well-formed pods but unbounded namespace totals, so one team's HPA burst can still eat the cluster: you need the pair. Interview extension: at real multi-tenant scale you layer namespace quotas with PriorityClass quotas plus scheduler-level fairness, and tools like Kueue handle queued batch admission on top: mentioning that shows awareness the primitives compose.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: '40'
    requests.memory: 80Gi
    limits.memory: 120Gi
    pods: '150'
    count/deployments.apps: '40'
    requests.storage: 2Ti
    persistentvolumeclaims: '30'
---
apiVersion: v1
kind: LimitRange
metadata:
  name: sane-defaults
  namespace: team-payments
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default: { memory: 256Mi }
      max: { cpu: '8', memory: 16Gi }
      maxLimitRequestRatio: { memory: '2' }
    - type: PersistentVolumeClaim
      max: { storage: 500Gi }

Answer

Every controller, built-in or custom, is the same machine: observe actual state, compare with desired state, act to converge, repeat. The naive implementation, polling LIST on every loop, would flatten the API server, so the client-go informer stack exists: an informer opens a LIST to seed a local cache, then a WATCH (long-lived streaming of change events with resourceVersion bookkeeping) to keep it current; your handlers fire on add/update/delete, and all reads go against the local indexed cache (a Lister), making reconciliation read-mostly-local. SharedInformerFactory deduplicates informers across controllers in one process.

Handlers do NOT do the work; they enqueue only a namespace/name key onto a rate-limited work queue. That indirection buys everything: dedup (five rapid updates to one object collapse into one queued key), retries with exponential backoff on error, and a crucial correctness property: by the time a worker processes the key, it re-reads CURRENT state from cache, so controllers are level-based (act on the latest state), not edge-based (react to each event): missed events cannot corrupt outcomes because a periodic resync re-enqueues everything anyway. A correct Reconcile is therefore: idempotent (running twice converges to the same place), stateless across invocations (all state re-derived from the API), conflict-tolerant (update races return 409; you re-queue and retry, or use server-side apply), and it writes status via the /status subresource to report observed truth.

Ownership and cleanup ride on ownerReferences (garbage collection deletes orphaned children) and finalizers (your controller's chance to release external resources before deletion completes: remove the finalizer or the object hangs Terminating forever, the bug everyone ships once). controller-runtime and kubebuilder package all of this: you write Reconcile, they run the informers, queues, leader election (a Lease so only one replica acts) and metrics. Build an operator when you are encoding operational KNOWLEDGE (failover, backup, upgrade ordering); use Helm when install-time templating suffices.

// controller-runtime reconciler skeleton (kubebuilder)
func (r *CacheReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cache dbv1.Cache
    if err := r.Get(ctx, req.NamespacedName, &cache); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err) // deleted: GC handles owned objects
    }

    var deploy appsv1.Deployment
    err := r.Get(ctx, req.NamespacedName, &deploy)
    if apierrors.IsNotFound(err) {
        deploy = r.desiredDeployment(&cache)
        if err := ctrl.SetControllerReference(&cache, &deploy, r.Scheme); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, r.Create(ctx, &deploy) // requeued via watch on Deployments
    } else if err != nil {
        return ctrl.Result{}, err
    }

    cache.Status.ReadyReplicas = deploy.Status.ReadyReplicas
    return ctrl.Result{}, r.Status().Update(ctx, &cache)
}

Answer

Admission is the API server's interception point after authentication and authorization but before persistence. Mutating webhooks run first (injecting sidecars, defaulting fields: this is how Istio's sidecar and cert-manager's CA injection work), then schema validation, then validating webhooks. Both are registered via MutatingWebhookConfiguration and ValidatingWebhookConfiguration: the API server calls YOUR HTTPS endpoint with an AdmissionReview and waits.

That synchronous network hop is the operational hazard, and interviewers want the failure story: failurePolicy: Fail means webhook down = matching requests rejected; scoped too broadly (all pods, all namespaces) that becomes 'nothing can schedule cluster-wide', and if the webhook's own pods are among the blocked, the cluster cannot self-heal: a real and recurring class of outage. Mitigations you must name: always exclude kube-system and the webhook's own namespace via namespaceSelector, keep timeoutSeconds low (webhooks cap at 30s; keep it in low single digits), scope objectSelector/rules as narrowly as possible, and choose failurePolicy per rule: Fail for security-critical validation, Ignore for convenience mutations. Also mind reinvocationPolicy: a later mutating webhook can invalidate an earlier one's work, so order-sensitive mutations need reinvocation enabled.

ValidatingAdmissionPolicy (GA since 1.30) removes the network hop entirely: policies are CEL expressions evaluated IN-PROCESS by the API server, with parameterisation via arbitrary config objects (paramKind), matchConditions for scoping, audit/warn/deny actions via ValidatingAdmissionPolicyBinding, and messageExpression for rich errors. No webhook server to run, no TLS to rotate, no availability coupling: for rule-shaped validation (deny :latest tags, require labels, bound replicas, forbid hostPath), it obsoletes validating webhooks. What still needs webhooks: MUTATION (though a CEL-based MutatingAdmissionPolicy is progressing through the release train), validations requiring external data lookups, and complex logic beyond CEL. The modern stack: VAP for cheap guardrails, Kyverno or Gatekeeper where you want a policy library and reporting, custom webhooks only when unavoidable.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-latest-tag
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ['apps']
        apiVersions: ['v1']
        operations: ['CREATE', 'UPDATE']
        resources: ['deployments', 'statefulsets']
  validations:
    - expression: >-
        object.spec.template.spec.containers.all(c,
          !c.image.endsWith(':latest') && c.image.contains('@sha256:')
          || c.image.matches('.*:v[0-9].*'))
      message: images must use a pinned version tag or digest, not :latest
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-latest-tag-prod
spec:
  policyName: deny-latest-tag
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels: { env: prod }

Answer

etcd is the only stateful component of the control plane; lose it unrecoverably and the cluster's entire declarative state (every object, secret, CRD) is gone, even though workloads keep running headless until kubelets need answers. Backup is etcdctl against a member: `etcdctl snapshot save` with the member's TLS certs produces a point-in-time snapshot file; production practice is a CronJob or systemd timer snapshotting frequently, shipping off-cluster (S3 with lifecycle rules), verifying restorability (etcdutl snapshot status, plus periodic actual restore drills: an unverified backup is a hope, not a backup), and ALSO backing up the certificates directory, since a restored etcd is useless if the API server cannot authenticate to it. Restore uses etcdutl snapshot restore to materialise a fresh data directory with a NEW cluster identity, then you point the (static-pod) etcd manifest at it; multi-member clusters restore each member from the same snapshot with a new initial-cluster config.

Know what restore means semantically: the cluster time-travels; controllers will reconcile reality against older desired state, and anything created after the snapshot becomes orphaned from the API's perspective: restore is a disaster measure, not an undo button. Design constraints etcd imposes: it is a Raft quorum system, so run odd member counts (3 or 5; 5 tolerates 2 failures) and never even ones; every write is a quorum round-trip plus fsync, making DISK LATENCY the dominant health factor (etcd's own guidance is low-millisecond fsync; slow disks manifest as leader elections and apply lag: put etcd on fast local SSD, never network storage with jittery latency). Size: the backend quota defaults to 2GiB and the project advises keeping the DB under about 8GiB; the space is consumed by revisions, so compaction (the API server requests it periodically) plus occasional defrag reclaim space, and exceeding quota puts etcd into a read-only alarm state, a cluster-wide outage with a five-minute fix (compact, defrag, disarm alarm) that people burn hours diagnosing. Chronic bloat usually traces to Events volume or a misbehaving controller writing huge objects; Events can live in a separate etcd instance on big clusters.

# snapshot (run against one member, with its certs)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcdutl snapshot status /backup/etcd-2026-08-11.db -w table

# restore into a fresh data dir, then repoint the static pod
etcdutl snapshot restore /backup/etcd-2026-08-11.db \
  --data-dir /var/lib/etcd-restored \
  --name master-1 \
  --initial-cluster master-1=https://10.0.0.11:2380 \
  --initial-advertise-peer-urls https://10.0.0.11:2380

# space pressure first aid
etcdctl endpoint status -w table   # DB size, leader
etcdctl defrag --cluster
etcdctl alarm disarm

Answer

Native sidecars, stable in 1.33 after the 1.29 beta, are expressed as an entry in initContainers with restartPolicy: Always: syntactically an init container, behaviourally a new third category. Startup: like any init container it starts in list order BEFORE the app containers, but instead of running to completion it keeps running; the sequence proceeds once its startupProbe passes (or immediately without one), giving you guaranteed readiness ordering: the Envoy proxy, log shipper, or secrets agent is demonstrably up before the app boots. Shutdown: sidecars terminate AFTER the app containers finish, in reverse order, so the proxy keeps serving until the app's in-flight work drains.

Restarts: restartPolicy Always means the kubelet restarts a crashed sidecar independently, even during pod termination grace. Now the problems this closes, which is where interview credit lives. Jobs-with-sidecars: for years, a Job pod with an Istio proxy or cloudsql-proxy NEVER completed, because the sidecar kept running after the main container exited; the ecosystem's workarounds (quitquitquit endpoints, pkill wrappers, shareProcessNamespace hacks) are all obsolete: sidecars do not block Job completion, and the Job controller treats them correctly.

Startup races: apps that crashed on boot because the mesh proxy was not yet routing (the celebrated 'my app needs a sleep 5' bug class) are fixed by ordering. Shutdown races: log shippers being killed before the app's final logs flushed, losing crash evidence, are fixed by reverse-order termination. Interaction details worth knowing: sidecar restarts do NOT re-trigger the app containers; sidecars can have readiness probes contributing to pod readiness (regular init containers cannot); their requests count toward the pod's effective request differently from run-once inits; and meshes exploit all this: Istio's native sidecar mode sets the annotation-driven injection to emit the initContainers form. Migration is mechanical: move the container from containers to initContainers and add restartPolicy: Always, but only on clusters at or beyond the stable version.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-report
spec:
  template:
    spec:
      restartPolicy: Never
      initContainers:
        - name: cloud-sql-proxy       # native sidecar
          image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.0
          restartPolicy: Always       # <- the magic field
          args: ['--port=5432', 'project:asia-south1:orders-db']
          startupProbe:
            tcpSocket: { port: 5432 }
            periodSeconds: 2
            failureThreshold: 30
      containers:
        - name: report
          image: registry.example.com/reporter:v9
          # connects to localhost:5432; proxy guaranteed ready first,
          # and the Job COMPLETES when this container exits

Answer

The version skew policy dictates the order and everything else follows from it. kube-apiserver must be the newest component; controller-manager, scheduler and cloud-controller-manager may trail it by one minor; kubelet may be up to three minors behind the apiserver (the window widened in 1.28); kube-proxy tracks its own node's kubelet; kubectl is supported within one minor either way. Control planes move one minor at a time and never skip, and kubeadm refuses to let you. So the sequence is audit, control plane, add-ons, nodes.

Audit first: read the release notes and the deprecated API guide for the target version, then find what in YOUR cluster still calls a removed API. The apiserver_requested_deprecated_apis gauge names the group, version, resource and the release that removes it, and joining it against apiserver_request_total by user_agent tells you which controller or CI job to fix; pluto and kubent do the same scan against manifests and Helm releases. Check compatibility matrices for everything that is not core: CNI, CSI drivers, the ingress or gateway controller, metrics-server, cert-manager, and every admission webhook, because a webhook that cannot parse the new API version rejects the objects that carry it.

Snapshot etcd, rehearse on a staging cluster with the same add-on set, then upgrade the control plane (kubeadm upgrade apply on the first node, kubeadm upgrade node on the others; on EKS or GKE it is an API call and a wait). Add-ons next, since they must tolerate both old and new kubelets during the node phase. Nodes last, and 200 of them is where strategy matters: in-place (cordon, drain, kubeadm upgrade node, uncordon) is slow and safe, while blue-green node pools (stand up a pool at the new version, drain the old one, delete it) is the managed-cluster default and gives instant rollback.

Karpenter treats a version bump as node drift and rolls nodes itself, which is precisely why disruption budgets must be correct first. What breaks in practice: PDBs allowing zero disruption hang drains forever, singleton pods block them, workloads on removed beta APIs fail admission the moment the control plane moves, and DaemonSets passing kubelet flags that no longer exist crash on the new nodes. Schedule outside IST business hours, and for payments platforms outside settlement windows.

# 1. who still calls APIs the next release removes?
#    PromQL against the API server metrics:
#    apiserver_requested_deprecated_apis
#      * on(group,version,resource,subresource) group_right()
#      sum by(group,version,resource,subresource,user_agent)
#        (rate(apiserver_request_total[7d]))
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

# 2. control plane, one minor at a time
kubeadm upgrade plan
sudo kubeadm upgrade apply $TARGET_VERSION

# 3. nodes, respecting PodDisruptionBudgets
kubectl drain node-17 --ignore-daemonsets \
  --delete-emptydir-data --timeout=15m
sudo kubeadm upgrade node && sudo systemctl restart kubelet
kubectl uncordon node-17

# 4. verify skew before continuing
kubectl get nodes -o custom-columns=\
NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion

Answer

API Priority and Fairness (flowcontrol.apiserver.k8s.io, GA since 1.29) replaced the blunt max-in-flight request limits, which could count requests but could not tell a node heartbeat apart from a runaway list loop. APF classifies every inbound request with a FlowSchema, matched on subject (user, group or ServiceAccount) plus resource and verb rules, and routes it to a PriorityLevelConfiguration that owns a share of the server's total concurrency. Inside a level, requests are bucketed into flows by a distinguisher (typically the requesting user or namespace) and served through queues with shuffle sharding, so one noisy client collides with only a fraction of the queues rather than starving the whole level.

Overflow is rejected with HTTP 429 plus a Retry-After header, which client-go honours by backing off. Two shipped levels matter by name: the exempt level (system:masters, never queued, which is why an admin script with cluster-admin can still hurt you) and the dedicated levels for leader election and node heartbeats, so a busy cluster does not lose leases and mark nodes NotReady under load. Diagnosis follows the metrics. apiserver_flowcontrol_rejected_requests_total split by flow_schema and priority_level names who is being throttled; apiserver_flowcontrol_current_inqueue_requests and apiserver_flowcontrol_request_wait_duration_seconds show queueing before rejection; and comparing apiserver_request_duration_seconds against etcd_request_duration_seconds separates a busy API server from a slow etcd.

The usual culprit is a controller doing repeated full LISTs of pods, secrets or events with no resourceVersion, which bypasses the watch cache, goes to etcd, and materialises the entire collection in the API server's heap. One such operator can drive apiserver memory and p99 latency for everyone. Fix in that order: repair the client (informers instead of polling, list with resourceVersion=0 so the watch cache serves it, paginate with limit and continue, metadata-only informers for large objects), and only then reach for APF configuration. When a third-party controller cannot be fixed quickly, write a FlowSchema matching its ServiceAccount and pin it to a low-share priority level: it gets throttled first and the rest of the cluster keeps working, which is exactly the containment answer interviewers are listening for.

# quarantine a noisy controller into a small share of concurrency
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: noisy-operators
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 5
    limitResponse:
      type: Queue
      queuing: { queues: 16, handSize: 4, queueLengthLimit: 50 }
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: legacy-inventory-operator
spec:
  matchingPrecedence: 500
  priorityLevelConfiguration: { name: noisy-operators }
  distinguisherMethod: { type: ByUser }
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount: { name: inventory-operator, namespace: platform }
      resourceRules:
        - verbs: ['*']
          apiGroups: ['*']
          resources: ['*']
          clusterScope: true
          namespaces: ['*']
---
# who is getting throttled, and who is causing it
# sum by (flow_schema, priority_level) (
#   rate(apiserver_flowcontrol_rejected_requests_total[5m]))
# topk(10, sum by (user_agent, verb, resource) (
#   rate(apiserver_request_total{verb='LIST'}[5m])))

Answer

Measure before cutting, because you pay for what is RESERVED on nodes, not what is used. The headline number is request efficiency: actual usage divided by summed requests. Most clusters that have never been tuned sit somewhere in the 15-30% range for CPU, which is where a 40% reduction comes from without touching a single reliability property.

Levers in return-on-effort order. First, right-size requests from real data: p95 CPU and peak memory per container over two to four weeks, produced by the VPA in recommender-only mode (updateMode: Off) or Goldilocks. Never right-size memory to p95: memory is incompressible, so you size it on the observed peak plus headroom or you trade cost for OOMKills.

Second, fix bin packing: fewer, larger nodes pack better than many small ones, and Karpenter consolidation continuously replaces underused nodes with cheaper arrangements. Third, change what you buy: spot capacity for anything interruptible (batch, CI runners, stateless web tiers with healthy PDBs) and arm64 instances (Graviton on AWS, Tau on GCP), which needs multi-arch images built with buildx. Fourth, stop paying for idle: non-production namespaces scaled to zero outside IST working hours via a KEDA cron trigger removes roughly two thirds of the week's hours.

Fifth, hunt the invisible line items: orphaned PVs left behind by Retain reclaim policies, snapshots nobody prunes, one cloud load balancer per Service where a shared Gateway would serve all of them, and cross-zone data transfer, which the topology-aware routing field (spec.trafficDistribution: PreferClose on the Service) keeps in-zone. Attribution decides whether any of it sticks: OpenCost, or Kubecost layered on it, splits node cost across pods by requests and rolls it up by namespace, label or team, but only when workloads carry consistent ownership labels, so enforce those with an admission policy before you build the dashboard. Report cost AND efficiency per team, because cost alone punishes teams for growing. Guardrails to state explicitly: keep PDBs so consolidation and spot reclaims stay graceful, leave headroom for HPAs to burst into, never put the stateful tier on spot, and move one lever at a time so savings are attributable.

# CPU request efficiency cluster-wide: used vs reserved
sum(rate(container_cpu_usage_seconds_total{container!=''}[7d]))
  /
sum(kube_pod_container_resource_requests{resource='cpu'})

# biggest wasted reservations by namespace
sort_desc(
  sum by (namespace) (kube_pod_container_resource_requests{resource='cpu'})
  -
  sum by (namespace) (rate(container_cpu_usage_seconds_total[7d]))
)

# size memory on the observed peak, never on p95
max_over_time(container_memory_working_set_bytes{container!=''}[14d])

# recommendations without enforcement
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-recommender
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout }
  updatePolicy:
    updateMode: 'Off'   # advise only, do not evict pods

Answer

The device plugin API is the old contract: a DaemonSet registers with the kubelet over a socket and advertises an extended resource name such as nvidia.com/gpu, which the scheduler then treats as an opaque integer requested under resources.limits. That model has no vocabulary for attributes. You cannot express 'a GPU with at least 40GB of memory', 'two GPUs connected over NVLink', or 'a slice of one GPU'.

The workarounds were node labels from GPU Feature Discovery plus nodeAffinity, per-node time-slicing configuration, and exposing MIG profiles as separately named resources, all of which push topology decisions out of the scheduler and into node-labelling conventions that nobody can audit. Dynamic Resource Allocation replaces it, with the core resource.k8s.io APIs reaching GA in 1.34. Drivers publish what each node actually has into ResourceSlice objects, carrying structured attributes (model, memory size, driver version) and capacity.

A DeviceClass names a category of device with CEL selectors over those attributes. Workloads create a ResourceClaim, or a ResourceClaimTemplate when each pod needs its own instance, and reference it from the pod spec; the scheduler allocates devices as part of ordinary scheduling, so device constraints, node fit and topology are solved in one pass instead of two. Claims can also be shared: several pods referencing the same ResourceClaim receive the same device, which is how you deliberately co-locate inference replicas on one accelerator.

What to say about production: DRA does not itself make a GPU divisible. Sharing still comes from vendor mechanisms (MIG partitions, time-slicing, MPS), but the driver now exposes those as selectable devices with attributes rather than hand-rolled resource names. Given how scarce and expensive GPU capacity is in Indian cloud regions, the practical wins are packing inference onto MIG slices while keeping training on whole devices.

Recognise the failure mode: a pod sits Pending with an unallocated claim, and the reason lives in kubectl describe resourceclaim, not in describe pod. Before relying on autoscaling for GPU nodes, verify your autoscaler understands DRA claims, since group-based simulation and device claims are a newer combination than pod resource requests.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: inference-slice
spec:
  selectors:
    - cel:
        expression: >-
          device.driver == 'gpu.nvidia.com' &&
          device.attributes['gpu.nvidia.com'].productName.contains('MIG')
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: one-inference-slice
  namespace: ml-serving
spec:
  spec:
    devices:
      requests:
        - name: accel
          exactly:
            deviceClassName: inference-slice
            allocationMode: ExactCount
            count: 1
---
# pod template referencing the claim
spec:
  resourceClaims:
    - name: accel
      resourceClaimTemplateName: one-inference-slice
  containers:
    - name: server
      image: registry.example.com/llm-serve:v4
      resources:
        claims:
          - name: accel

Answer

Historically a pod's resources were immutable: changing requests or limits meant deleting the pod and letting the controller recreate it, which is why vertical autoscaling was disruptive and why almost nobody ran the VPA in updateMode: Auto against production singletons. In-place pod vertical scaling changes that. It reached beta and became enabled by default in 1.33 (check the exact graduation state on your cluster before designing around it), and it works through a dedicated pods/resize subresource: you patch spec.containers[].resources through that subresource and the kubelet applies new cgroup values to the RUNNING container, no recreation, same node, same pod IP, same PVC attachment.

Per-resource behaviour is declared with resizePolicy: NotRequired means apply live, RestartContainer means the container must restart for the change to take effect (typical for JVM heaps sized from cgroup limits at startup, and the usual choice for memory). Observe the outcome through pod conditions and status: a resize the node cannot satisfy right now is Deferred and retried, one it can never satisfy is Infeasible, and containerStatuses report both allocated and actual resources so you can see what really landed. The constraints matter in interviews: resizing never moves the pod, so a resize larger than the node's remaining capacity simply waits; QoS class cannot change, so a Guaranteed pod stays Guaranteed and its requests and limits must move together; memory limit reductions are the risky direction because the kernel cannot reclaim pages that are already in use.

What it changes downstream: the VPA can grow a pod without evicting it, which finally makes vertical autoscaling viable for stateful and slow-starting workloads, and a memory-leak mitigation becomes a resize rather than a restart. What it does not change: HPA and VPA still conflict when both act on CPU, because the VPA moves requests and CPU utilisation is measured against requests, so the HPA's input shifts under it. The safe division stays the same: HPA on RPS or CPU, VPA on memory, or VPA in recommender-only mode feeding a human or a pipeline.

spec:
  containers:
    - name: app
      image: registry.example.com/api:v7
      resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired      # apply live
        - resourceName: memory
          restartPolicy: RestartContainer # JVM reads the limit at boot
      resources:
        requests: { cpu: 500m, memory: 1Gi }
        limits:   { cpu: '2',  memory: 1Gi }
---
# raise CPU on a live pod through the resize subresource
kubectl patch pod api-7f9c-x2v --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"1"},"limits":{"cpu":"3"}}}]}}'

# what actually got applied, and whether it is pending
kubectl get pod api-7f9c-x2v -o jsonpath=\
'{.status.containerStatuses[0].resources}{"\n"}'
kubectl get pod api-7f9c-x2v -o jsonpath='{.status.conditions}'

Answer

Start by naming the tenancy model, because the answer is entirely different for trusted internal teams than for untrusted code. For 40 internal teams the baseline is namespace-per-team, but the namespace is worthless without the bundle that goes with it: RBAC bound to IdP groups rather than individuals, ResourceQuota plus LimitRange so quota admission has numbers to add up, a default-deny NetworkPolicy with an explicit DNS egress allow, Pod Security Admission at restricted with warn and audit ahead of enforce, PriorityClass-scoped quota so no team can hoard high-priority capacity, and ownership labels for cost attribution. The engineering point interviewers want is that this bundle must be generated, not documented: a Kyverno generate policy, an Argo CD ApplicationSet, or a Terraform module that stamps every artefact when a namespace appears.

Hand-created namespaces drift within a quarter, and drift is how the one namespace without a NetworkPolicy becomes the incident. Where namespaces stop: CRDs, ClusterRoles, StorageClasses, admission policies, PriorityClasses and webhooks are cluster-scoped, so one team's operator upgrade changes an API for everybody; a badly written controller consumes API server concurrency shared by all tenants (APF containment is the mitigation); CoreDNS, the ingress or gateway layer, the CNI and the node pool are shared failure domains; and pods from different tenants sit on the same kernel, so a container escape crosses every namespace boundary at once. The escalation ladder, cheapest first: dedicated node pools with taints and tolerations to isolate noisy or regulated workloads; RuntimeClass with a sandboxed runtime (gVisor or Kata) for anything running customer-supplied code; vcluster when a team genuinely needs its own CRDs and cluster-admin without owning infrastructure; and separate clusters when the boundary is regulatory or adversarial, which in Indian fintech is usually the PCI scope decision, or when blast radius during upgrades is the concern.

Finish with the platform contract: paved-road charts, a self-service namespace request that runs the automation, published quotas, and a per-team cost and efficiency report. Multi-tenancy fails on governance far more often than on missing primitives.

# every namespace labelled tenant=true gets the safety bundle
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-namespace-bundle
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: ['Namespace']
              selector:
                matchLabels: { tenant: 'true' }
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny-all
        namespace: '{{request.object.metadata.name}}'
        synchronize: true
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]
---
# untrusted workloads get a sandboxed runtime, not just a namespace
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
---
spec:
  runtimeClassName: gvisor

Answer

A CRD carries a list of versions, each with two independent flags: served (clients may read and write this version) and storage (exactly one version is true, and it is the form actually persisted in etcd). Migration is therefore a sequence, not a switch. Add v1 as served: true, storage: false so clients can start using it; teach the controller to reconcile both; flip storage to v1 once traffic has moved; then rewrite existing objects, because everything created earlier is still STORED as v1beta1 and the API server converts it on read.

Skipping that rewrite is the classic production failure: the day you finally drop v1beta1 from the CRD, every object still stored in it becomes unreadable. The tooling for the rewrite is a no-op read-and-write pass (kubectl get, then kubectl replace) or the StorageVersionMigration API and the kube-storage-version-migrator, after which you prune status.storedVersions on the CRD so it truthfully lists only v1. Conversion itself is spec.conversion.strategy.

None simply rewrites apiVersion and is safe only when the schemas are compatible, which in practice means purely additive changes with defaults. Anything structural (renaming a field, splitting one field into two, changing units) needs strategy: Webhook, where the API server posts a ConversionReview to your endpoint for every read and write of a non-storage version; cert-manager's ca-injector annotation is the usual way to keep the caBundle valid. Treat that webhook as a hard availability dependency: while it is down, nobody can read those custom resources, including your own controller, so it needs multiple replicas, a PDB and tight timeouts.

Schema mechanics that go with this: structural schemas mean unknown fields are pruned silently unless you set x-kubernetes-preserve-unknown-fields, which is how a client's new field vanishes when the CRD lags; default: fills gaps for older clients; x-kubernetes-validations with CEL expresses rules OpenAPI cannot, including transition rules using oldSelf for immutable fields, which removes a whole class of validating webhooks. Do not forget the subresources: /status keeps controller status writes from fighting user spec applies, and /scale lets an HPA scale your custom resource.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: caches.db.example.com
spec:
  group: db.example.com
  scope: Namespaced
  names: { plural: caches, singular: cache, kind: Cache }
  conversion:
    strategy: Webhook
    webhook:
      conversionReviewVersions: ['v1']
      clientConfig:
        service: { name: cache-operator-webhook, namespace: platform, path: /convert }
  versions:
    - name: v1beta1
      served: true
      storage: false      # still served, no longer stored
      schema: { openAPIV3Schema: { type: object } }
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
        scale:
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-validations:
                - rule: 'self.engine == oldSelf.engine'
                  message: engine is immutable after creation
              properties:
                engine: { type: string, enum: ['redis', 'valkey'] }
                replicas: { type: integer, default: 3, minimum: 1 }
---
# rewrite stored objects before dropping v1beta1
# kubectl get caches -A -o json | kubectl replace -f -

Answer

First the automatic part, because it runs whether or not you wake up. The kubelet renews a Lease object in kube-node-lease every few seconds; when the node controller sees no renewal for the node monitor grace period (40 seconds by default), it sets the Ready condition to Unknown. The node lifecycle controller then applies node.kubernetes.io/unreachable or not-ready with the NoExecute effect.

Pods carry a default toleration for those taints with tolerationSeconds 300, injected by an admission plugin, so about five minutes after the heartbeat stops, pods are marked for deletion and controllers recreate them elsewhere. Two consequences to state: pods with volumes attached, and StatefulSet pods in particular, sit Terminating indefinitely because the control plane cannot prove the old kubelet is dead rather than partitioned, and the workload only recovers when the Node object is deleted (cloud controllers do this when the instance is truly gone) or a human force-deletes and accepts split-brain risk. And NotReady does not mean the containers stopped: a node that is healthy but network-partitioned from the API server keeps serving traffic through whatever routes still reach it.

Triage path: kubectl describe node reads the conditions (MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable) with their messages, plus events; kubectl get lease -n kube-node-lease shows exactly when heartbeats stopped. Then get onto the node without SSH: kubectl debug node/<name> gives a privileged pod with the host filesystem at /host, so chroot /host and read journalctl -u kubelet, systemctl status containerd, and crictl ps. The recurring root causes: /var full from image and log accumulation (check imageGCHighThresholdPercent and whether anything writes to the node filesystem), inode exhaustion, kubelet or containerd starved because system-reserved and kube-reserved were never configured so pods consumed the node's own headroom, a PLEG not healthy message meaning the runtime is hung on relist, CNI failure so the kubelet cannot reach the API server, and expired or unrotatable kubelet client certificates showing as x509 errors in the journal.

Remediation on managed clusters is usually blunt and correct: cordon, drain what you can, terminate the instance and let the node group or Karpenter replace it. Prevention is Node Problem Detector plus alerts on lease staleness and on reserved-resource pressure, not on Ready flapping after the fact.

kubectl get nodes -o wide | grep -v ' Ready'
kubectl describe node ip-10-0-3-17 | sed -n '/Conditions/,/Events/p'
kubectl get lease -n kube-node-lease ip-10-0-3-17 -o yaml | grep renewTime

# get on the node without SSH
kubectl debug node/ip-10-0-3-17 -it --image=busybox
# chroot /host
# journalctl -u kubelet --since '30 min ago' | tail -100
# crictl ps -a | head; df -h /var; df -i /var

# prevention: reserve headroom for the node's own daemons
# /var/lib/kubelet/config.yaml
systemReserved:  { cpu: 200m, memory: 512Mi, ephemeral-storage: 2Gi }
kubeReserved:    { cpu: 200m, memory: 768Mi, ephemeral-storage: 2Gi }
evictionHard:    { memory.available: 500Mi, nodefs.available: 10% }
imageGCHighThresholdPercent: 80