Jenkins Interview Questions and Answers

Answer

The controller is the JVM that owns JENKINS_HOME: it stores job configuration, build records, credentials, plugin state and the queue, it serves the web UI and REST API, it schedules builds, and it evaluates the CPS Groovy program that drives every Pipeline build. What it should not do is run build steps. Every `sh`, `bat` or `checkout` step is dispatched to an executor on some node, and the built-in node (formerly called master) is just a node that happens to live inside the controller JVM.

Setting its executor count to 0 under Manage Jenkins, Nodes, Built-In Node is standard hardening for three reasons. First, security: a build running on the built-in node has direct filesystem access to JENKINS_HOME, which means it can read secrets/master.key, credentials.xml and every job config, so any repository with a Jenkinsfile effectively becomes an admin. Second, stability: a build that fills the disk or forks a runaway process takes the controller down with it, not just one agent.

Third, performance: build work competes with the controller for heap and CPU, and the resulting garbage collection pauses show up as a UI that freezes for ten seconds at a time. The pattern interviewers want to hear is controller for orchestration, ephemeral agents for work, with `agent none` at the top of the Jenkinsfile and explicit per-stage agents underneath.

Answer

A Freestyle job stores its whole definition as XML in JENKINS_HOME/jobs/<name>/config.xml, edited only through the web UI. That means the build definition is not versioned with the code, cannot be reviewed, cannot differ per branch, and disappears if the controller is lost without a backup. A Pipeline job stores a tiny config that points at a Jenkinsfile in the repository, so the build definition travels with the commit that changed it.

Practical consequences interviewers probe: with a Jenkinsfile you get branch-specific behaviour for free through Multibranch Pipelines, you get pull-request builds that use the PR author's own pipeline changes, you can review CI changes in the same MR as the code change, and a controller rebuild restores every job from source. You also gain durability, a Pipeline build can survive a controller restart and resume, which a Freestyle job cannot. What you give up is the point-and-click plugin UI: many older plugins expose a Freestyle build step but only a partially documented Pipeline step, and you sometimes have to look up the Pipeline Syntax generator at /pipeline-syntax to find the right invocation.

Visualisation also changed: Blue Ocean, which used to be the recommended pipeline view, is in maintenance mode and receives no new features, so modern installs use the Pipeline: Graph View plugin or the classic Stage View instead. In an interview, frame the migration as configuration-as-code rather than as a syntax preference.

Answer

Both run on the same Pipeline engine and both are Groovy underneath. Declarative wraps the whole build in a `pipeline { }` block with a fixed grammar: known directives (`agent`, `environment`, `options`, `parameters`, `triggers`, `tools`, `stages`, `post`) in known positions. Because the grammar is fixed, Jenkins can validate the file before running it, render it properly in the stage view, and give you `post` conditions, `when`, `matrix` and restart-from-stage.

Scripted starts with `node { }` and is plain CPS Groovy: you get `if`, `for`, `try/catch` and arbitrary method calls, but no validation, no automatic stage-level post conditions, and much easier ways to shoot yourself. The practical rule used on most teams is Declarative by default, with a `script { }` block for the occasional imperative bit, and a shared library for anything longer than about fifteen lines of Groovy. Reach for full Scripted only when the shape of the pipeline is genuinely dynamic, for example when you have to build a `parallel` map whose keys come from reading a config file at runtime, and even that is usually better solved by generating the map inside a `script` block or a shared library step.

A common interview trap: candidates say Declarative cannot do loops. It can, inside `script { }`. The honest answer is that Declarative constrains where imperative code may appear, which is the point.

// Declarative: fixed grammar, validated, stage-view friendly
pipeline {
  agent { label 'linux' }
  options { timestamps(); timeout(time: 30, unit: 'MINUTES') }
  stages {
    stage('Build') {
      steps {
        sh './gradlew clean build -x test'
        script {
          // imperative escape hatch, keep it short
          env.SHORT_SHA = sh(script: 'git rev-parse --short HEAD', returnStdout: true).trim()
        }
      }
    }
  }
  post { always { junit 'build/test-results/**/*.xml' } }
}

// Scripted equivalent: plain CPS Groovy, no grammar guardrails
node('linux') {
  stage('Build') {
    checkout scm
    sh './gradlew clean build -x test'
  }
}

Answer

The minimum valid Declarative pipeline is `pipeline { agent ... stages { stage('name') { steps { ... } } } }`. Four things are mandatory: the top-level `pipeline` block, exactly one top-level `agent` directive (which may be `agent none`), a `stages` block, and at least one `stage` containing a `steps` block. Every stage must have a name and must contain exactly one of `steps`, `stages` (nested), `parallel`, or `matrix`.

If you put a step such as `sh` directly inside `stage` rather than inside `steps`, the Declarative parser rejects the file before the build starts with a message of the form 'Expected one of "steps", "stages", "parallel", or "matrix" for stage "Build"'. That early failure is a feature: the whole file is parsed and validated up front, so a typo in stage 40 fails in seconds instead of after a 20-minute build. The same validator rejects unknown directives, misplaced `post` blocks and `environment` entries that are not simple key-value assignments.

Other frequent parse errors worth recognising in an interview: 'Undefined section "stage"' means you forgot the `stages` wrapper, and 'No such DSL method "xyz" found among steps' means the step exists in no installed plugin, which is nearly always a missing plugin rather than a syntax error. Note that Declarative also forbids arbitrary Groovy at the top level, variable assignment outside `environment` or `script` will not parse.

pipeline {
  agent any
  stages {
    stage('Checkout') {
      steps { checkout scm }
    }
    stage('Test') {
      // WRONG: sh directly under stage fails validation
      // sh 'npm test'
      steps { sh 'npm ci && npm test' }
    }
    stage('Fan out') {
      parallel {
        stage('unit')  { steps { sh 'npm run test:unit' } }
        stage('lint')  { steps { sh 'npm run lint' } }
      }
    }
  }
}

Answer

`agent` decides where a stage's steps execute and, importantly, when a workspace and an executor are allocated. `agent any` takes any online node with a free executor. `agent { label 'linux && jdk21' }` uses a label expression, and label expressions support `&&`, `||` and `!`, which is how teams pin builds to nodes with the right toolchain or the right data-centre. `agent none` at the top level allocates nothing, so every stage must declare its own agent, this is the recommended pattern because it keeps an executor from being held during `input` approvals. `agent { docker { image 'maven:3.9-eclipse-temurin-21'; args '-v $HOME/.m2:/root/.m2' } }` runs the stage inside a container on a node that has Docker, mounting the workspace automatically. `agent { dockerfile { filename 'ci/Dockerfile'; dir '.' } }` builds the image from the repository first, useful when the toolchain itself is versioned with the code. `agent { kubernetes { yaml ... } }` provisions a throwaway pod per build through the Kubernetes plugin. Two gotchas interviewers like: a stage-level agent re-checks-out the source by default, so if you skip that with `options { skipDefaultCheckout() }` you must run `checkout scm` yourself, and the workspace on a new agent is empty, so anything produced in an earlier stage has to move through `stash`/`unstash` or an artifact repository rather than being assumed present on disk.

pipeline {
  agent none
  stages {
    stage('Build') {
      agent { docker { image 'maven:3.9-eclipse-temurin-21'; args '-v $HOME/.m2:/root/.m2' } }
      steps {
        sh 'mvn -B -DskipTests package'
        stash name: 'jar', includes: 'target/*.jar'
      }
    }
    stage('Approve') {
      // no agent: does not hold an executor while waiting
      steps { timeout(time: 1, unit: 'HOURS') { input message: 'Deploy to prod?' } }
    }
    stage('Deploy') {
      agent { label 'deploy && ap-south-1' }
      steps { unstash 'jar'; sh './deploy.sh' }
    }
  }
}

Answer

`environment` sets environment variables for the scope it is declared in. At the top level of `pipeline` it applies to every stage, inside a `stage` it applies to that stage only, and the stage-level block wins on conflict. Values are strings and support Groovy interpolation of other env vars, so `VERSION = "1.0.${env.BUILD_NUMBER}"` works.

The special `credentials()` helper looks up a credential by ID from the Credentials store and binds it to the variable, but it only supports a few types and behaves differently per type. Secret text binds the secret to the named variable. Username/password binds three variables: `MY_CRED` as `user:password`, plus `MY_CRED_USR` and `MY_CRED_PSW`.

Secret file binds the variable to a path of a temporary file that Jenkins deletes at the end. SSH private key credentials are not supported by `credentials()`, you need `withCredentials([sshUserPrivateKey(...)])` for those. Two production gotchas: the values are masked in the console log only when they appear verbatim, so if your build base64-encodes or URL-encodes a secret and prints it, the masking will not catch it, and if the credential ID does not exist the build fails at start with 'Could not find credentials entry with ID'. Prefer folder-scoped credentials over global ones so a team's Jenkinsfile cannot bind another team's production keys just by guessing an ID.

pipeline {
  agent any
  environment {
    APP     = 'billing-api'
    VERSION = "1.4.${env.BUILD_NUMBER}"
    // secret text -> single variable
    SONAR_TOKEN = credentials('sonar-token')
    // username/password -> DOCKER_USR and DOCKER_PSW as well
    DOCKER = credentials('dockerhub-bot')
  }
  stages {
    stage('Publish') {
      environment { REGISTRY = 'registry.example.co.in' }
      steps {
        sh 'echo $DOCKER_PSW | docker login $REGISTRY -u $DOCKER_USR --password-stdin'
        sh 'docker build -t $REGISTRY/$APP:$VERSION . && docker push $REGISTRY/$APP:$VERSION'
      }
    }
  }
}

Answer

`post` attaches follow-up steps that run after a `stage` or after the whole `pipeline`, regardless of how the body ended. The available conditions are `always`, `changed`, `fixed`, `regression`, `aborted`, `failure`, `success`, `unstable`, `notBuilt` and `cleanup`, and Declarative evaluates them in exactly that documented order, with `cleanup` always last. So in a failed build, `always` runs first, then `failure`, then `cleanup`. `changed` fires when the current result differs from the previous build, `fixed` when the previous build failed or was unstable and this one succeeded, and `regression` when the previous build was better than this one, those three are what you wire notifications to if you do not want Slack spam on every green build.

Two behaviours interviewers check. First, steps inside `post` still need a workspace for anything filesystem-related, so if your pipeline used `agent none`, a `post { always { junit '**/target/*.xml' } }` at pipeline level will fail with a missing FilePath context, you either put the `post` on the stage that had the agent, or wrap it in a `node` block. Second, a failure inside `post` marks the build failed even if the body succeeded, which is why cleanup steps are usually wrapped in `catchError` or `warnError`. Use `cleanup` for `deleteDir()` and `cleanWs()` so it runs after the notification steps have read whatever they needed.

post {
  always {
    junit allowEmptyResults: true, testResults: 'build/test-results/**/*.xml'
  }
  fixed {
    slackSend channel: '#ci-billing', color: 'good', message: "Back to green: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
  }
  regression {
    slackSend channel: '#ci-billing', color: 'danger', message: "Broke: ${env.BUILD_URL}"
  }
  unstable {
    echo 'Tests failed but the build itself compiled'
  }
  cleanup {
    cleanWs(deleteDirs: true, notFailBuild: true)
  }
}

Answer

`when` decides at runtime whether a stage's steps execute. The built-in conditions include `branch 'main'` (glob or a `pattern`/`comparator` pair), `buildingTag()`, `tag 'v*'`, `changeRequest()` for pull requests, `changeset '**/*.sql'` for path-based triggering, `environment name: 'DEPLOY', value: 'true'`, `equals`, `expression { }` for arbitrary Groovy returning a boolean, plus `allOf`, `anyOf` and `not` for composition. By default `when` is evaluated after the stage's agent has been allocated and after `input` has been answered, which surprises people: a skipped stage can still spin up a Kubernetes pod and then do nothing, wasting thirty seconds and a node slot per build. `beforeAgent true` moves the evaluation before agent allocation, `beforeInput true` moves it before the input prompt, and `beforeOptions true` before stage options such as `timeout` are applied.

On any pipeline with containerised agents you should set `beforeAgent true` on every conditional deploy stage as a matter of routine. One more subtlety: `branch` only works where a branch name exists, that is, in a Multibranch Pipeline where `env.BRANCH_NAME` is set. In a plain Pipeline job pointed at a single repository there is no `BRANCH_NAME`, so `when { branch 'main' }` silently never matches, and candidates lose a lot of time to this. Use `expression { env.GIT_BRANCH == 'origin/main' }` in that situation.

stage('Deploy to prod') {
  when {
    beforeAgent true
    beforeInput true
    allOf {
      branch 'main'
      not { changeRequest() }
      expression { return currentBuild.currentResult == 'SUCCESS' }
    }
  }
  agent { label 'deploy' }
  input { message 'Ship to production?'; submitter 'release-managers' }
  steps { sh './deploy.sh production' }
}

stage('DB migration') {
  when { beforeAgent true; changeset '**/migrations/*.sql' }
  steps { sh 'flyway migrate' }
}

Answer

The `parameters` directive declares inputs the user supplies when clicking Build with Parameters. The core types are `string`, `text`, `booleanParam`, `choice`, `password`, and `file`, and the Active Choices plugin adds dynamic dropdowns computed by Groovy. Values arrive in the `params` map, so `params.ENVIRONMENT`, with types preserved: `booleanParam` gives a real Boolean, `choice` and `string` give Strings.

Jenkins also injects them as environment variables, but reading `env.ENVIRONMENT` gives you a String even for booleans, which is why `if (env.DRY_RUN)` is always truthy and `if (params.DRY_RUN)` is correct. The classic gotcha is that the `parameters` block is part of the Jenkinsfile, and Jenkins only learns about it by running the build. So the first build after you add or change a parameter runs with the old parameter set (often with the parameter simply missing and `params.X` returning null), and only the second build shows the new form.

On Multibranch jobs the same applies per branch. Defensive teams write `params.TIMEOUT ?: '30'` so the first run does not NPE, and note it in the PR description. Two more points worth raising: parameter values are stored in the build record and are visible to anyone who can see the build, so never pass secrets through a `string` parameter, use `password` or, better, a credential; and a parameterised job triggered by a webhook uses defaults unless the trigger explicitly supplies values.

pipeline {
  agent any
  parameters {
    choice(name: 'ENVIRONMENT', choices: ['staging', 'preprod', 'production'], description: 'Target')
    booleanParam(name: 'DRY_RUN', defaultValue: true, description: 'Plan only')
    string(name: 'IMAGE_TAG', defaultValue: '', description: 'Blank = build from HEAD')
  }
  stages {
    stage('Deploy') {
      steps {
        script {
          // params.DRY_RUN is a real Boolean; env.DRY_RUN is the string 'true'
          def flag = params.DRY_RUN ? '--dry-run' : ''
          def tag  = params.IMAGE_TAG ?: env.GIT_COMMIT.take(8)
          sh "./deploy.sh ${params.ENVIRONMENT} ${tag} ${flag}"
        }
      }
    }
  }
}

Answer

`sh 'command'` writes your script to a temporary file in the workspace's sibling `@tmp` directory, executes it with the node's shell, streams stdout and stderr into the build log, and fails the build if the exit status is non-zero. Because each invocation is a separate shell, state does not carry over: `sh 'cd build'` followed by `sh 'make'` runs make in the workspace root, not in build. Use the `dir('build') { sh 'make' }` step or chain with `&&` inside a single `sh`.

Multi-line scripts use triple quotes, and you should prefer triple single quotes `'''` unless you need Groovy interpolation, because with `"""` the Groovy interpreter substitutes `${VAR}` before the shell ever sees it, which both breaks shell variables and prints secrets into the log. The two named arguments people are always asked about: `returnStdout: true` makes the step return the captured stdout as a String (remember `.trim()`, the trailing newline is included), and `returnStatus: true` makes it return the exit code instead of failing the build. There is also `label:` which sets a friendly name in the Pipeline Graph View, and `encoding:`.

On Windows nodes the equivalents are `bat` and `powershell`, and `pwsh` for PowerShell Core. One behaviour that catches people: by default the generated script does not run with `set -e` semantics across a multi-line body in every shell, so an early failing line in a long `sh ''' ... '''` block may not fail the build. Put `set -euo pipefail` at the top of any multi-line shell step.

// capture output
def sha = sh(script: 'git rev-parse --short HEAD', returnStdout: true).trim()

// tolerate failure and branch on the exit code
def status = sh(script: 'npm audit --audit-level=high', returnStatus: true)
if (status != 0) { unstable('npm audit found high severity issues') }

// multi-line: single quotes so the shell, not Groovy, expands $VARS
sh '''
  set -euo pipefail
  export GRADLE_USER_HOME="$WORKSPACE/.gradle"
  ./gradlew --no-daemon clean build
'''

// working directory does not persist across steps
dir('services/api') {
  sh label: 'API unit tests', script: 'go test ./...'
}

Answer

Jenkins injects a standard set into every build: `BUILD_NUMBER`, `BUILD_ID`, `BUILD_DISPLAY_NAME`, `BUILD_TAG`, `BUILD_URL`, `JOB_NAME`, `JOB_BASE_NAME`, `JENKINS_URL`, `WORKSPACE`, `NODE_NAME`, `NODE_LABELS`, `EXECUTOR_NUMBER` and `JENKINS_NODE_COOKIE`. The Git plugin adds `GIT_COMMIT`, `GIT_BRANCH`, `GIT_URL` and `GIT_PREVIOUS_SUCCESSFUL_COMMIT` once a checkout has happened, note the ordering, they are not available before `checkout scm`. Multibranch Pipelines add `BRANCH_NAME`, and for pull requests `CHANGE_ID`, `CHANGE_TARGET`, `CHANGE_BRANCH`, `CHANGE_AUTHOR`, `CHANGE_TITLE` and `CHANGE_URL`.

Testing `env.CHANGE_ID != null` is the idiomatic way to ask 'is this a PR build'. `currentBuild` is not an env var but a Groovy object exposing `result`, `currentResult`, `previousBuild`, `duration`, `displayName` and `description`, and it is the standard way to set a readable build name such as `currentBuild.displayName = "1.4.${env.BUILD_NUMBER}"`. Two practical notes. `BUILD_ID` used to be a timestamp and is now the same as `BUILD_NUMBER`, older blog posts that tell you to set `BUILD_ID=dontKillMe` to keep a daemon alive are stale, the correct variable is `JENKINS_NODE_COOKIE`. And the full live list for any build is at `<BUILD_URL>injectedEnvVars` if the EnvInject plugin is present, or simply `sh 'env | sort'`, which is what most engineers actually do when debugging. You can also see the global list at `<JENKINS_URL>env-vars.html`.

pipeline {
  agent any
  stages {
    stage('Context') {
      steps {
        script {
          echo "job=${env.JOB_NAME} build=${env.BUILD_NUMBER} node=${env.NODE_NAME}"
          if (env.CHANGE_ID) {
            echo "PR #${env.CHANGE_ID} from ${env.CHANGE_BRANCH} into ${env.CHANGE_TARGET}"
            currentBuild.displayName = "PR-${env.CHANGE_ID}"
          } else {
            echo "branch build: ${env.BRANCH_NAME}"
          }
          currentBuild.description = env.GIT_COMMIT?.take(8)
        }
        sh 'env | sort'
      }
    }
  }
}

Answer

An agent is a separate JVM running `agent.jar` (the Remoting library) that opens a channel to the controller and executes build steps on its behalf. There are two connection directions. Outbound, usually called 'Launch agents via SSH', means the controller SSHes into the machine, copies `agent.jar`, and starts it, this needs the controller to reach the agent on port 22 and needs SSH credentials plus a host key verification strategy in the node config.

Inbound, historically called JNLP, means the agent starts itself and dials the controller, which suits machines behind NAT, in a customer DMZ, or in a Kubernetes cluster where nothing can route inward. Inbound agents connect either over the TCP agent port using the JNLP4-connect protocol or, increasingly, over HTTP(S) with the `-webSocket` flag, which is the right choice when the controller sits behind an ingress or load balancer that will not forward a raw TCP port. Older Remoting protocols (JNLP1 to JNLP3) are removed in current versions, so an agent stuck on an ancient `slave.jar` will simply fail to handshake.

Interviewers commonly ask what breaks: a Remoting version far out of step with the controller logs 'Remoting version mismatch' or refuses to start, an agent whose clock is skewed fails TLS, and the agent needs a compatible JDK because Remoting runs on the agent's Java, not the controller's. Also remember the agent needs a persistent work directory (`-workDir`) for its own logs and caches.

# Inbound agent over WebSocket (works through an ingress, no TCP agent port)
java -jar agent.jar \
  -url https://jenkins.example.co.in/ \
  -webSocket \
  -name build-node-07 \
  -secret @/etc/jenkins/secret-file \
  -workDir /var/lib/jenkins-agent

# Same thing as a systemd unit fragment
# [Service]
# User=jenkins
# ExecStart=/usr/bin/java -XX:+UseSerialGC -Xmx256m -jar /opt/jenkins/agent.jar \
#   -url https://jenkins.example.co.in/ -webSocket -name build-node-07 \
#   -secret @/etc/jenkins/secret-file -workDir /var/lib/jenkins-agent
# Restart=always

Answer

JENKINS_HOME is the single source of truth for a controller. The important entries are `config.xml` (global config, security realm, authorization strategy), `jobs/<name>/config.xml` plus `jobs/<name>/builds/` (build records and logs), `plugins/` (the installed .jpi files and their pinned versions), `users/` (user records when using the internal database), `nodes/` (agent definitions), `secrets/` including `master.key` and `hudson.util.Secret`, `credentials.xml`, `init.groovy.d/` (startup Groovy hooks), and `workspace/` on the built-in node. The minimum restorable backup is `config.xml`, `jobs/*/config.xml`, `nodes/`, `users/`, `credentials.xml`, `secrets/` and a pinned list of plugin versions.

Two things people get wrong. First, `credentials.xml` is encrypted with the keys in `secrets/`, so backing up one without the other gives you a file full of ciphertext you can never decrypt, restore both or restore neither. Second, build history and `workspace/` are the bulk of the disk but are usually not worth backing up, excluding `builds/*/archive` and `workspace/` turns a 400 GB backup into a few hundred megabytes.

The modern answer to this question in 2026 is that you should not be restoring a controller from a tarball at all: keep the whole configuration in a Configuration as Code YAML file plus a `plugins.txt`, build the controller as a container image, and treat only `credentials.xml` plus `secrets/` (or an external secrets backend) as stateful. Interviewers explicitly listen for that shift.

# Practical rsync backup that skips the bulky, disposable parts
rsync -a --delete \
  --exclude 'workspace/' \
  --exclude 'caches/' \
  --exclude 'jobs/*/builds/*/archive/' \
  --exclude 'war/' \
  --exclude 'plugins/*/' \
  /var/lib/jenkins/ /backup/jenkins-home/

# Record exactly which plugin versions were installed
java -jar jenkins-cli.jar -s "$JENKINS_URL" -auth @token list-plugins \
  | awk '{print $1":"$NF}' | sed 's/[()]//g' | sort > /backup/plugins.txt

# secrets/ + credentials.xml must be backed up together or not at all
tar czf /backup/secrets.tgz -C /var/lib/jenkins secrets credentials.xml

Answer

Jenkins is mostly plugins, and installing them by clicking Manage Plugins on a running controller is how estates become unreproducible. Every plugin declares required Jenkins core version and dependencies on other plugins with a minimum version. When you install or upgrade one from the UI, the update centre resolves those transitively, but it will happily leave you in a state where plugin A needs `workflow-cps` 3.9 and plugin B pins an older one, and after restart the log shows lines like 'Failed to Load Plugin: <name> v1.2 (dependency errors: <other> v2.5 is older than required v2.9)'.

The affected plugin is disabled and, if a job used its steps, builds start failing with 'No such DSL method'. The reproducible approach is `jenkins-plugin-cli` (the Plugin Installation Manager Tool bundled in the official images) driven by a `plugins.txt` file, run at image build time so the container starts with a frozen, resolved plugin set. Pinning to the Jenkins plugin BOM for your core line avoids most conflicts because the BOM is tested as a set. Other things worth saying: always check the plugin's health score and last release date on plugins.jenkins.io before adopting it, a lot of once-popular plugins are unmaintained; upgrade plugins in a staging controller restored from production config first; and remember that a core upgrade can force plugin upgrades, which is exactly what happened to installs crossing the Java 17 and Jakarta EE boundary.

# plugins.txt (versions pinned, or 'latest' only in dev)
# workflow-aggregator:600.vb_57cdd26fdd7
# git:5.7.0
# configuration-as-code:1932.v75cb_b_f1b_698d
# kubernetes:4308.vd0env
# credentials-binding:687.v619cb_15e923f

FROM jenkins/jenkins:lts-jdk21
USER root
RUN apt-get update && apt-get install -y --no-install-recommends git curl && rm -rf /var/lib/apt/lists/*
USER jenkins
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt --latest false
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"
ENV CASC_JENKINS_CONFIG=/var/jenkins_conf/casc.yaml
COPY casc.yaml /var/jenkins_conf/casc.yaml

Answer

They solve different problems and both push data through the controller, which is the detail interviewers care about. `stash name: 'x', includes: '...'` creates a temporary tar of workspace files stored on the controller and readable by `unstash 'x'` on any other agent in the same build. It is deleted when the build ends. Use it to move a build output from a build stage on one agent to a deploy stage on another, or to share a compiled binary across parallel test shards. `archiveArtifacts artifacts: 'target/*.jar', fingerprint: true` copies files into the build record permanently (under `builds/<n>/archive`), makes them downloadable from the build page and the REST API, and with `fingerprint: true` records an MD5 so Jenkins can later tell you which builds produced or consumed that exact file.

Use it for things a human or another job will fetch later. The shared failure mode is size. Both write through the controller's disk and heap, so stashing a 2 GB `node_modules` directory or archiving every Docker layer will fill the controller disk and slow every other build on the instance.

The guidance most platform teams settle on is: stash only small build outputs (a jar, a tarball of `dist/`), never dependency caches; archive only release artefacts, and for anything large push to Nexus, Artifactory, S3 or a container registry and archive just the coordinates or the digest. Also set `allowEmptyArchive` deliberately, the default fails the build when the glob matches nothing, which is usually what you want.

stage('Build') {
  agent { label 'build' }
  steps {
    sh './gradlew bootJar'
    // small, temporary, this build only
    stash name: 'app-jar', includes: 'build/libs/*.jar'
    // permanent, downloadable, fingerprinted for traceability
    archiveArtifacts artifacts: 'build/libs/*.jar',
                     fingerprint: true,
                     onlyIfSuccessful: true
  }
}

stage('Deploy') {
  agent { label 'deploy' }
  steps {
    unstash 'app-jar'
    sh 'scp build/libs/*.jar app@10.0.3.11:/opt/app/'
  }
}

Answer

The `junit` step parses JUnit-format XML and attaches the results to the build: trend graphs, a test list, failure diffs against the previous build, and flagging of new failures. Almost every test runner can emit that format (Surefire and Failsafe for Maven, Gradle's `test` task, pytest with `--junitxml`, jest-junit, go-junit-report, nunit converters), which is why it is the lingua franca. The key semantic is that `junit` sets the build result to UNSTABLE when there are test failures, not FAILURE.

Jenkins distinguishes three outcomes: SUCCESS, UNSTABLE (the build itself ran fine but a quality signal failed) and FAILURE (a step errored). That distinction drives `post { unstable { ... } }`, the yellow ball in the UI, and downstream `build job:` calls that can be configured to propagate one but not the other. Practical settings you should mention: `allowEmptyResults: true` when a stage legitimately produces no tests, otherwise the step fails; `skipPublishingChecks` when you do not want GitHub checks duplicated; and `keepLongStdio` when you need full test stdout.

Related steps in the same family are `recordIssues` from Warnings Next Generation for static analysis (Checkstyle, SpotBugs, ESLint, golangci-lint) with quality gates that can set UNSTABLE or FAILURE, `publishHTML` for coverage reports, and `recordCoverage` for JaCoCo or Cobertura with delta thresholds against the target branch. Put all of these in `post { always { ... } }` so results are published even when the test command exits non-zero.

stage('Test') {
  steps {
    // do not let a non-zero exit code skip result publishing
    sh 'mvn -B verify -Dmaven.test.failure.ignore=true'
  }
  post {
    always {
      junit testResults: '**/target/surefire-reports/*.xml',
            allowEmptyResults: false,
            keepLongStdio: true
      recordCoverage tools: [[parser: 'JACOCO', pattern: '**/jacoco.xml']],
                     qualityGates: [[threshold: 70.0, metric: 'LINE', baseline: 'PROJECT', criticality: 'UNSTABLE']]
      recordIssues tools: [spotBugs(pattern: '**/spotbugsXml.xml'), checkStyle()],
                   qualityGates: [[threshold: 1, type: 'TOTAL_ERROR', unstable: false]]
    }
  }
}

Answer

The `triggers` directive supports `cron('...')` for time-based builds, `pollSCM('...')` to check the repository on a schedule and build only if something changed, and `upstream(upstreamProjects: 'a,b', threshold: hudson.model.Result.SUCCESS)` to chain jobs. Multibranch jobs add periodic branch indexing, and the real-world default for source changes is a webhook from GitHub, GitLab or Bitbucket hitting `/github-webhook/` or `/gitlab/build_now`, which needs no trigger block at all beyond the SCM configuration. Jenkins cron has five fields like Unix cron but adds the `H` symbol, which stands for hash. `H` tells Jenkins to pick a value in the allowed range deterministically from the job name, so `H 2 * * *` runs every job at a stable but different minute past 2 AM.

Without it, `0 2 * * *` on two hundred jobs means two hundred builds hit the queue in the same second, agents cannot be provisioned fast enough, the controller thrashes, and nightly runs start failing. You can also write ranges, `H/15 * * * *` for roughly every fifteen minutes and `H(0-29) 3 * * 1-5` for a weekday early-morning slot. The other message to give an interviewer is that `pollSCM` does not scale: every poll opens an SCM connection from the controller, so a hundred jobs polling every five minutes is a hundred `git ls-remote` calls per five minutes plus the associated threads. Use webhooks, and keep `pollSCM` only for SCMs that genuinely cannot push events.

pipeline {
  agent any
  triggers {
    // nightly, but at a hashed minute so 200 jobs do not stampede
    cron('H H(1-4) * * *')
    // fallback poll only if webhooks are unavailable
    pollSCM('H/30 * * * *')
    // rebuild when the shared base image job succeeds
    upstream(upstreamProjects: 'platform/base-image/main', threshold: hudson.model.Result.SUCCESS)
  }
  options {
    // ignore commits pushed within 60s of each other
    quietPeriod(60)
  }
  stages { stage('Build') { steps { sh 'make ci' } } }
}

Answer

Every job exposes `POST <JENKINS_URL>/job/<name>/build` and, for parameterised jobs, `POST .../buildWithParameters?FOO=bar`. Authentication is a username plus an API token, never the account password, tokens are created per user under the user's Security page and can be revoked individually. Jenkins protects state-changing requests with CSRF protection, so a bare POST returns HTTP 403 with 'No valid crumb was included in the request'.

You fetch a crumb from `/crumbIssuer/api/json` and send it as the `Jenkins-Crumb` header on the same session, which is why you also need a cookie jar in curl. Recent versions issue the crumb per session, so fetching and posting must share cookies. The alternative for machine callers is the per-job authentication token: enable 'Trigger builds remotely' in the job, set a token, and call `/job/<name>/build?token=<secret>`, which bypasses the crumb requirement but is a static shared secret and should only be used inside a trusted network.

A build queued this way returns 201 with a `Location` header pointing at a queue item, poll `<queue_url>api/json` until `executable.number` appears if you need the build number. The same REST surface with `?tree=` is how you query state efficiently, for example `/job/x/lastBuild/api/json?tree=number,result,duration`. Interviewers often follow up with: how do you avoid a 403 from a script? The answer is crumb plus API token, and check whether your reverse proxy is stripping the header.

JENKINS=https://jenkins.example.co.in
USER=ci-bot
TOKEN=11aabbccddeeff00112233445566778899

# 1. fetch a crumb, keeping the session cookie
CRUMB=$(curl -s -c /tmp/jar -u "$USER:$TOKEN" \
  "$JENKINS/crumbIssuer/api/json" | jq -r '.crumb')

# 2. trigger with parameters, sending the crumb and the same cookie
curl -s -b /tmp/jar -u "$USER:$TOKEN" \
  -H "Jenkins-Crumb: $CRUMB" \
  -X POST "$JENKINS/job/billing-api/job/main/buildWithParameters" \
  --data-urlencode 'ENVIRONMENT=staging' --data-urlencode 'DRY_RUN=false' -i

# 3. read result compactly with the tree parameter
curl -s -u "$USER:$TOKEN" \
  "$JENKINS/job/billing-api/job/main/lastBuild/api/json?tree=number,result,duration"

Answer

A Multibranch Pipeline points at a repository rather than a branch. Branch indexing is the periodic scan (and the webhook-driven scan) where Jenkins lists refs from the SCM, checks which ones contain the configured script path (`Jenkinsfile` by default) and creates, updates or removes a child job for each. Discovery is configured by behaviours: 'Discover branches' with a strategy for whether branches that are also filed as PRs get built, 'Discover pull requests from origin' and 'from forks' with a merge-vs-head strategy, and 'Discover tags'.

The merge strategy matters: 'Merge with target branch revision' builds the result of merging the PR into the target, which is what you want for a real gate, and it sets `CHANGE_TARGET`. Orphaned Item Strategy controls how long a job for a deleted branch survives with its history. Fork PRs are the security hinge, building a `Jenkinsfile` from an untrusted fork means executing attacker-controlled Groovy and shell on your agents with your credentials in scope, so restrict fork discovery to collaborators or require an explicit approval, and never expose production credentials to PR builds. Operationally, the failure everyone hits is indexing cost: an org folder over a GitHub organisation with hundreds of repositories will burn through the GitHub API rate limit and take a long time, so use webhooks to trigger indexing instead of a short scan interval, filter with name patterns or the Basic Branch Build Strategies plugin, and cache the SCM with a reference repository for large clones.

// Same Jenkinsfile serving branch builds and PR builds
pipeline {
  agent any
  stages {
    stage('Verify') {
      steps { sh 'make verify' }
    }
    stage('Publish snapshot') {
      when { beforeAgent true; allOf { not { changeRequest() }; branch 'develop' } }
      steps { sh 'make publish-snapshot' }
    }
    stage('Release') {
      when { beforeAgent true; buildingTag() }
      steps { sh "make release VERSION=${env.TAG_NAME}" }
    }
  }
}

Answer

A shared library is a Git repository with three top-level directories. `vars/` holds global variables, each file `vars/foo.groovy` becomes a step named `foo` whose `call()` method is what runs, and an optional `vars/foo.txt` provides help text shown in the Pipeline Syntax page. `src/` holds regular Groovy classes in a package layout, added to the classpath, used for real logic and unit-testable code. `resources/` holds non-Groovy files loaded with the `libraryResource` step, typically shell scripts, JSON templates or Kubernetes pod YAML. Loading happens three ways. Global Pipeline Libraries are configured in Manage Jenkins and can be marked 'Load implicitly', which makes their `vars/` steps available in every pipeline without any declaration, they are trusted and run outside the Groovy sandbox, so anyone who can push to that repository effectively has admin on the controller.

Folder-level libraries are scoped to a folder, useful for giving one business unit its own library. Dynamic loading with the `library` step happens at runtime and can take a version at runtime. The usual declaration is `@Library('platform-lib@v3') _` at the very top of the Jenkinsfile, where the trailing underscore is required because the annotation must be attached to something. Version can be a branch, a tag or a commit SHA, and pinning to a tag is strongly recommended: teams that point every pipeline at `@main` discover that one bad library commit breaks every build in the organisation at once.

// Repository layout
// vars/buildJavaService.groovy
// vars/buildJavaService.txt
// src/co/in/example/ci/Notifier.groovy
// resources/co/in/example/ci/pod.yaml

@Library('platform-lib@v3.2.0') _        // pinned tag, trailing underscore required
@Library('experimental-lib@feature/x') _  // second library, different version

buildJavaService(
  jdk: '21',
  sonarProject: 'billing-api',
  deployTo: ['staging']
)

// Dynamic load when the version is only known at runtime
library(identifier: "platform-lib@${params.LIB_VERSION}",
        retriever: modernSCM([$class: 'GitSCMSource', remote: 'https://git.example.co.in/ci/platform-lib.git']))

Answer

A file `vars/deployService.groovy` becomes a step called `deployService`. Whatever arguments the caller passes land in `call()`. There are two conventional shapes.

The map shape, `def call(Map config)`, is simplest and reads well: `deployService(app: 'billing', env: 'staging')`. The closure shape, `def call(Closure body)`, lets callers write a small block, and you implement it by creating a config map, setting `body.resolveStrategy = Closure.DELEGATE_FIRST` and `body.delegate = config`, then calling `body()`, after which the map holds whatever the caller assigned. The closure shape looks nicer for multi-line configuration and is what most platform libraries use for job templates.

Inside `call()` you have access to all pipeline steps (`sh`, `echo`, `error`, `withCredentials`) because the script is bound to the pipeline's CPS context, but be careful: methods in `vars/` run through the CPS transformer, so the same serialization rules apply as in the Jenkinsfile, and calling a `src/` class that itself calls pipeline steps requires passing the script object through, conventionally as `this`. Always validate the config and call `error()` with a clear message for missing keys, because a null-pointer deep inside a shared library produces a stack trace that means nothing to the application team consuming it. Finally, keep `vars/` steps thin and put the branching logic in `src/` classes so you can unit-test them with JenkinsPipelineUnit or Spock outside Jenkins.

// vars/deployService.groovy
def call(Closure body) {
  def config = [environment: 'staging', replicas: 2, timeoutMinutes: 10]
  body.resolveStrategy = Closure.DELEGATE_FIRST
  body.delegate = config
  body()

  if (!config.app) { error 'deployService: "app" is required' }

  withCredentials([file(credentialsId: "kubeconfig-${config.environment}", variable: 'KUBECONFIG')]) {
    timeout(time: config.timeoutMinutes, unit: 'MINUTES') {
      sh "kubectl -n ${config.environment} set image deploy/${config.app} app=${config.image}"
      sh "kubectl -n ${config.environment} rollout status deploy/${config.app}"
    }
  }
}

// Jenkinsfile
deployService {
  app         = 'billing-api'
  image       = "registry.example.co.in/billing-api:${env.BUILD_NUMBER}"
  environment = 'production'
  replicas    = 4
}

Answer

`withCredentials` binds credentials to variables for the duration of a block and registers them with the log filter so their literal value is replaced by `****` in the console. The binding types cover what `credentials()` cannot: `string(credentialsId:, variable:)`, `usernamePassword(credentialsId:, usernameVariable:, passwordVariable:)`, `sshUserPrivateKey(credentialsId:, keyFileVariable:, passphraseVariable:, usernameVariable:)`, `file(credentialsId:, variable:)` and `certificate(...)`. The block scoping matters: outside the block the variables are unset, and for `file` and `sshUserPrivateKey` the temporary file is deleted on exit.

Now the leaks, which is what senior interviews focus on. Masking is a literal string replacement on the log stream, so anything that transforms the secret defeats it: base64-encoding it, URL-encoding it, printing it character by character, or letting a tool echo it in a slightly different form. `sh "curl -u user:${PASSWORD} ..."` is a double failure, Groovy interpolates the secret into the command string before the shell runs, so it lands in the `set -x` trace and in the process list where any other process on the agent can read it via `ps`. Always use single quotes so the shell expands the variable, and prefer `--password-stdin` or a credentials file over command-line arguments.

Other leak paths: `env | sort` in a debug step, a crash dump, a `writeFile` of the environment, and archived artefacts that embed a token. Also remember that anyone who can edit a Jenkinsfile in a repository can bind any credential that repository's job has access to, which is the argument for folder-scoped credentials.

withCredentials([
  usernamePassword(credentialsId: 'nexus-deployer', usernameVariable: 'NEXUS_USR', passwordVariable: 'NEXUS_PSW'),
  string(credentialsId: 'sonar-token', variable: 'SONAR_TOKEN'),
  sshUserPrivateKey(credentialsId: 'deploy-key', keyFileVariable: 'SSH_KEY')
]) {
  // WRONG: Groovy interpolation puts the secret in the command line and in ps output
  // sh "curl -u $NEXUS_USR:$NEXUS_PSW https://nexus/repo/ --upload-file app.jar"

  // RIGHT: single quotes, the shell expands it, and stdin instead of argv
  sh '''
    set -euo pipefail
    printf '%s' "$NEXUS_PSW" | curl --user "$NEXUS_USR" --config - \
      --upload-file app.jar https://nexus.example.co.in/repo/
    ssh -i "$SSH_KEY" -o StrictHostKeyChecking=yes deploy@10.0.3.11 'systemctl restart app'
  '''
}

Answer

The Kubernetes plugin registers a cloud that provisions a pod per build (or per stage). Each pod contains an agent container, named `jnlp` by convention, running Remoting and connecting back to the controller, plus whatever tool containers you declare. The workspace is an `emptyDir` volume mounted into every container, so all containers see the same files, and the `container('maven') { ... }` step routes subsequent `sh` steps into that container instead of `jnlp`.

When the build finishes the pod is deleted, which gives you clean, reproducible builds and elastic capacity. The classic failures are worth knowing cold. If you declare your own `jnlp` container and override its image or command, the agent never starts and the build hangs until the pod times out with 'Pod has terminated' or the slave agent 'was not connected in 100 seconds', usually because you replaced the entrypoint that launches Remoting.

If a tool container has no long-running command, Kubernetes restarts it in a crash loop, hence the ubiquitous `command: ['cat']` with `tty: true`. If you omit resource requests, the scheduler packs pods until nodes are starved and builds get OOMKilled with exit code 137, always set requests and limits, and remember the JVM inside the container needs `-XX:MaxRAMPercentage` or an explicit `-Xmx` below the container limit. Also configure `podRetention`, `idleMinutes` for reusing a pod across stages, and the WebSocket connection mode when the controller is behind an ingress that will not pass the TCP agent port.

pipeline {
  agent {
    kubernetes {
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: maven:3.9-eclipse-temurin-21
    command: ['cat']
    tty: true
    resources:
      requests: { cpu: '1', memory: '2Gi' }
      limits:   { cpu: '2', memory: '3Gi' }
    env:
    - name: MAVEN_OPTS
      value: '-XX:MaxRAMPercentage=70'
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    command: ['sleep', 'infinity']
'''
      defaultContainer 'maven'
      idleMinutes 5
    }
  }
  stages {
    stage('Build') { steps { sh 'mvn -B package' } }
    stage('Image') {
      steps {
        container('kaniko') {
          sh '/kaniko/executor --context $WORKSPACE --destination registry.example.co.in/app:$BUILD_NUMBER'
        }
      }
    }
  }
}

Answer

Two mechanisms. The Declarative `agent { docker { image '...' args '...' } }` directive runs the whole stage in a container: Jenkins starts the container with the workspace bind-mounted and the same path inside, then executes each `sh` with `docker exec`. The Scripted equivalent from the Docker Pipeline plugin is `docker.image('node:22').inside('-v ...') { ... }`, plus `docker.build()`, `docker.withRegistry()` and `image.push()`.

The ownership problem is the most common real-world issue. Jenkins runs the container as the agent's own UID by default in `inside`, but if you pass `-u root` or use an image whose entrypoint switches users, files written into the mounted workspace end up owned by root, and the next build on that agent (running as `jenkins`) fails with 'permission denied' when it tries to clean the workspace, or `git clean` leaves stale root-owned directories forever. The fixes are to keep the default UID mapping, pass `args '-u 1000:1000'` explicitly where you know the agent UID, and if the image insists on root, add a `sh 'chown -R 1000:1000 .'` before exiting the container. Two more gotchas: the container inherits the agent's Docker socket only if you mount it, so building images inside a container needs `-v /var/run/docker.sock:/var/run/docker.sock` (docker-out-of-docker) with the security implications that carries, and `alwaysPull true` matters because a floating tag like `node:22` will silently keep using a months-old cached layer on a long-lived agent, so pin by digest for reproducible builds.

pipeline {
  agent none
  stages {
    stage('Test') {
      agent {
        docker {
          image 'node:22-bookworm'
          args '-u 1000:1000 -v $HOME/.npm:/home/node/.npm'
          alwaysPull true
          reuseNode true
        }
      }
      steps { sh 'npm ci && npm test' }
    }
    stage('Image') {
      agent { label 'docker' }
      steps {
        script {
          docker.withRegistry('https://registry.example.co.in', 'registry-creds') {
            def img = docker.build("billing-api:${env.BUILD_NUMBER}", '--pull -f ci/Dockerfile .')
            img.push()
            img.push('latest')
          }
        }
      }
    }
  }
}

Answer

In Declarative, a stage may contain a `parallel` block whose children are themselves stages, each with its own `agent`, `when`, `post` and `steps`. In Scripted you call the `parallel` step with a Map of branch name to closure. All branches run concurrently, each holding its own executor, and the parent stage completes when all branches finish. `failFast true` (or `parallel firstFailure: true` in Scripted, or the pipeline-level `options { parallelsAlwaysFailFast() }`) tells Jenkins to abort the remaining branches as soon as one fails.

The subtlety interviewers probe: aborting is done by interrupting the other branches, so their `post` blocks still run but their result is ABORTED, and any external side effect they had already started, a deployment, a cloud resource, a database migration, is not rolled back. Never use `failFast` on branches that mutate shared state. Two capacity issues are worth raising.

First, parallel branches consume executors, so a matrix of twelve branches on a pool of eight executors serialises anyway and can deadlock if the branches wait on each other. Second, each branch with its own `agent` gets its own workspace, so shared inputs must be stashed and produced outputs unstashed or archived, people frequently write parallel test shards that all assume the build output is already on disk and then wonder why only one branch works. Also note that `parallel` inside a `script` block lets you build the branch map dynamically, which is the standard way to shard a test suite across N agents.

stage('Verify') {
  failFast true
  parallel {
    stage('unit') {
      agent { label 'linux' }
      steps { unstash 'src'; sh 'make test-unit' }
      post { always { junit 'reports/unit/*.xml' } }
    }
    stage('integration') {
      agent { label 'linux && docker' }
      steps { unstash 'src'; sh 'make test-integration' }
    }
    stage('sast') {
      agent { label 'linux' }
      steps { unstash 'src'; sh 'semgrep ci --json -o semgrep.json' }
    }
  }
}

// Scripted: build the branch map at runtime to shard tests
script {
  def shards = [:]
  (1..4).each { i ->
    shards["shard-${i}"] = {
      node('linux') { unstash 'src'; sh "./run-tests.sh --shard ${i} --of 4" }
    }
  }
  parallel shards
}

Answer

`matrix` generates one parallel cell per combination of the declared axes, so a two-axis matrix of three JDKs and three operating systems produces nine cells without you writing nine stages. Each cell gets the axis values as environment variables, can have its own `agent` (usually a label built from the axis values), and shares a common `stages` block defined once under the matrix. `excludes` removes specific combinations, which is how you drop the combinations that make no sense, for example JDK 17 on the ARM image you have not published. You can also add per-cell `environment`, `when` and `post`.

Use `matrix` when the work is genuinely the same across a product of dimensions: cross-platform compilation, library compatibility testing against several framework versions, or Terraform plans across regions. Use hand-written `parallel` when the branches do different things, unit tests and linting and SAST are not axes of anything and expressing them as a matrix is contrived. The operational warning is combinatorial explosion: a three-axis matrix is easy to write and can quietly demand 27 executors, saturate your agent pool, and starve every other job on the controller.

Pair matrices with `options { throttle(...) }` from the Throttle Concurrent Builds plugin, or cap concurrency by using a dedicated label with a limited node count. Also remember that each cell runs a separate checkout unless you stash first, and that the matrix stage view can become very wide in the classic UI.

stage('Cross-build') {
  matrix {
    axes {
      axis { name 'JDK';      values '17', '21' }
      axis { name 'PLATFORM'; values 'linux-amd64', 'linux-arm64', 'windows' }
    }
    excludes {
      exclude {
        axis { name 'PLATFORM'; values 'windows' }
        axis { name 'JDK';      values '17' }
      }
    }
    agent { label "${PLATFORM}" }
    environment { GRADLE_USER_HOME = "${WORKSPACE}/.gradle" }
    stages {
      stage('Compile') {
        steps { sh "./gradlew -Dorg.gradle.java.home=/opt/jdk-${JDK} assemble" }
      }
      stage('Test') {
        steps { sh './gradlew test' }
        post { always { junit allowEmptyResults: true, testResults: '**/test-results/**/*.xml' } }
      }
    }
  }
}

Answer

They solve two different concurrency problems and interviewers like candidates who separate them. `lock('resource-name')` from the Lockable Resources plugin is mutual exclusion: only one build at a time may hold the named resource, others queue. Use it around anything with shared external state, a staging environment, a database migration, a Terraform state file, a hardware test rig. It supports `label` and `quantity` for pools ('any 1 of the 4 device labs'), `inversePrecedence` to give the newest build the lock first, and `variable:` to learn which resource you got. `milestone(N)` is ordering, not exclusion: when a build passes milestone N, Jenkins aborts any older build of the same job that has not yet passed it.

That is the fix for the out-of-order deploy problem, where commit 100 finishes its slow tests after commit 101 already deployed, and then overwrites production with the older artefact. Milestones cost nothing and should be placed before every deploy stage. Together the pattern is: `milestone()` immediately before the deploy stage to kill stale builds, then `lock()` around the deploy itself so two different branches cannot deploy simultaneously.

Complementary options are `disableConcurrentBuilds()` in `options`, which serialises the entire job (blunt but simple, and `abortPrevious: true` makes it cancel the running build instead of queueing), and `throttle()` from Throttle Concurrent Builds for cross-job caps. A gotcha: a build waiting on `lock()` inside a `node` block is still holding an executor, so acquire the lock outside the agent allocation where possible.

pipeline {
  agent none
  options { disableConcurrentBuilds(abortPrevious: true) }
  stages {
    stage('Build')  { agent { label 'build' }; steps { sh 'make dist'; stash name: 'dist', includes: 'dist/**' } }
    stage('Test')   { agent { label 'build' }; steps { unstash 'dist'; sh 'make test' } }

    stage('Deploy staging') {
      steps {
        // kill any older build that has not reached this point
        milestone(ordinal: 10, label: 'pre-deploy')
        // one deploy to staging at a time, across every branch and job
        lock(resource: 'staging-env', inversePrecedence: true) {
          node('deploy') {
            unstash 'dist'
            sh './deploy.sh staging'
          }
        }
      }
    }
    stage('Device tests') {
      steps {
        lock(label: 'android-device', quantity: 1, variable: 'DEVICE') {
          node('deploy') { sh "./run-device-tests.sh --device ${env.DEVICE}" }
        }
      }
    }
  }
}

Answer

`input` pauses the build and waits for a human to click Proceed or Abort. If the surrounding stage has an agent, the build is holding an executor and a workspace for the entire wait, which might be minutes, hours, or until someone comes back from lunch. On a pool of ten executors, three pipelines waiting on production approval means thirty percent of your capacity is doing nothing, and in a Kubernetes setup it means an idle pod burning cluster resources.

The correct pattern is `agent none` at the pipeline level with the approval in its own agentless stage, or the Declarative `input` directive on a stage, which by default evaluates before the agent is allocated. Beyond capacity there are correctness details worth mentioning. `submitter: 'release-managers'` restricts who may approve, and `submitterParameter: 'APPROVER'` captures who did, which auditors ask for in Indian BFSI environments. `input` can collect parameters too, returning a single value or a Map depending on how many parameters you declared, a frequent source of ClassCastException when someone adds a second parameter and the assignment still expects a String. Always wrap it in `timeout`, otherwise an unanswered approval keeps the build in the running state indefinitely and it will still be there weeks later; combine with `catchError(buildResult: 'SUCCESS', stageResult: 'ABORTED')` if a timeout should mean 'skip the deploy' rather than 'fail the build'. Note that a build aborted at an `input` after a controller restart may not resume cleanly if durability is set to PERFORMANCE_OPTIMIZED.

pipeline {
  agent none
  stages {
    stage('Build') { agent { label 'build' }; steps { sh 'make dist' } }

    stage('Approve production') {
      // no agent: nothing is held while we wait
      steps {
        timeout(time: 4, unit: 'HOURS') {
          script {
            def resp = input(
              message: 'Promote build to production?',
              ok: 'Promote',
              submitter: 'release-managers,platform-leads',
              submitterParameter: 'APPROVER',
              parameters: [choice(name: 'REGION', choices: ['ap-south-1', 'ap-southeast-1'], description: '')]
            )
            env.APPROVER = resp['APPROVER']
            env.REGION   = resp['REGION']
          }
        }
      }
    }
    stage('Deploy') {
      agent { label 'deploy' }
      steps { sh "./deploy.sh production ${env.REGION}  # approved by ${env.APPROVER}" }
    }
  }
}

Answer

`timeout(time: 20, unit: 'MINUTES') { }` aborts the enclosed block, and by default aborts the build. `timeout(activity: true, time: 10, unit: 'MINUTES')` is the more useful variant for chatty builds: it triggers only after ten minutes of no log output, so a slow but progressing build is not killed. `retry(3) { }` reruns the block on failure and is right for genuinely flaky external calls, a registry push, an artifact download, never for tests, because retrying tests hides real bugs and interviewers will say so. `retry` combined with `sleep` is the poor man's backoff. `catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE', message: '...') { }` runs a block, and when it throws, sets the build and stage results to the values you specify instead of failing outright, this is how you make an optional step (a nightly performance run, a documentation publish) degrade rather than block a release. `warnError('message') { }` is shorthand for `catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE')`. `unstable('reason')` sets the result directly without an exception, useful after inspecting the exit code from `sh(returnStatus: true)`. Two important behaviours: `currentBuild.result` starts as null, not 'SUCCESS', so `if (currentBuild.result == 'SUCCESS')` is false during the build, use `currentBuild.currentResult`; and Jenkins never downgrades a result, once a build is FAILURE, setting it to UNSTABLE has no effect. Also, an abort caused by `timeout` throws `FlowInterruptedException`, so a broad `try/catch (Exception e)` will swallow the abort and keep the build going, which is almost never what you want.

stage('Publish') {
  steps {
    // kill only if nothing is logged for 10 minutes
    timeout(activity: true, time: 10, unit: 'MINUTES') {
      retry(3) {
        sh 'docker push registry.example.co.in/billing-api:$BUILD_NUMBER'
      }
    }

    // optional quality step: degrade the build, do not block the release
    catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE', message: 'perf suite regressed') {
      sh './gradlew jmh'
    }

    warnError('docs publish failed') { sh './publish-docs.sh' }

    script {
      def rc = sh(script: 'npm audit --audit-level=critical', returnStatus: true)
      if (rc != 0) { unstable('critical CVEs found in dependencies') }
      echo "result so far: ${currentBuild.currentResult}"   // not currentBuild.result
    }
  }
}

Answer

The configuration-as-code plugin lets you describe the entire controller, security realm, authorization strategy, clouds, agents, tools, global libraries, plugin settings and seed jobs, in one or more YAML files, replacing hand-clicking in Manage Jenkins. You point `CASC_JENKINS_CONFIG` at a file, a directory, or an HTTP URL, and the plugin applies it at boot and on demand from Manage Jenkins, Configuration as Code, Reload. Two features make adoption practical: the Export button generates YAML from your current running configuration, giving you a starting point from a legacy controller, and the View Configuration page documents every schema key the installed plugins expose, which is the only reliable reference because the schema depends entirely on your plugin set.

Secrets never go in the YAML. JCasC resolves `${VARIABLE}` from environment variables, from files in a directory named by `SECRETS` (Kubernetes secret mounts, one file per key), or from a supported secret backend, and `${VAR:-default}` supplies a fallback. In practice on Kubernetes you mount a Secret at `/run/secrets/jenkins` and set `SECRETS=/run/secrets/jenkins`.

The strong version of this answer is the operating model: the controller image is built from a Dockerfile with `plugins.txt` and `casc.yaml` baked in, the UI is read-only for configuration, and any change goes through a pull request. Then a lost controller is a `kubectl apply` away rather than a restore project. The caveat to mention: a JCasC reload does not revert configuration made through the UI unless that key is present in the YAML, so drift is still possible until you enforce it.

jenkins:
  systemMessage: "Managed by JCasC. Do not edit in the UI."
  numExecutors: 0
  mode: EXCLUSIVE
  securityRealm:
    ldap:
      configurations:
        - server: "ldaps://ldap.example.co.in"
          rootDN: "dc=example,dc=co,dc=in"
          managerDN: "cn=jenkins,ou=svc,dc=example,dc=co,dc=in"
          managerPasswordSecret: "${LDAP_BIND_PASSWORD}"
  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: "admin"
            permissions: ["Overall/Administer"]
            entries: [{ group: "platform-admins" }]
  clouds:
    - kubernetes:
        name: "k8s"
        namespace: "jenkins-agents"
        jenkinsUrl: "http://jenkins.jenkins.svc.cluster.local:8080"
        containerCapStr: "40"
unclassified:
  location:
    url: "https://jenkins.example.co.in/"
  globalLibraries:
    libraries:
      - name: "platform-lib"
        defaultVersion: "v3.2.0"
        implicit: false

Answer

Every build writes a directory under `jobs/<name>/builds/<n>/` containing `build.xml`, `log`, the Pipeline flow-node store, `changelog.xml` and any archived artefacts. Nothing deletes these unless you tell Jenkins to. On a controller with two hundred jobs running hourly, that is easily a terabyte within a year, and the pain is not only disk: Jenkins lazily loads build records, but the more builds a job has, the slower job page loads and the more heap the loaded records occupy.

The primary control is the build discarder, `options { buildDiscarder(logRotator(numToKeepStr: '30', daysToKeepStr: '30', artifactNumToKeepStr: '5', artifactDaysToKeepStr: '7')) }`. Note the two pairs: the first keeps whole build records, the second keeps only the archived artefacts while retaining the metadata and logs, which is usually what you want because a 400 MB jar is disposable but the test trend is not. For Multibranch, the parent folder has its own discarder plus the Orphaned Item Strategy for deleted branches.

Additional levers: cap console log size with the Build Log Rotator or by not printing dependency-resolution noise, use `cleanWs()` so workspaces on agents do not accumulate, and remember archived artefacts should mostly be in Nexus, Artifactory or S3 rather than the controller. For an inherited estate, the Disk Usage plugin and a Groovy script through the script console will tell you the top offenders quickly. Also worth saying: deleting build records that a fingerprint or downstream job references leaves broken links, so communicate retention changes before applying them.

// Per-job, in the Jenkinsfile
options {
  buildDiscarder(logRotator(
    numToKeepStr: '50',          // keep 50 build records
    daysToKeepStr: '60',
    artifactNumToKeepStr: '5',   // but only 5 sets of archived artefacts
    artifactDaysToKeepStr: '14'
  ))
  disableConcurrentBuilds()
  timestamps()
}

// Estate-wide audit from Manage Jenkins > Script Console
Jenkins.instance.getAllItems(Job.class)
  .collect { job -> [job.fullName, job.builds.size(), job.rootDir.directorySize()] }
  .sort { -it[2] }
  .take(20)
  .each { println "${it[0]}\tbuilds=${it[1]}\tbytes=${it[2]}" }

Answer

Pipeline Groovy from a Jenkinsfile runs inside the Script Security sandbox, a whitelist-based interpreter that intercepts every method call and field access and rejects anything not on the approved list. This exists because a Jenkinsfile is code from a repository, and without the sandbox any developer with commit access could call `Jenkins.instance` and become a controller administrator. When your pipeline calls something outside the whitelist you get a build failure reading 'Scripts not permitted to use method <signature>' or a `RejectedAccessException`, and an administrator can approve that exact signature under Manage Jenkins, In-process Script Approval.

Approving is a real security decision, not a formality: signatures like `staticMethod jenkins.model.Jenkins getInstance` or anything on `java.lang.Runtime` hand over the controller, and approving `new java.io.File java.lang.String` gives every pipeline arbitrary filesystem read on the controller. Things that are not sandboxed: Global Pipeline Libraries marked trusted (all global libraries are trusted, folder libraries are not by default), `init.groovy.d` scripts, the Script Console, and Job DSL scripts depending on configuration, which is exactly why write access to the shared library repository is equivalent to Jenkins admin. The practical guidance for teams is: do not chase approvals, move the logic that needs privileged Groovy into a trusted shared library step with a narrow, reviewed interface, and keep the Jenkinsfile to sandbox-safe calls. In regulated Indian environments this separation is also what auditors expect to see, application teams cannot self-approve script signatures.

Answer

Authentication and authorization are separate settings. The security realm handles who you are (internal user database, LDAP or Active Directory, or SAML/OIDC through a plugin, which is what most Indian enterprises use so access follows the corporate directory). The authorization strategy handles what you can do.

The built-in Matrix Authorization Strategy grants permissions per user or group globally, and its project-based variant lets a job or folder define its own matrix, optionally inheriting from the parent. The Role-based Authorization Strategy plugin is what most multi-team estates use: you define global roles (admin, read-only), item roles matched by a regular expression against the full item name, and agent roles matched against node names, then assign groups from your directory to them. Combined with folders, one folder per business unit and an item role pattern of `billing/.*` gives each team build, cancel and configure rights only inside their own folder.

Details that separate a real operator from someone reciting docs: keep Overall/Read for authenticated users so people can see the instance at all; grant Job/Configure inside folders rather than globally; remember that credentials are scoped to folders too, so a team with folder-level configure rights can bind credentials stored in that folder and nothing above it; keep the agent-to-controller access control enabled so a build cannot read controller files over Remoting; and never grant Overall/RunScripts, which is Script Console access and therefore full admin. Finally, anonymous read should be off on anything internet-facing, which was the aggravating factor in several publicised Jenkins breaches.

Answer

Three mechanisms, and a good answer uses all three. First, the Declarative linter: `java -jar jenkins-cli.jar -s $JENKINS_URL -auth @token declarative-linter < Jenkinsfile` parses the file against the Declarative grammar and reports errors with line numbers, and the same check is exposed over HTTP at `POST /pipeline-model-converter/validate` with the file in a `jenkinsfile` form field, which is easy to call from a pre-commit hook or from a GitHub Action guarding the repository. The linter catches structural problems, misplaced directives, unknown sections, a `steps` block in the wrong place, but it cannot tell you whether a step exists or whether your Groovy is semantically right.

Second, Replay: open a previous build, click Replay, edit the pipeline and any shared library files in the browser, and run it as a one-off without committing. This is by far the fastest debugging loop for a pipeline that fails only on the CI agents, and the resulting script is visible in the build so you can copy it back into the repository. Third, unit tests: JenkinsPipelineUnit mocks the pipeline DSL so you can assert that your shared library called the right steps with the right arguments, and jenkinsfile-runner actually executes a Jenkinsfile in a headless Jenkins for smoke testing. In a mature setup the repository's own CI runs the linter on every pull request, so a syntax error never reaches the shared controller and never wastes an agent slot.

# 1. CLI linter (structural validation, exits non-zero on error)
java -jar jenkins-cli.jar -s "$JENKINS_URL" -auth "$USER:$TOKEN" \
  declarative-linter < Jenkinsfile

# 2. Same check over HTTP, crumb included, easy to run in a pre-commit hook
CRUMB=$(curl -s -c /tmp/jar -u "$USER:$TOKEN" "$JENKINS_URL/crumbIssuer/api/json" | jq -r .crumb)
curl -s -b /tmp/jar -u "$USER:$TOKEN" -H "Jenkins-Crumb: $CRUMB" \
  -F "jenkinsfile=<Jenkinsfile" "$JENKINS_URL/pipeline-model-converter/validate"
# -> "Jenkinsfile successfully validated."  or a list of errors with line numbers

# 3. git hook so nothing invalid is ever pushed
cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
git diff --cached --name-only | grep -q '^Jenkinsfile$' || exit 0
java -jar "$HOME/bin/jenkins-cli.jar" -s "$JENKINS_URL" -auth @"$HOME/.jenkins-token" \
  declarative-linter < Jenkinsfile
EOF
chmod +x .git/hooks/pre-commit

Answer

That error, `MissingContextVariableException`, means a step that needs a workspace or a node ran where none was allocated. The message usually continues with 'Perhaps you forgot to surround the code with a step that provides this, such as: node'. It happens most often in three places: a pipeline-level `post` block on a pipeline declared `agent none`, so `junit`, `archiveArtifacts`, `readFile` or `cleanWs` have no workspace to look at; a shared library step that calls `sh` but is invoked from outside a node context; and a `stage` with `agent none` inheriting a step that needs a FilePath.

The fix is either to move the step into a stage that has an agent, or to wrap it explicitly in `node('label') { ... }` which allocates an executor and workspace. Other context errors follow the same shape: missing `hudson.Launcher` means no node for process execution, and missing `hudson.model.Run` means you called something from outside a build entirely. While we are on classic pipeline errors, the ones every Jenkins engineer should recognise on sight are: 'No such DSL method "xyz" found among steps' (plugin not installed, or a typo, or you are in a `script` block calling a method that does not exist), 'java.io.NotSerializableException' (CPS serialization, discussed separately), 'Scripts not permitted to use ...' (script security), 'hudson.remoting.ChannelClosedException: Channel is already closed' (agent died or was OOM-killed mid-build), and 'ERROR: script returned exit code 143', where 143 is SIGTERM and usually means a `timeout` or an aborted build rather than a real failure.

// BROKEN: agent none at the top, so pipeline-level post has no workspace
pipeline {
  agent none
  stages { stage('T') { agent { label 'linux' }; steps { sh 'make test' } } }
  post { always { junit '**/target/*.xml' } }   // MissingContextVariableException
}

// FIX A: put post on the stage that owns the agent
stage('T') {
  agent { label 'linux' }
  steps { sh 'make test' }
  post { always { junit '**/target/*.xml' } }
}

// FIX B: allocate a node explicitly in the pipeline-level post
post {
  always {
    node('linux') {
      unstash 'reports'
      junit '**/target/*.xml'
    }
  }
}

Answer

Jenkins runs a ProcessTreeKiller at the end of every build. It scans processes on the node and terminates any whose environment contains the `JENKINS_NODE_COOKIE` value that the build injected, which is how it reaps compilers, test servers and daemons that a build left running. That is normally exactly what you want, an orphaned Selenium grid or a stuck Gradle daemon would otherwise leak until the agent runs out of memory.

But when you deliberately want a process to outlive the build, starting an application server for a later job, launching a monitoring sidecar, it is what silently kills it. The fix is to launch the process with that variable overridden, `JENKINS_NODE_COOKIE=dontKillMe nohup ./server &`, which removes it from the kill set. Older documentation says to use `BUILD_ID`, which no longer works because `BUILD_ID` became an alias for `BUILD_NUMBER`.

You can also disable the killer globally with `-Dhudson.util.ProcessTree.disable=true`, which is a bad idea on a shared agent. The workspace side of the same operational story: workspaces persist on agents between builds, which is good for incremental builds and bad for disk. `cleanWs()` from the Workspace Cleanup plugin deletes it, with `patterns` for selective cleanup and `notFailBuild: true` so a cleanup failure does not fail an otherwise green build. Also remember concurrent builds of the same job create `workspace@2`, `@3` directories, so a job that builds a 5 GB monorepo with concurrency enabled quietly needs three times the disk you planned for.

stage('Integration') {
  steps {
    // this survives the build; without the cookie override Jenkins kills it
    sh 'JENKINS_NODE_COOKIE=dontKillMe nohup ./mock-payment-gateway --port 9099 > gw.log 2>&1 &'
    sh './wait-for.sh localhost:9099 -- make test-integration'
  }
  post {
    always {
      sh 'pkill -f mock-payment-gateway || true'
      archiveArtifacts artifacts: 'gw.log', allowEmptyArchive: true
    }
    cleanup {
      cleanWs(
        deleteDirs: true,
        notFailBuild: true,
        patterns: [[pattern: '.gradle/**', type: 'EXCLUDE'], [pattern: '**/*', type: 'INCLUDE']]
      )
    }
  }
}

Answer

Pipeline Groovy does not run as ordinary Groovy. The workflow-cps plugin rewrites your script into continuation-passing style so that execution can be paused and the entire program state written to disk, which is what allows a build to survive a controller restart and resume mid-stage. The consequence is that every local variable that is live across a step boundary must be Java-serializable.

Hold a `java.util.regex.Matcher`, a `JsonSlurper` result graph containing non-serializable nodes, a `File`, or a database connection across an `sh` call and the build dies with `java.io.NotSerializableException: java.util.regex.Matcher`. The idiomatic fixes are to scope the offending object tightly so it is dead before the next step, or to move that code into a `@NonCPS` method. A `@NonCPS` method runs as normal compiled Groovy: no CPS rewriting, so it is fast, it may use closures such as `each`, `collect` and `sort` freely, and its local variables never need to be serializable.

The rules are strict: a `@NonCPS` method must not call any pipeline step (`sh`, `echo`, `checkout`), because there is no continuation to suspend, and it must not be resumable, so it runs to completion in one go. Use it for pure data transformation: parsing JSON or XML, filtering a list, formatting a message. Two more CPS symptoms worth naming: iterating with `.each { }` in CPS code can behave oddly or be slow because each closure invocation is transformed, so a plain `for` loop is preferred in CPS scope, and a very long loop of steps generates thousands of flow nodes, which bloats the build record.

// Fails: Matcher is not serializable and is live across the sh step
// def m = (log =~ /version=(\d+)/)
// sh 'sleep 1'
// echo m[0][1]

// Correct: pure parsing isolated in a @NonCPS method
@NonCPS
String extractVersion(String log) {
  def m = (log =~ /version=([0-9.]+)/)
  return m ? m[0][1] : 'unknown'   // Matcher dies inside this method
}

@NonCPS
List<String> failedModules(String junitXml) {
  new XmlSlurper().parseText(junitXml)
    .'**'.findAll { it.name() == 'testsuite' && it.@failures.toInteger() > 0 }
    .collect { it.@name.toString() }        // closures are fine here
}

node('linux') {
  def out = sh(script: './build.sh', returnStdout: true)
  def version = extractVersion(out)         // safe: nothing unserializable escapes
  echo "built ${version}"
  // NEVER call sh, echo or checkout inside a @NonCPS method
}

Answer

Every Pipeline build persists two things: the CPS program state in `builds/<n>/program.dat` and the graph of flow nodes representing the steps that have executed. On a clean restart Jenkins reloads that state and resumes the build where it stopped, agents reconnect, and running `sh` steps are picked back up through the durable-task wrapper, which writes exit status to a file on the agent rather than relying on a live channel. That resilience costs fsyncs, which is why there are three durability levels, set globally in Manage Jenkins or per pipeline with `options { durabilityHint('PERFORMANCE_OPTIMIZED') }`.

MAX_SURVIVABILITY writes each flow node atomically as its own XML file and flushes the program aggressively, giving the best chance of resuming but the highest I/O, and on a busy controller with slow storage it is a genuine bottleneck. SURVIVABLE_NONATOMIC writes the same data without atomic replacement, faster but corruptible on an unclean shutdown. PERFORMANCE_OPTIMIZED batches flow nodes in memory and writes them in bulk, roughly an order of magnitude less I/O, at the cost that an unexpected crash loses the in-flight build, which then shows as a failure with 'Resume disabled' or a truncated log.

Most teams set PERFORMANCE_OPTIMIZED globally and keep MAX_SURVIVABILITY only for long deploy pipelines they cannot afford to rerun. Related knobs: `options { disableResume() }` tells Jenkins not to attempt a resume for a job whose steps are not idempotent, which is safer for deploy jobs than resuming halfway through a rollout, and the Preserve Stashes option keeps stashes for restart-from-stage. Interviewers use this question to test whether you understand that fast pipelines and resumable pipelines are a trade-off, not a free lunch.

pipeline {
  agent none
  options {
    // batch flow-node writes: far less controller disk I/O
    durabilityHint('PERFORMANCE_OPTIMIZED')
    // a half-finished rollout must not silently resume after a restart
    disableResume()
    buildDiscarder(logRotator(numToKeepStr: '40'))
  }
  stages {
    stage('Long build') {
      agent { label 'build' }
      options { retry(2) }
      steps { sh './long-build.sh' }
    }
  }
}

// Global default via JVM property (systemd drop-in or container env):
// -Dorg.jenkinsci.plugins.workflow.flow.FlowDurabilityHint.default=PERFORMANCE_OPTIMIZED

Answer

Freezes of that shape are almost always garbage collection pauses or lock contention, so the first move is data, not guesswork. Enable GC logging with `-Xlog:gc*,gc+heap=info:file=/var/log/jenkins/gc.log:time,uptime:filecount=10,filesize=50M` and look at pause durations and whether the heap is climbing after each full GC. Take thread dumps from `<JENKINS_URL>/threadDumps` during a freeze, or install the Support Core plugin and generate a support bundle, which packages GC stats, thread dumps, system properties and slow-request reports.

The usual root causes, in the order I check them: heap too small for the number of loaded jobs and build records, so raise `-Xmx` (4 to 16 GB is the normal band, larger heaps mostly mean longer pauses, not fewer), and switch to G1 with `-XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+AlwaysPreTouch`. Second, builds running on the built-in node, set its executors to 0. Second-and-a-half, huge console logs streaming through the controller, a build printing 500 MB of Maven download noise will do this on its own.

Third, flow-node explosion: a pipeline that runs thousands of tiny steps in a loop creates thousands of persisted flow nodes, and the Pipeline Stage View plugin then tries to build a model of all of them, so collapse the loop into a single shell script. Fourth, storage latency on JENKINS_HOME, network filesystems for JENKINS_HOME are a well known cause of stalls, use local SSD or fast block storage. Fifth, plugins doing synchronous work on the request thread, which the slow-request reports in the support bundle will name.

# /etc/sysconfig/jenkins or the container JAVA_OPTS
JAVA_OPTS="\
  -Xms8g -Xmx8g \
  -XX:+UseG1GC \
  -XX:+ParallelRefProcEnabled \
  -XX:+UseStringDeduplication \
  -XX:+AlwaysPreTouch \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/var/log/jenkins/heapdump.hprof \
  -Xlog:gc*,gc+heap=info:file=/var/log/jenkins/gc.log:time,uptime:filecount=10,filesize=50M \
  -Djenkins.install.runSetupWizard=false \
  -Dhudson.model.DirectoryBrowserSupport.CSP=\"sandbox; default-src 'none';\""

# During a freeze: capture evidence rather than restarting blindly
curl -s -u "$USER:$TOKEN" "$JENKINS_URL/threadDumps" > /tmp/dump-$(date +%s).txt
jcmd $(pgrep -f jenkins.war) GC.heap_info
jcmd $(pgrep -f jenkins.war) Thread.print > /tmp/threads.txt

Answer

Start by separating the two capacity problems. Executor capacity is easy to scale: run ephemeral agents from a cloud, the Kubernetes plugin for pods or the EC2 plugin for spot instances, so capacity tracks the queue instead of sitting idle overnight. Tune the provisioner so it reacts quickly, `-Dhudson.slaves.NodeProvisioner.initialDelay=0` and a lower `MARGIN0`, and set a container cap so a runaway job cannot consume the cluster.

Controller capacity is the hard one, because there is exactly one controller per instance and it is not horizontally scalable. Everything that touches it must be minimised: no builds on the built-in node, webhooks instead of `pollSCM`, aggressive build discarders, artefacts pushed to Nexus or S3 rather than archived, PERFORMANCE_OPTIMIZED durability, and shell steps batched so a build produces tens of flow nodes rather than thousands. Then measure: the Prometheus metrics plugin exposes queue length, executor utilisation and build durations, and the number worth alerting on is queue wait time at p95, because that is what developers actually feel.

When one controller is genuinely saturated, the answer is not a bigger machine, it is sharding: split into several controllers by business unit or by workload type, each with its own JCasC config and agent pool, with shared libraries and a common controller image keeping them consistent. Large Indian services organisations typically run per-account or per-programme controllers for exactly this reason, and it also contains the blast radius, a plugin upgrade that breaks one controller does not stop every delivery team in the company.

Answer

Around the 2.475 weekly and the 2.479.x LTS line in late 2024, Jenkins core moved to Jetty 12 with Jakarta EE 9, which renamed the servlet API package from `javax.servlet` to `jakarta.servlet`, and raised the minimum Java runtime to 17 with Java 21 supported. That is the largest compatibility break in years, because any plugin that referenced the servlet API or Spring's older APIs had to be recompiled against the new core. Practically, if you upgrade a controller across that boundary without upgrading plugins first, unmaintained plugins fail to load with `NoClassDefFoundError: javax/servlet/ServletException` or a dependency error, and their steps disappear, which means every pipeline using them fails with 'No such DSL method'.

The safe sequence is: take a controller-config backup, stand up a staging controller from that config, upgrade Java on it first and confirm the current Jenkins runs on 17 or 21, then upgrade all plugins to versions compatible with the target core, then upgrade core, then run a representative set of real pipelines including your shared libraries. Use the Plugin Manager's compatibility warnings and the plugin BOM for the target line, and identify unmaintained plugins early, they need a replacement or a fork, not an upgrade. Two more things to raise: agents also run their own JVM, so agent images need the new JDK and a Remoting version in step with the controller, and Groovy behaviour differences between JDKs occasionally surface in shared libraries, so include library unit tests in the staging pass. Budget an upgrade window, not an evening.

Answer

CVE-2024-23897, disclosed in January 2024 and fixed in 2.442 and LTS 2.426.3, was an arbitrary file read in the built-in CLI: the args4j command parser expanded an argument beginning with `@` into the contents of a file on the controller, so a request could read files off the controller filesystem. With read-level access an attacker could pull `secrets/master.key` and `hudson.util.Secret` and decrypt stored credentials, and public exploit chains escalated it to remote code execution. It was weaponised quickly against internet-exposed controllers, and the lesson the community took was simple: a Jenkins controller is a build-and-deploy oracle holding every credential your organisation uses, so treat it like a production identity provider, not like a developer tool. A hardening list worth reciting: do not expose it to the internet, put it behind a VPN or a zero-trust proxy with SSO; enforce TLS and set the Jenkins URL correctly so links and crumbs work; disable anonymous read; use SAML or OIDC with your corporate directory plus Role Strategy for authorization; zero executors on the built-in node and keep agent-to-controller access control enabled; scope credentials to folders and prefer short-lived cloud credentials or Vault over static keys; keep the Groovy sandbox on and stop bulk-approving script signatures; subscribe to the Jenkins security advisory mailing list and treat plugin advisories as production incidents; run the controller as a non-root user in a container with a read-only filesystem where possible; and audit `init.groovy.d` and shared library write access, both of which are unsandboxed admin equivalents.

Answer

Immutable means the controller's configuration is built into an image and the running instance is disposable. The image is a Dockerfile based on the official `jenkins/jenkins:lts-jdk21`, with `plugins.txt` installed at build time by `jenkins-plugin-cli`, a `casc.yaml` baked in and `CASC_JENKINS_CONFIG` pointing at it, the setup wizard disabled, and optionally seed Job DSL scripts that create the folder and job structure. Nothing is configured through the UI, and to make that real you either remove Overall/Administer from humans in normal operation or accept that any UI change will be lost on the next deploy, which is the point.

State that genuinely cannot live in the image is separated: `JENKINS_HOME` on a persistent volume for build history, and credentials in an external store, AWS Secrets Manager, HashiCorp Vault or the CyberArk integration common in Indian banks, so the disaster-recovery path never depends on restoring `secrets/master.key`. Disaster recovery then becomes: deploy the image, mount (or re-create) the volume, and jobs reappear because Multibranch scanning rebuilds them from the repositories. Measure it, actually run the restore quarterly and time it, because a DR plan that has never been executed is a document, not a capability.

The honest caveats to raise: build history is lost if the volume is gone, so decide up front whether history is a compliance requirement or a nice-to-have; and a small amount of state, fingerprints, queue, in-flight builds, is genuinely ephemeral. Interviewers value hearing that trade-off stated explicitly rather than a claim that everything is stateless.

# docker-compose fragment for an immutable controller
services:
  jenkins:
    image: registry.example.co.in/platform/jenkins:2026.02.1   # built from Dockerfile + plugins.txt + casc.yaml
    user: "1000:1000"
    read_only: false
    environment:
      CASC_JENKINS_CONFIG: /var/jenkins_conf/casc.yaml
      SECRETS: /run/secrets/jenkins           # JCasC reads ${VAR} from files here
      JAVA_OPTS: >-
        -Xms6g -Xmx6g -XX:+UseG1GC -XX:+AlwaysPreTouch
        -Djenkins.install.runSetupWizard=false
        -Dorg.jenkinsci.plugins.workflow.flow.FlowDurabilityHint.default=PERFORMANCE_OPTIMIZED
    volumes:
      - jenkins_home:/var/jenkins_home        # build history only
      - ./secrets:/run/secrets/jenkins:ro
    ports: ['8080:8080']
volumes:
  jenkins_home:

Answer

Once a shared library serves hundreds of pipelines, changing it blind is unacceptable, so the mature setup has three layers. Layer one is JenkinsPipelineUnit, a JVM test library that loads your `vars/` and `src/` Groovy, mocks the pipeline DSL, and lets you assert on the call stack: which steps were invoked, with what arguments, and in what order. You register the steps your code uses with `helper.registerAllowedMethod`, drive the library from a JUnit or Spock test, and assert that, say, a production deploy called `withCredentials` with the production credential ID and never called it for a feature branch.

It runs in milliseconds in a normal Gradle or Maven build, so the library repository gets a real CI pipeline of its own. Layer two is jenkinsfile-runner, which boots a headless single-shot Jenkins in a container and actually executes a Jenkinsfile against it, catching things a mock cannot: a missing plugin, a step whose signature changed, a Declarative validation error. Layer three is a canary: promote the library tag on a handful of pilot jobs before moving the default version, which is why pinning `@Library('platform-lib@v3.2.0')` rather than `@main` matters so much.

Add the Declarative linter in the pull-request check for consumer repositories and the loop is closed. Also test the negative paths, that a missing required config key calls `error` with a readable message, because that is what determines whether an application team can debug their own pipeline failure or files a ticket with your platform team.

// test/co/in/example/DeployServiceSpec.groovy (Spock + JenkinsPipelineUnit)
import com.lesfurets.jenkins.unit.BasePipelineTest
import org.junit.Before
import org.junit.Test

class DeployServiceSpec extends BasePipelineTest {

  @Before void setUp() {
    super.setUp()
    helper.registerAllowedMethod('withCredentials', [List, Closure]) { l, c -> c() }
    helper.registerAllowedMethod('timeout', [Map, Closure]) { m, c -> c() }
    helper.registerAllowedMethod('sh', [String]) { cmd -> println "sh: ${cmd}" }
    helper.registerAllowedMethod('error', [String]) { msg -> throw new RuntimeException(msg) }
  }

  @Test void 'production deploy waits for rollout'() {
    def script = loadScript('vars/deployService.groovy')
    script.call { app = 'billing-api'; image = 'img:1'; environment = 'production' }
    assertJobStatusSuccess()
    assert helper.callStack.findAll { it.methodName == 'sh' }
      .any { it.args[0].toString().contains('rollout status') }
  }

  @Test(expected = RuntimeException) void 'missing app fails fast'() {
    loadScript('vars/deployService.groovy').call { environment = 'staging' }
  }
}

Answer

This is a judgement question and the wrong answer is a blanket one. Keep Jenkins when the constraints are structural: builds that must run inside a private data centre or an air-gapped network with no egress, hardware-in-the-loop or licensed-toolchain builds on specific machines, orchestration that spans systems no SaaS runner reaches, regulated environments where the auditors already have an approved control set built around your controller, or simply a decade of shared-library logic whose rewrite cost dwarfs any benefit. Jenkins also still wins on plugin breadth and on the freedom to run any agent anywhere.

Move to GitHub Actions or GitLab CI when the workload is ordinary build-test-publish on code you already host there: you get zero controller to operate, a marketplace ecosystem, per-repository config, and no JVM to tune, which is worth a lot on small platform teams. For deployment specifically, the strong 2026 pattern is to stop deploying from CI at all: Jenkins or Actions builds and publishes an immutable artefact and updates a manifest, and Argo CD or Flux reconciles the cluster from Git, which removes long-lived cluster credentials from the CI system entirely. If you do migrate, do it per repository behind a shared library facade, run both systems in parallel until the new one is authoritative, and migrate the noisy high-frequency repositories first because that is where the operational relief is. Say explicitly that migration is a multi-quarter programme with a decommissioning date, because the failure mode everyone has seen is an organisation running two CI systems permanently.