CircleCI Interview Questions and Answers

Answer

CircleCI is a hosted continuous integration and delivery platform that runs builds, tests, and deployments in response to code changes. It solves three problems that older tools (self-hosted Jenkins, Travis CI) handled poorly: (1) infrastructure overhead, no servers to maintain, (2) slow feedback, first-class parallelism and Docker layer caching speed up large test suites, (3) reusability, Orbs let teams share pipeline code like npm packages. You declare your pipeline in `.circleci/config.yml` at the root of your repo, CircleCI detects the file via a GitHub/Bitbucket/GitLab webhook, and runs the jobs on Linux, Windows, macOS, or self-hosted runners.

Pricing is credit-based, you pay for compute minutes scaled by resource class. Mechanically, a push fires a webhook, CircleCI compiles `config.yml` into a fully static plan (Orbs inlined, parameters substituted, matrix entries expanded), creates a pipeline, then schedules each workflow job onto an executor as its `requires:` dependencies clear. Every job boots a clean environment, so nothing survives between jobs unless you explicitly move it with a cache, a workspace, or an artifact.

Senior interviewers follow up on where CircleCI actually sits in 2026: GitHub Actions took the default-for-new-repos slot, while CircleCI keeps teams that need macOS capacity, `setup: true` dynamic config, per-job `resource_class` tuning, or timing-based sharding via `circleci tests split --split-by=timings`. Mentioning that the GitHub App integration (not the legacy OAuth 'CircleCI Checks' one) is what unlocks tag-push triggers, multiple pipeline definitions per repo, and pipeline parameters supplied by an API trigger signals recent hands-on use rather than a memory of the 2019 product.

Answer

Every config starts with a `version: 2.1` declaration (2.1 is required for Orbs, reusable executors, and commands). Below that, you define `jobs` (the units of work), optionally `executors` (the runtime environment), `commands` (reusable step sequences), and `workflows` (the orchestration graph that decides which jobs run when). A job consists of an executor and a sequence of `steps`, typically `checkout`, dependency restore, `run` commands, and cache/artifact storage.

Two more top-level keys matter in 2026: `setup: true`, which marks the file as a setup config whose only purpose is to generate and continue into another config, and `parameters:`, which declares pipeline parameters that an API trigger or a `continuation/continue` call can supply. Key order inside the file is irrelevant because references resolve after parsing, so a workflow may name a job declared below it. Validate locally with `circleci config validate` and inspect the compiled result with `circleci config process .circleci/config.yml`, which inlines Orbs and substitutes `<< parameters.x >>`.

That is the fastest way to debug 'Cannot find a definition for job named ...' or 'Unexpected argument(s)' errors, because both are compile-time failures that never reach an executor. Note that `version: 2.0` configs still execute but cannot use Orbs, `commands`, `executors`, `parameters`, matrix jobs, or dynamic config, so any current answer should assume 2.1. Plain YAML anchors and aliases also work, but parameters and reusable commands are preferred since anchors cannot cross file boundaries into an Orb.

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/node:20.10
    steps:
      - checkout
      - run: npm ci
      - run: npm test
      - store_test_results:
          path: ./test-results

workflows:
  build-and-test:
    jobs:
      - build

Answer

A job is the smallest billable unit of work in CircleCI, a discrete sequence of steps that runs inside a single executor. Each job gets a fresh environment (no state carries between jobs unless you use workspaces, caches, or artifacts). A job has an executor declaration (`docker:`, `machine:`, `macos:`, or `executor:`), an optional `resource_class` (small / medium / large / xlarge / 2xlarge), an optional `parallelism` value, and a list of `steps`.

You'd typically have separate jobs for build, lint, unit tests, integration tests, and deploy so they can run in parallel and fail independently. Mechanics worth knowing: each `run` step launches a fresh shell (`/bin/bash -eo pipefail` on the `cimg/*` images) but shares the job's filesystem, so a `cd` in one step does not persist while a file written in one step does. Use `working_directory:` or a multi-line `command:` block when you need directory state. `resource_class` defaults to `medium` and the credit multiplier scales with it, which makes the class a per-job cost decision rather than a project-wide one.

Billing counts spin-up time as well as run time, so a chain of six tiny jobs is frequently slower and more expensive than two well-packed ones. Cloud jobs are capped at 5 hours, and any single step that prints nothing for 10 minutes is killed with 'Too long with no output (exceeded 10m0s)', which you override per step with `no_output_timeout: 20m`. A job's exit status is the first non-zero step exit code, so add `when: on_fail` or `when: always` steps if you still want logs and JUnit results uploaded after a failure.

jobs:
  integration:
    docker:
      - image: cimg/node:20.10
    resource_class: large
    working_directory: ~/project/apps/api
    steps:
      - checkout:
          path: ~/project
      - run:
          name: Migrate and run integration tests
          no_output_timeout: 20m
          command: |
            npm ci
            npm run db:migrate
            npm run test:integration
      - run:
          name: Dump service logs on failure
          when: on_fail
          command: cat /tmp/api.log || true
      - store_test_results:
          path: ./test-results

Answer

Four main executor types: (1) **docker**, the fastest and cheapest, runs your steps inside a container; ideal for stateless web/API builds. Limitation: can't run Docker-in-Docker without `setup_remote_docker`. (2) **machine**, a full Linux VM with Docker pre-installed; needed for Docker Compose, kernel access, or when you need privileged operations. Slower to spin up (~30s vs ~5s for docker). (3) **macos**, required for iOS/macOS builds, expensive (10× the credit cost). (4) **windows**, for .NET, Win32 builds.

There's also `arm.medium` and GPU executors (`gpu.nvidia.small/medium`) for ML workloads. Details interviewers probe: under the `docker` executor only the FIRST image in the list is the primary where your steps run, and every additional image is a service container reachable on `localhost`, so Postgres is `localhost:5432` and not a named host. CircleCI's own `cimg/*` images sit on its registry, boot in a few seconds from a warm cache, and are exempt from Docker Hub rate limits. `machine` images are selected by tag (`ubuntu-2204:current`, `ubuntu-2404:current`); `current` moves over time, so pin a dated tag when reproducibility matters. macOS jobs must name an `xcode:` version CircleCI still hosts, and Apple silicon classes are named `macos.m1.medium.gen1` upward.

The classic trap: with `setup_remote_docker` the daemon lives on a different host, so `docker run -v $(pwd):/app` silently mounts an empty directory instead of your checkout. Copy files across with `docker cp`, or bake them into the image, or switch to the `machine` executor and accept the slower boot.

jobs:
  unit-tests:
    docker:
      - image: cimg/node:20.10
    resource_class: medium
    steps: [checkout, run: npm test]

  integration:
    machine:
      image: ubuntu-2204:current
    resource_class: large
    steps: [checkout, run: docker compose up -d, run: npm run test:int]

  ios-build:
    macos:
      xcode: 15.4.0
    resource_class: macos.m1.medium.gen1
    steps: [checkout, run: xcodebuild test]

Answer

A workflow is a directed acyclic graph (DAG) of jobs that defines execution order, dependencies, and conditions. Workflows let you fan-out (run lint, unit tests, build in parallel), fan-in (deploy only after all tests pass), gate progress with manual approvals, filter by branch or tag, and schedule recurring runs. Without workflows, jobs run sequentially in declaration order, almost never what you want for modern pipelines.

Details that separate a real user from a reader: a job name may appear only once per workflow unless you give each instance a distinct `name:`, which is also how matrix expansion generates unique names. When an upstream job fails, downstream jobs show as 'blocked' rather than 'failed', the workflow itself goes to 'failed', and a workflow waiting on an approval sits at 'on hold' without burning credits. Workflow-level `when:` and `unless:` accept a pipeline parameter or a logic block (`and`, `or`, `not`, `equal`), which is how you switch entire workflows on from an API trigger without generating config. `pre-steps` and `post-steps` inject steps around a job from inside the workflow declaration, handy for attaching a workspace or posting to Slack without editing the job itself. Scheduled pipelines, configured in project settings or through the Terraform provider, replaced the deprecated `triggers: schedule` key that used to live inside a workflow; unlike the old syntax they carry pipeline parameters and run under an explicit attribution actor whose permissions decide what contexts the scheduled run can read.

workflows:
  ci:
    jobs:
      - lint
      - unit-tests
      - build:
          requires: [lint, unit-tests]
      - deploy:
          requires: [build]
          filters:
            branches:
              only: main

Answer

Orbs are shareable, versioned packages of CircleCI config, think of them as npm modules for pipeline code. An Orb can export jobs, commands, and executors that you reference from your own config. Common uses: `aws-cli/setup`, `slack/notify`, `node/install`, `docker/build`.

Three Orb categories: **Certified** (built by CircleCI, audited), **Partner** (built by vendors like AWS, GCP, HashiCorp), **Community** (open-source contributions, vary in quality). You import an Orb at the top of your config and reference its jobs/commands using `<orb-name>/<element>` syntax. Version resolution is the part people get wrong: `@5.2.0` pins an exact published version, `@5.2` resolves to the newest patch, `@5` to the newest minor, and `@volatile` to the most recent publish of any kind.

Dev tags such as `@dev:my-branch` are mutable and expire after 90 days, so they must never appear in a production config. Because Orbs are inlined at compile time, a floating tag means another team's publish can change your pipeline with no commit in your repo, which is why security-conscious orgs pin exact versions and read the diff with `circleci orb source circleci/node@5.2.0` before bumping. Private Orbs, available on paid plans, are visible only inside your organisation and are the right home for deploy logic that embeds internal hostnames or account IDs. Use `circleci orb info circleci/aws-cli` to see usage stats and the version history, and `circleci config process` to see exactly what YAML the Orb expanded into before you trust it in a release path.

version: 2.1

orbs:
  node: circleci/node@5.2.0
  slack: circleci/slack@4.13.1

jobs:
  test:
    executor: node/default
    steps:
      - checkout
      - node/install-packages
      - run: npm test
      - slack/notify:
          event: fail
          channel: deploys

Answer

`checkout` is a built-in step that clones your source repository into the current working directory (`~/project` by default). It handles SSH key setup, shallow clone optimisations, and submodule init if configured. For large repos, you can pass `path:` to clone elsewhere, or skip it entirely and use `git clone` manually if you need a custom strategy (e.g., partial clone, sparse checkout).

On self-hosted runners, you may need to authenticate the runner with a deploy key explicitly. What `checkout` actually does is not a plain `git clone`: it writes an SSH config and known_hosts entry, then either clones or, when the cache directory already holds the repo, fetches and hard-resets to `$CIRCLE_SHA1`. On a pull request it checks out the head commit, not a merge commit, so a PR that is green in CircleCI can still break `main` if someone merged in between; teams that care about this add a `git merge --no-edit origin/main` step before the build.

Submodules are not initialised by default, you need an explicit `git submodule sync && git submodule update --init --recursive` step and a deploy key that can read the submodule repos. For very large repos, skipping `checkout` in favour of `git clone --filter=blob:none --depth 50` plus a sparse-checkout of the paths a job needs can cut a minute or more per job. The `path:` argument matters in monorepos where each job sets a different `working_directory` under a shared checkout root.

steps:
  - checkout:
      path: ~/project
  - run:
      name: Init submodules
      command: |
        git submodule sync --recursive
        git submodule update --init --recursive
  - run:
      name: Verify the PR still merges cleanly into main
      command: |
        if [ -n "$CIRCLE_PULL_REQUEST" ]; then
          git fetch origin main
          git merge --no-edit origin/main
        fi

Answer

Three levels: (1) **Job-level** via `environment:` block, visible inside that job only, good for non-secret config like `NODE_ENV=test`. (2) **Project-level** via the CircleCI UI Project Settings → Environment Variables, encrypted at rest, available to all jobs in that project. (3) **Context-level**, same as project-level but shared across multiple projects in an organisation; this is how you'd share AWS credentials or registry tokens between repos. Job-level env declared in YAML is plain-text and should never contain secrets. Precedence runs from narrowest to broadest: a value exported inside a `run` step beats the step's `environment:`, which beats the job's `environment:`, which beats the container's `environment:`, which beats context values, which beat project variables, which beat CircleCI's own built-ins such as `CIRCLE_BRANCH`, `CIRCLE_SHA1`, and `CIRCLE_BUILD_NUM`.

A detail that trips people up: `environment:` values are static YAML, so you cannot write `PATH: $PATH:/opt/bin` or interpolate another variable there. To compute a value at runtime, append it to `$BASH_ENV`, the file CircleCI sources before every subsequent step, which is the sanctioned way to pass state between steps. Project and context values are decrypted into the job environment, so anything with shell access to the job can read them, meaning fork PR builds must never receive production secrets (the 'Pass secrets to builds from forked pull requests' project setting must stay off). CircleCI masks known secret values in log output, but only for values longer than four characters and only when printed verbatim, so `echo $TOKEN | base64` still leaks.

jobs:
  deploy:
    docker:
      - image: cimg/python:3.12
    environment:
      DEPLOY_REGION: ap-south-1
      LOG_LEVEL: info
    steps:
      - run: ./deploy.sh   # AWS_* secrets come from a context

workflows:
  release:
    jobs:
      - deploy:
          context: aws-prod

Answer

**Workspaces** pass files BETWEEN jobs in the same workflow run, e.g., the `build` job produces `dist/` and the `deploy` job consumes it. Use `persist_to_workspace` and `attach_workspace`. **Caches** persist files BETWEEN runs of the same project, e.g., `node_modules/` survives across workflow runs to speed up `npm ci`. Use `save_cache` and `restore_cache` with a key that invalidates when dependencies change.

Confusing them is a top interview gotcha: caches are best-effort and shared; workspaces are guaranteed and run-scoped. The behavioural differences follow from that. A cache key is immutable once written: `save_cache` with an existing key is a silent no-op, which is why every real config puts a version prefix like `v3-` in the key so you can force a rebuild.

Workspace layers, in contrast, are additive, and if two parallel jobs persist the same path into the same workspace the result is a race whose winner is undefined, so give each producer job a distinct subdirectory. `attach_workspace` restores relative to the `root:` you persisted from, so persisting with `root: ~/project, paths: [dist]` and attaching `at: .` inside a job whose working directory is `~/project` is the combination that behaves as expected. Workspaces are scoped to a single pipeline and expire with the default 15-day retention, while caches are scoped to the project and are also readable by jobs on other branches, which is the reason you should never write a secret or a signed artifact into a cache. On a rerun of a single failed job, the workspace from the original run is reattached, which makes 'rerun workflow from failed' cheap.

# build job
- persist_to_workspace:
    root: .
    paths: [dist, package.json]

# deploy job
- attach_workspace:
    at: .
- run: ./deploy.sh

Answer

`store_artifacts` uploads files to CircleCI's artifact storage where they're browseable from the job page and accessible via API. Use for build outputs (binaries, web bundles), coverage reports, screenshots, or anything you want a human to inspect. Default retention is 30 days.

Don't confuse with `store_test_results` which parses JUnit/XUnit XML to populate the Tests tab, those don't appear under artifacts. The distinction has a practical consequence: only `store_test_results` feeds CircleCI's timing database, and `circleci tests split --split-by=timings` silently falls back to splitting by file name when that data is missing, so a pipeline that stores no test results gets badly unbalanced shards and nobody notices until one container runs three times as long as the rest. Other operational details: `path:` can be a file or a directory, `destination:` renames the prefix shown in the Artifacts tab, and paths must be inside the job's filesystem (globs are not expanded, so wrap wildcards in a `run` step that copies matches into one folder first).

Artifact storage is billed by GB-month once you pass the plan allowance, so uploading `node_modules` or an entire Cypress video directory on every run is a common and avoidable cost line. Pair `store_artifacts` with `when: always` so failing runs, the ones you actually need to inspect, still publish their screenshots and logs. Artifacts are fetchable programmatically from the v2 API endpoint `/project/{project-slug}/{job-number}/artifacts`.

- store_artifacts:
    path: dist/app.tar.gz
    destination: build-artifacts
- store_test_results:
    path: ./test-results/junit.xml

Answer

Use the `filters:` block on a job within a workflow. Filters support `branches:` and `tags:` with `only:` / `ignore:` sub-keys, accepting strings or regex. By default, tag pushes don't trigger workflows, you must explicitly opt in with `tags: { only: /.*/ }`.

Branch filters are also widely used for environment promotion (deploy to staging from `develop`, prod from `main`). The rule people forget: tag filtering is opt-in per job AND inherited nowhere, so if `deploy` requires `build` and only `deploy` has a tag filter, the workflow never runs because `build` was filtered out of the tag pipeline. Every job in the dependency chain needs `filters: { tags: { only: /.*/ } }`.

Combining `tags: only` with `branches: ignore: /.*/` is the standard way to build a release-only job. Regexes must match the entire string (CircleCI anchors them implicitly), so `only: /v.*/ ` matches `v1.2.3` but `only: /1.2/` does not match `v1.2.3`. Filters cannot see file paths or commit messages, which is why monorepo selectivity needs the `path-filtering` Orb or dynamic config instead. For anything more expressive than a branch or tag pattern, prefer workflow-level `when:` on a pipeline parameter, since that can encode conditions such as 'only when triggered by the nightly schedule' via `<< pipeline.trigger_source >>` or a parameter set by the API trigger.

workflows:
  release:
    jobs:
      - build:
          filters:
            tags:
              only: /^v\d+\.\d+\.\d+$/
            branches:
              ignore: /.*/

Answer

An approval job pauses a workflow until a human clicks 'Approve' in the CircleCI UI. You declare it with `type: approval` and no executor or steps. Subsequent jobs use `requires:` to depend on it.

Most common use: production deploys that need a release manager sign-off after staging tests pass. Combine with branch filters (only on `main`) and Slack notifications for visibility. The approving user is recorded in audit logs.

Practical behaviour: an approval job consumes no credits while it waits, the workflow shows status 'on hold', and it will sit there until someone acts or the workflow is cancelled. There is no built-in expiry or auto-approve timeout, so a team that leaves holds open accumulates a queue of half-finished releases; most orgs add a scheduled job that cancels workflows older than a day via the v2 API. The approver is any user with write access to the project, and CircleCI does not enforce 'not the author', so genuine two-person review has to come from the VCS side (branch protection requiring a reviewer) plus a context restricted to a security group.

An approval job cannot run steps, cannot set outputs, and cannot be conditional on anything except the same `filters:` and `requires:` other jobs use. If you need an approval that carries information, such as which environment to promote to, model it as a parameterised API trigger instead of a hold.

workflows:
  deploy:
    jobs:
      - test
      - deploy-staging:
          requires: [test]
      - hold-for-prod:
          type: approval
          requires: [deploy-staging]
      - deploy-prod:
          requires: [hold-for-prod]

Answer

Every job gets `CI=true` plus a `CIRCLE_*` set: `CIRCLE_SHA1` (full commit SHA), `CIRCLE_BRANCH`, `CIRCLE_TAG`, `CIRCLE_JOB`, `CIRCLE_BUILD_NUM`, `CIRCLE_BUILD_URL`, `CIRCLE_WORKFLOW_ID`, `CIRCLE_PROJECT_USERNAME`, `CIRCLE_PROJECT_REPONAME`, `CIRCLE_NODE_INDEX`, `CIRCLE_NODE_TOTAL`, `CIRCLE_WORKING_DIRECTORY`, and `CIRCLE_OIDC_TOKEN` / `CIRCLE_OIDC_TOKEN_V2` when a context is attached. The reliable ones are `CIRCLE_SHA1`, `CIRCLE_BUILD_NUM` and `CIRCLE_WORKFLOW_ID`, which is why immutable image tags should be built from `$CIRCLE_SHA1` and never from a branch name. The unreliable ones are where interviews go. `CIRCLE_BRANCH` is empty on a tag-triggered pipeline and `CIRCLE_TAG` is empty on a branch pipeline, so any script that assumes both exist breaks on release day; guard with `${CIRCLE_TAG:-$CIRCLE_BRANCH}`. `CIRCLE_PULL_REQUEST` and `CIRCLE_PR_NUMBER` are only populated for pull requests from forks, so 'run extra checks on PRs' logic written against them silently no-ops for same-repo branches; query the VCS API by branch instead. `CIRCLE_COMPARE_URL` was removed with the newer VCS integrations, and configs that still reference it get an empty string rather than an error, which is exactly the kind of silent breakage that makes a deploy script skip its diff check.

steps:
  - run:
      name: Compute an immutable image tag
      command: |
        REF="${CIRCLE_TAG:-$CIRCLE_BRANCH}"
        SAFE_REF=$(echo "$REF" | tr '/' '-')
        echo "export IMAGE_TAG=${SAFE_REF}-${CIRCLE_SHA1:0:7}" >> "$BASH_ENV"
  - run:
      name: PR-only checks that work for same-repo branches too
      command: |
        if [ "$CIRCLE_BRANCH" != "main" ]; then
          npm run lint:changed
        fi
  - run: echo "Built $IMAGE_TAG in workflow $CIRCLE_WORKFLOW_ID"

Answer

The old `triggers: schedule` block inside a workflow is deprecated. The replacement is scheduled pipelines, configured in Project Settings under Triggers, through the v2 API (`POST /api/v2/project/{project-slug}/schedule`), or with the CircleCI Terraform provider. A schedule carries four things the old syntax could not: a target branch or tag, a timetable (`per-hour`, `hours-of-day`, `days-of-week`), a set of pipeline parameters, and an attribution actor.

The actor matters: a schedule attributed to the 'scheduling system' user cannot read contexts that are restricted to a security group, so a nightly deploy that worked in testing fails with an authorisation error once someone locks the context down. Attribute it to a service account instead. The idiomatic pattern is to declare a boolean pipeline parameter, have the schedule set it to true, and gate the heavy workflow on it with a workflow-level `when:`, which keeps one config file serving both push and nightly runs.

You can also branch on `<< pipeline.trigger_source >>`, which is `webhook`, `api`, or `scheduled_pipeline`. Nightly jobs are the right home for the full matrix, the whole monorepo build, and dependency audits, all of which are too expensive per pull request.

version: 2.1

parameters:
  nightly:
    type: boolean
    default: false

workflows:
  pr-checks:
    when:
      not: << pipeline.parameters.nightly >>
    jobs: [lint, unit-tests]

  nightly-full:
    when: << pipeline.parameters.nightly >>
    jobs:
      - full-matrix-tests
      - dependency-audit

# Create the schedule (runs 02:00 IST = 20:30 UTC previous day):
# curl -X POST https://circleci.com/api/v2/project/gh/acme/api/schedule \
#   -H "Circle-Token: $CIRCLE_TOKEN" -H 'Content-Type: application/json' \
#   -d '{"name":"nightly","parameters":{"nightly":true,"branch":"main"},
#        "timetable":{"per-hour":1,"hours-of-day":[20],"days-of-week":["MON","TUE","WED","THU","FRI"]},
#        "attribution-actor":"system"}'

Answer

Install the CLI (`brew install circleci` or the install script), authenticate with `circleci setup`, then use three commands. `circleci config validate` checks the schema and resolves Orbs, catching 'Cannot find a definition for job named ...' before you push; add `--org-slug gh/your-org` when the config imports a private Orb, otherwise resolution fails with a 404. `circleci config process .circleci/config.yml` prints the fully compiled config with Orbs inlined and `<< parameters.x >>` substituted, which is the only honest way to see what CircleCI will actually run. `circleci local execute --job build` runs one job on your own Docker daemon. Know its limits, because interviewers ask: local execute runs a single job with no workflow, so `requires:`, approvals and filters are ignored; `save_cache` and `restore_cache` are no-ops; workspaces do not exist; contexts are unavailable so you pass secrets with `-e KEY=value`; and only the `docker` executor works, so `machine` and `macos` jobs cannot be reproduced. For a 2.1 config you either pipe `config process` output in or let the CLI process it for you. Add `circleci config validate` as a pre-commit hook, since a config typo costs a full round trip through the queue.

# Validate, including private orbs
circleci config validate --org-slug gh/acme .circleci/config.yml

# See the compiled config (orbs inlined, parameters substituted)
circleci config process .circleci/config.yml > /tmp/processed.yml

# Run one job locally with a fake secret injected
circleci local execute \
  --config /tmp/processed.yml \
  --job unit-tests \
  -e NPM_TOKEN=dummy

# Pack a split config directory into one file
circleci config pack .circleci/src > .circleci/config.yml

Answer

Caching has three components: a stable prefix (for fallback restores), a hash of the lockfile (invalidates on dependency change), and the path to cache. The pattern: `restore_cache` with a list of keys from most-specific to least-specific, run install (no-op if cache hit), `save_cache` with the most-specific key. Mistakes to avoid: caching `node_modules` across Node versions (use `arch` in the key), forgetting to include OS in the key for matrix builds, and caching mutable directories like `.git`.

For monorepos, hash all relevant lockfiles together. CircleCI caches are write-once per key, bumping the prefix is how you force-invalidate. Two behaviours explain most cache bugs.

First, partial-key restore is a prefix match that returns the MOST RECENT matching cache, so `v3-deps-{{ arch }}-` will happily hand you a three-week-old `node_modules` built against a different lockfile; that is fine for `npm ci` (which prunes and reinstalls from the lockfile) and dangerous for `npm install` (which does not). Second, `save_cache` runs even when the job later fails unless you guard it, so a half-installed dependency tree can get baked into a key that then poisons every subsequent build until the prefix moves. Cache the package manager's global store rather than the project directory where possible (`~/.npm`, `~/.m2`, `~/.gradle/caches`, `~/.cache/pip`, `~/go/pkg/mod`) because those are content-addressed and safe to restore stale.

Available key templates are `{{ checksum "package-lock.json" }}`, `{{ arch }}`, `{{ .Branch }}`, `{{ .Environment.VAR }}`, `{{ .Revision }}` and `{{ epoch }}`; using `epoch` guarantees a miss on every run and is a classic accidental cost bug. Caches expire after 15 days of no reads and count against storage billing.

steps:
  - checkout
  - restore_cache:
      keys:
        - v3-deps-{{ arch }}-{{ checksum "package-lock.json" }}
        - v3-deps-{{ arch }}-
  - run: npm ci
  - save_cache:
      key: v3-deps-{{ arch }}-{{ checksum "package-lock.json" }}
      paths:
        - ~/.npm
        - node_modules

Answer

Set `parallelism: N` on a job and CircleCI spins up N identical containers running the SAME job. Inside each, `CIRCLE_NODE_INDEX` (0..N-1) and `CIRCLE_NODE_TOTAL` (N) let you shard work. The `circleci tests split` CLI distributes test files by timing data so each container finishes in roughly equal time.

This is different from defining multiple jobs in a workflow: multiple jobs run different work in parallel; `parallelism` runs the same work split across N containers. Each parallel container consumes credits independently, `resource_class: large` × `parallelism: 8` × 5 minutes = 40 large-minutes billed. The failure modes are worth naming.

Splitting only works if the command receives a file list, so a runner that discovers tests itself (`pytest` with no arguments, `go test ./...`) will run the full suite in all N containers and quietly multiply your bill while reporting green. Timing-based splitting needs historical data from `store_test_results`; without it CircleCI falls back to filename splitting and one shard ends up carrying the slow integration file. `circleci tests split` also supports `--split-by=filesize` for test suites with no timing history, and `--total`/`--index` overrides if you are sharding something that is not tests. Shared external state is the other trap: eight containers hitting one staging database will interleave, so give each shard its own schema keyed on `$CIRCLE_NODE_INDEX`.

Parallelism has diminishing returns because every container pays the same spin-up, checkout and dependency-restore cost, so a 4-minute suite split 16 ways can be slower end to end than split 4 ways. Report the per-shard durations from the Tests tab when tuning N rather than guessing.

jobs:
  test:
    docker: [{ image: cimg/node:20.10 }]
    parallelism: 4
    steps:
      - checkout
      - run: npm ci
      - run:
          name: Run sharded tests
          command: |
            TESTS=$(circleci tests glob "**/*.test.ts" | circleci tests split --split-by=timings)
            npm test -- $TESTS
      - store_test_results:
          path: ./test-results

Answer

**Project env vars** live inside one project and are visible to every job/branch in that project. **Contexts** live at the organisation level and can be attached to any project's workflow via `context:`. The big practical differences: (1) Contexts can be **restricted to specific security groups** (only members of `prod-deployers` can run a workflow using `aws-prod`), (2) Contexts can be **restricted to specific branches** (only `main` can use `prod-secrets`), (3) Contexts are **rotatable across many projects** at once. For any credential that touches production, use a context with security group restrictions, project env vars are too broad.

Enforcement details matter in an interview. Context restrictions are checked when the job is scheduled, not when the YAML is written, so a pull request that adds `context: aws-prod` to its own config will fail at run time with 'Unauthorized' rather than leaking anything, which is exactly the property you want. Both kinds of variable are decrypted into the job environment and neither can be read back through the UI once saved, so rotation means overwrite, never inspect.

Contexts can be managed as code through the v2 API or the CircleCI Terraform provider, which is how teams keep 40 repos consistent and produce an audit trail of who changed which secret. The gap people miss: restricting a context to `main` does not stop a job on `main` from being triggered by a rerun of an older pipeline, and it does nothing about the fact that any engineer who can merge to `main` can add a step that prints the credential to a log. Real isolation comes from short-lived OIDC credentials plus scoped IAM roles, with contexts as the coarse first gate.

# In CircleCI UI: Organization Settings → Contexts → aws-prod
#   AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
#   Restrict to: security-group 'prod-deployers'
#   Restrict to: branch main

workflows:
  deploy:
    jobs:
      - deploy-prod:
          context: aws-prod
          filters:
            branches: { only: main }

Answer

Workflow: (1) Register a namespace in your org (`circleci namespace create <name> github <org>`). One namespace per organisation. (2) Create an Orb under it. (3) Author the Orb in a folder structure, `src/jobs/`, `src/commands/`, `src/executors/`, `src/@orb.yml`, and use the `orb-tools` Orb to pack, validate, and publish. (4) Publish dev versions (`<ns>/<orb>@dev:branch`) automatically on every PR, and semver-tagged production versions (`@1.2.3`) from `main`. Pin consumers to a specific minor version (`@1.2`) so patch fixes flow but breaking changes don't.

Things that bite first-time Orb authors: an Orb is compiled into the consumer's config, so it cannot read the consumer's env vars at author time and must accept anything variable as a parameter, ideally with `type: env_var_name` so consumers pass the NAME of a variable rather than its value. Orb parameters do not support arbitrary shell interpolation, `<< parameters.x >>` is substituted at compile time while `$VAR` is resolved at run time, and mixing the two is the source of most 'command not found' reports. Published production versions are immutable, so a bad `@1.2.3` can only be fixed by publishing `@1.2.4`; there is no unpublish.

Test the Orb by actually running it: `orb-tools/pack` plus a dev publish, then an integration workflow that consumes `@dev:${CIRCLE_SHA1:0:7}` and asserts real behaviour before the semver promotion job runs. Mark an Orb `private` at creation if it embeds internal detail, because you cannot flip a public Orb to private afterwards.

# .circleci/config.yml for the Orb itself
version: 2.1
setup: true
orbs:
  orb-tools: circleci/orb-tools@12.2.0

workflows:
  publish:
    jobs:
      - orb-tools/lint
      - orb-tools/pack
      - orb-tools/publish:
          orb_name: my-org/my-orb
          vcs_type: github
          context: orb-publishing
          requires: [orb-tools/pack]

Answer

Use the `matrix:` key under a job in a workflow to fan out across parameter combinations, e.g., test against Node 18/20/22 on Ubuntu and Alpine. CircleCI generates one job instance per combination. Combine with `exclude:` to skip invalid pairs (e.g., a deprecated Node × OS combination).

This is functionally similar to GitHub Actions' `strategy.matrix`. For matrix builds across architectures (x86 + ARM), declare a parametrised executor and pass the arch via matrix. Specifics that come up: the matrix expands at config-compile time, so the parameter lists must be literal YAML and cannot be computed from a previous job; if you need a runtime-computed matrix, that is precisely the case for dynamic config with `setup: true`.

Every combination becomes a separate job named `test-18.20-linux` style, and you can override the pattern with `alias:` when another job needs to `requires:` a specific cell. A `requires:` that names the bare job name waits for ALL cells of the matrix, which is usually what you want for a fan-in deploy but surprises people who expected a single instance. Parameters used in a matrix must be declared on the job with a `type`, and only `string`, `integer`, `boolean` and `enum` are usable as matrix values.

Watch the credit maths: three Node versions times two OSes is six jobs, and if each of those also sets `parallelism: 4` you have just scheduled 24 containers per push. Most teams run the full matrix nightly and a single representative cell on pull requests.

jobs:
  test:
    parameters:
      node-version:
        type: string
      os:
        type: string
    docker:
      - image: cimg/node:<< parameters.node-version >>
    steps: [checkout, run: npm test]

workflows:
  test-matrix:
    jobs:
      - test:
          matrix:
            parameters:
              node-version: ["18.20", "20.10", "22.1"]
              os: ["linux", "alpine"]
            exclude:
              - { node-version: "18.20", os: "alpine" }

Answer

A **command** is a named sequence of steps you can call from multiple jobs, DRY for repetitive workflows like 'restore deps + build + persist'. **Parameters** make commands (and jobs/executors) configurable: each parameter has a `type` (string, integer, boolean, enum, executor, steps, env_var_name) and a default. You reference them inside the body with `<< parameters.name >>`. This is the foundation of writing your own Orbs, Orbs are essentially packaged commands and jobs with parameters.

The subtleties: `<< parameters.x >>` is resolved by the config compiler before the job exists, so it can appear in places a shell variable never could, including a cache key, an image tag, or a `resource_class`. Conversely you cannot pass a runtime value into a parameter, which is why `type: env_var_name` exists: the consumer passes the identifier `PROD_TOKEN`, the command body writes `${<< parameters.token_var >>}`, and the secret itself never enters the YAML. The `type: steps` parameter is the most underused one, it lets a command accept a block of caller-supplied steps and wrap them, which is how you build 'run this inside a retry loop' or 'run this with the tunnel open' helpers.

Parameter names are scoped to the command or job that declares them, so a nested command cannot see its caller's parameters, and a missing default produces the compile error 'Missing required argument'. Reusable commands also make `circleci config process` output much larger, which is fine, but remember that CircleCI enforces a compiled config size limit, so extremely repetitive expansion is a real ceiling in big monorepos.

commands:
  install-deps:
    parameters:
      cache-version:
        type: string
        default: "v1"
      lockfile:
        type: string
        default: package-lock.json
    steps:
      - restore_cache:
          keys:
            - << parameters.cache-version >>-deps-{{ checksum "<< parameters.lockfile >>" }}
      - run: npm ci
      - save_cache:
          key: << parameters.cache-version >>-deps-{{ checksum "<< parameters.lockfile >>" }}
          paths: [node_modules]

jobs:
  test:
    docker: [{ image: cimg/node:20.10 }]
    steps:
      - checkout
      - install-deps:
          cache-version: v2
      - run: npm test

Answer

CircleCI issues a signed OIDC JWT to every job at `$CIRCLE_OIDC_TOKEN`. You configure AWS IAM to trust CircleCI's OIDC provider (`https://oidc.circleci.com/org/<org-id>`), create an IAM role with a trust policy that scopes by `oidc.circleci.com/org/<org-id>:sub` (which includes project + context), then `aws sts assume-role-with-web-identity` from the job. Net result: no long-lived `AWS_ACCESS_KEY_ID` in CircleCI, credentials are short-lived (15 min) and tightly scoped.

This is the modern best practice and what large Indian fintechs like Razorpay use to satisfy compliance audits. Implementation details a senior interviewer will chase: the token's `iss` is `https://oidc.circleci.com/org/<org-id>`, `aud` is the org ID, and `sub` has the form `org/<org-id>/project/<project-id>/user/<user-id>`, so a trust policy that only checks `aud` grants every project in the org access to the role. Scope the condition on `sub` with a `StringLike` on the project ID, and note that the `user` segment is the triggering user, which makes 'only these engineers can assume the deploy role' expressible directly in IAM.

There is a second token, `$CIRCLE_OIDC_TOKEN_V2`, whose claims include the context IDs attached to the job, which is what lets you require that the job ran with a restricted context. Common failure: the token is only injected when the job has at least one context attached, so a job with no `context:` gets an empty variable and `assume-role-with-web-identity` fails with 'InvalidIdentityToken'. The same pattern works for GCP Workload Identity Federation and for HashiCorp Vault's JWT auth backend.

jobs:
  deploy:
    docker: [{ image: cimg/aws:2024.03 }]
    steps:
      - checkout
      - run:
          name: Assume AWS role via OIDC
          command: |
            CREDS=$(aws sts assume-role-with-web-identity \
              --role-arn arn:aws:iam::123:role/circleci-deploy \
              --role-session-name circleci-$CIRCLE_BUILD_NUM \
              --web-identity-token $CIRCLE_OIDC_TOKEN \
              --duration-seconds 900)
            export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r .Credentials.AccessKeyId)
            # ... etc
      - run: ./deploy.sh

Answer

**CircleCI**: best-in-class for hosted Linux/macOS, mature Orbs ecosystem, fast Docker layer caching, credit-based pricing. **GitHub Actions**: free for public repos, deeply integrated with PR/issue lifecycle, larger marketplace, but slower historically on big test suites and harder to do complex DAGs cleanly. **Jenkins**: self-hosted, infinitely extensible via plugins, but you own the infrastructure, security patching, scaling, high operational cost. Strong in regulated industries that can't use SaaS. **GitLab CI**: tightly coupled to GitLab the platform, great for teams already on GitLab; runners model is similar to CircleCI's self-hosted. In India, the trend in 2026 is GitHub-hosted projects → GitHub Actions for simple flows, CircleCI for complex multi-arch or macOS-heavy work, Jenkins still common in banking/finance.

Answer the mechanism, not just the vibe. Reuse: CircleCI uses Orbs (versioned packages inlined at compile time), Actions uses composite actions and reusable workflows (resolved at run time, so a floating `@v4` tag is a live supply-chain surface in both). Conditional execution: Actions has `if:` with expression syntax and `paths:` filters at the trigger level, CircleCI has no path trigger at all and forces you into `path-filtering` or `setup: true` dynamic config, which is more work but produces a real generated pipeline.

Test sharding: CircleCI ships `circleci tests split --split-by=timings` as a first-class feature; Actions leaves you to build matrix sharding yourself. Self-hosting: Actions runners are per-repo or per-org processes you scale yourself, CircleCI runners register against a named `resource_class` and are billed per active runner rather than per minute. Cost model: Actions bills wall-clock minutes with a per-runner-size multiplier, CircleCI bills credits by resource class, so a `2xlarge` job is dramatically more expensive per minute and right-sizing matters more.

# CircleCI: fan-out then fan-in, resource class per job
workflows:
  ci:
    jobs:
      - lint
      - test:
          parallelism: 6
      - build:
          requires: [lint, test]

# GitHub Actions equivalent
# jobs:
#   lint: { runs-on: ubuntu-latest }
#   test:
#     strategy: { matrix: { shard: [1,2,3,4,5,6] } }
#   build:
#     needs: [lint, test]

Answer

Combine `type: approval` jobs with `requires:` chains and branch filters. A typical prod pipeline: test → deploy-canary (1% traffic) → automated-canary-checks → hold-for-prod (approval) → deploy-100%. Use contexts restricted to security groups so only authorised engineers can approve prod.

For instant rollback, keep the previous release artifact in workspace/S3 and have a separate `rollback` workflow triggered via API or manual rerun. Avoid approvals for routine deploys, they create human bottlenecks; reserve them for high-blast-radius changes (schema migrations, infra changes). The part candidates usually miss is that CircleCI itself does not know what a canary is: it can only start a job, so the progressive logic lives in whatever you call (Argo Rollouts, Flagger, an ECS deployment circuit breaker, a Spinnaker pipeline).

A CircleCI job's honest role is to push an immutable artifact tagged with `$CIRCLE_SHA1`, patch the desired state, then poll for a health verdict and exit non-zero if the analysis fails. Make the promotion job idempotent and re-runnable, because 'rerun failed job' is the fastest rollback path a tired on-call engineer has at 2am. Never gate the rollback path behind an approval, an approval on rollback is how a five-minute incident becomes a fifty-minute one.

Guard the deploy job with a check that the artifact under promotion is exactly the one that passed tests (compare digests, not tags, since a mutable `:latest` tag defeats the whole chain). Finally, use a workspace or the artifact store to carry the previous known-good digest forward so the rollback job does not need to query the registry while the registry may itself be the thing that is broken.

workflows:
  release:
    jobs:
      - build-image
      - deploy-canary:
          requires: [build-image]
          context: k8s-prod
          filters: { branches: { only: main } }
      - canary-analysis:
          requires: [deploy-canary]
      - hold-for-full-rollout:
          type: approval
          requires: [canary-analysis]
      - deploy-full:
          requires: [hold-for-full-rollout]
          context: k8s-prod
      - rollback:
          requires: [build-image]
          type: approval   # always available, never gated behind the canary

Answer

When you use the `docker` executor, your job runs INSIDE a container, so you can't run `docker build` or `docker compose` directly (no Docker daemon). `setup_remote_docker` provisions a separate Docker daemon on a remote host and configures `DOCKER_HOST` so subsequent `docker` commands work. You can also enable `docker_layer_caching: true` (extra credit cost) to persist image layers between runs, dramatically speeding up rebuilds. Alternative: use the `machine` executor which has a local Docker daemon, usually simpler if you need lots of Docker work.

The consequences of 'remote' are the exam question. Your job container and the remote daemon do not share a filesystem, so bind mounts (`-v $(pwd):/src`) mount empty directories, and they do not share a network namespace, so a container you start with `-p 8080:8080` is NOT reachable at `localhost:8080` from your steps. The workarounds are `docker cp` into a running container, building context into the image, or running the test itself inside a container on the same user-defined network. `docker_layer_caching: true` costs extra credits per job and only helps when the early layers are genuinely stable; a Dockerfile that runs `COPY . .` before `npm ci` will miss every time and you have paid for nothing.

In 2026 most teams get better results from BuildKit's registry cache (`docker buildx build --cache-from type=registry,ref=...`) because it is portable, works on the `machine` executor too, and does not depend on landing on the same cache-warm host. Also pin `version:` on `setup_remote_docker` only when you need a specific daemon, otherwise you get pinned to a deprecated build.

jobs:
  build-image:
    docker: [{ image: cimg/base:2024.03 }]
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - run: docker build -t myorg/app:$CIRCLE_SHA1 .
      - run: docker push myorg/app:$CIRCLE_SHA1

Answer

Two strategies: (1) **Path filtering**, use the `path-filtering` Orb in a setup workflow to inspect the diff and trigger only the relevant downstream pipeline. (2) **Dynamic config** (preferred in 2026), a setup job runs first, computes which packages changed (via `nx affected`, `turborepo`, or git diff), and generates a `continue_config.yml` on the fly that's then triggered. Combine with the `continuation` Orb. Without this, every PR rebuilds the entire monorepo and you burn credits unnecessarily.

CRED and Razorpay-style Nx/Turborepo setups are common in India in 2026. Three details decide whether it actually works. First, `base-revision` must be the right comparison point: on a feature branch you want the merge base with `main`, not `main`'s tip, or a busy `main` makes every PR look like it touched everything; `git merge-base HEAD origin/main` is the correct value to pass.

Second, path mappings are regexes over changed files and they are additive, so a shared `packages/config` change should map to `run-all true` rather than being enumerated per consumer. Third, selective builds break required status checks: if `run-api` is false, the `api-test` job never reports, and a branch protection rule requiring `api-test` blocks the merge forever. The fix is a cheap always-run job that reports the aggregate status, or marking the checks non-required and gating on a single `ci-complete` job that `requires:` whatever did run. Also decide what a merge queue does, because 'only build what changed' plus 'merge many PRs quickly' means the combination that lands on `main` was never tested together; a nightly full build is the usual insurance.

# .circleci/config.yml (the setup config)
version: 2.1
setup: true
orbs:
  path-filtering: circleci/path-filtering@1.1.0

workflows:
  filter-and-continue:
    jobs:
      - path-filtering/filter:
          config-path: .circleci/continue_config.yml
          mapping: |
            apps/web/.* run-web true
            apps/api/.* run-api true
            packages/.*/.* run-all true
          base-revision: main

Answer

Docker Hub rate-limits unauthenticated pulls (100 per IP per 6h for anonymous, 200 for free accounts). Shared CI runners share an IP pool, so a busy CircleCI org can hit limits even on a small project. Three mitigations: (1) **Authenticate**, set `DOCKERHUB_USERNAME` / `DOCKERHUB_PASSWORD` as a context, log in before any pull; raises your limit to 200/5000 per day depending on plan. (2) **Mirror the image**, push the base image to ECR/GCR/GHCR/your private registry and pull from there; no rate limit and faster pulls from the same cloud region. (3) **Use `cimg/*` convenience images**, CircleCI ships pre-built images on its own registry with no rate limit.

The interview signal here is whether you've actually run into this in production, it's a top-five 'why is my pipeline red today' incident. Know the symptom precisely: the job fails during image pull, before any of your steps run, with 'toomanyrequests: You have reached your pull rate limit' or a 429 from `registry-1.docker.io`, and it is intermittent because the limit is per source IP and shared with everyone else scheduled on that host. That intermittency is why teams waste hours blaming their own Dockerfile.

Two extra points worth making: authentication must be configured under the `auth:` key of each image in the config (or via the `docker/check` Orb), because a `docker login` in a step happens far too late for the executor's own pull, and rate limits also apply to service images, so a job with three sidecars burns three pulls per run. `docker manifest inspect` counts against the limit as well, which surprises people who added an image-freshness check. The durable fix is a pull-through cache or a mirrored copy in ECR/Artifact Registry in the same region as your runners, which also removes a cross-continent network hop from every job.

jobs:
  build:
    docker:
      - image: cimg/node:20.10   # CircleCI registry, no rate limit
      - image: myorg/postgres:15  # mirrored to ECR
        auth:
          username: $ECR_USER
          password: $ECR_TOKEN
    steps: [checkout, run: npm test]

Answer

Top wins: (1) **Right-size resource_class**, defaulting to `large` when `medium` suffices doubles your bill. Profile with `top` / CircleCI's resource metrics. (2) **Cache aggressively**, `npm ci` from cache is 10-30s, fresh is 2-3 min. (3) **Skip unnecessary jobs** via path filtering on monorepos. (4) **Use `when:` / `unless:`** to short-circuit jobs (e.g., skip integration tests for docs-only PRs). (5) **Avoid `docker_layer_caching` if you don't need it**, it costs an extra 200 credits/run. (6) **Don't run parallel CI on Dependabot/Renovate forks**, bot PRs can saturate your queue. (7) **Self-host runners** for steady-state work where you have idle on-prem capacity; cloud burst remains on CircleCI hosted. Measure before you cut: the Insights page and the `/insights/{project-slug}/workflows/{workflow}/jobs` API endpoint give per-job duration, credit consumption, and success rate, and the answer is almost always that two or three jobs account for most of the spend.

Two structural levers beat micro-optimisation. First, cancel redundant work: enable 'Auto-cancel redundant workflows' so a force-push does not leave three obsolete pipelines running, and add a `[skip ci]` convention for docs commits. Second, stop paying for spin-up you do not need: merging a 20-second lint job into the test job removes a whole container boot, a checkout, and a dependency restore.

Beware two false economies. Dropping from `large` to `medium` on a memory-bound test suite trades credits for OOM flakiness that costs more engineer-hours than it saves, and turning off `store_test_results` to save a few seconds destroys the timing data that makes `--split-by=timings` work. Storage is billed too, so set artifact and cache retention deliberately rather than uploading full Cypress videos on every green run.

version: 2.1

parameters:
  run-integration:
    type: boolean
    default: false

jobs:
  unit:
    docker: [{ image: cimg/node:20.10 }]
    resource_class: small        # right-sized, not the default medium
    steps: [checkout, run: npm ci && npm run test:unit]

  integration:
    machine: { image: ubuntu-2204:current }
    resource_class: large
    steps: [checkout, run: npm run test:int]

workflows:
  cheap-ci:
    jobs:
      - unit
      - integration:
          requires: [unit]
          # skip the expensive VM job unless the pipeline asked for it
          filters: { branches: { ignore: /docs\/.*/ } }

Answer

There are two unrelated features that both spell themselves `when`, and confusing them is a reliable interview filter. The first is the `when:` ATTRIBUTE on a step, which accepts exactly `on_success` (default), `on_fail`, or `always`, and controls whether that step runs given the job's status so far. That is how you upload logs after a failure.

The second is the `when:` STEP TYPE, which takes a `condition:` and a nested `steps:` list, and is evaluated at config-compile time against parameters, not at run time against shell state. Because it is compile-time, `condition: $MY_VAR` never works: the compiler has no idea what your shell will contain. The same compile-time `when:` and `unless:` exist at workflow level, where they accept logic statements built from `and`, `or`, `not`, `equal`, and `matches` (with `pattern` and `value`).

Workflow-level conditions read pipeline values such as `<< pipeline.git.branch >>`, `<< pipeline.parameters.x >>` and `<< pipeline.trigger_source >>`. Jobs themselves have no `when:` key, so 'run this job only if X' is expressed either as a workflow-level condition or by wrapping the job's body in conditional steps. If the decision depends on runtime data such as a git diff, no conditional will help and you need dynamic config.

version: 2.1
parameters:
  deploy-target:
    type: enum
    enum: ["none", "staging", "prod"]
    default: "none"

commands:
  maybe-notify:
    parameters:
      notify:
        type: boolean
        default: true
    steps:
      - when:                      # compile-time step type
          condition: << parameters.notify >>
          steps:
            - run: ./scripts/slack.sh

jobs:
  test:
    docker: [{ image: cimg/node:20.10 }]
    steps:
      - checkout
      - run: npm test
      - run:
          name: Upload logs even if tests failed
          when: always           # step attribute, run-time status
          command: ./scripts/upload-logs.sh

workflows:
  deploy:
    when:
      not:
        equal: ["none", << pipeline.parameters.deploy-target >>]
    jobs: [test]

Answer

`circleci tests split` only divides a file list across containers. `circleci tests run` wraps the actual execution: it reads the test list on stdin, applies the split itself, invokes your command through `xargs`, and reports which specific tests ran. That extra reporting is what enables the 'Rerun failed tests only' button, because CircleCI needs to know the mapping from test name to shard in order to re-dispatch just the failures on a rerun. Prerequisites people miss: the job must upload JUnit XML via `store_test_results` (that is both the timing source for `--split-by=timings` and the failure list), the wrapped command must accept test names as arguments, and the feature is gated to paid plans.

Behavioural notes worth raising: on a rerun, only the failing tests are dispatched, so global fixtures, database seeding, and build steps still execute in full and must be idempotent; shard balance is meaningless on a rerun because there may be three tests left across four containers, so drop parallelism for the rerun path if credit cost matters; and a test that fails during collection or setup, rather than during execution, may not appear in the JUnit output at all and therefore will not be re-dispatched. Exit codes propagate from the wrapped command, so a runner that exits 0 on failure defeats the whole mechanism.

jobs:
  test:
    docker: [{ image: cimg/python:3.12 }]
    parallelism: 6
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run:
          name: Run tests with rerun-failed support
          command: |
            circleci tests glob "tests/**/test_*.py" | \
              circleci tests run \
                --command="xargs python -m pytest --junitxml=test-results/junit.xml -v" \
                --split-by=timings \
                --timings-type=filename \
                --verbose
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: test-results

Answer

Two viable approaches with very different cost profiles. The emulated path uses one job: `docker buildx create --use`, then `docker buildx build --platform linux/amd64,linux/arm64 --push`. QEMU handles the foreign architecture, which is simple but often several times slower for anything that compiles native code, and it is a common cause of jobs hitting `no_output_timeout` during a long `pip install` or `cargo build`.

The native path builds each architecture on its own executor, an x86 `machine` job and an `arm.medium` job, pushes each as a per-architecture tag, then joins them in a third job with `docker buildx imagetools create -t repo:1.2.3 repo:1.2.3-amd64 repo:1.2.3-arm64`. Native is usually two to five times faster for compiled languages and it parallelises, so the wall clock is one architecture's build time rather than the sum. Two errors to recognise: `docker exporter does not currently support exporting manifest lists` means you tried to build multi-platform without `--push`, since the local daemon image store cannot hold a manifest list; and `exec format error` at runtime means a single-arch image got pushed under a tag something else pulled on a different architecture. Always verify the result with `docker buildx imagetools inspect`.

jobs:
  build-arch:
    parameters:
      arch: { type: string }
    machine:
      image: ubuntu-2204:current
    resource_class: << parameters.arch >>
    steps:
      - checkout
      - run: |
          echo "$REG_TOKEN" | docker login -u "$REG_USER" --password-stdin "$REG"
          SUFFIX=$([ "<< parameters.arch >>" = "arm.medium" ] && echo arm64 || echo amd64)
          docker build -t "$REG/app:$CIRCLE_SHA1-$SUFFIX" .
          docker push "$REG/app:$CIRCLE_SHA1-$SUFFIX"

  join-manifest:
    docker: [{ image: cimg/base:2024.03 }]
    steps:
      - setup_remote_docker
      - run: |
          docker buildx imagetools create \
            -t "$REG/app:$CIRCLE_SHA1" \
            "$REG/app:$CIRCLE_SHA1-amd64" "$REG/app:$CIRCLE_SHA1-arm64"
          docker buildx imagetools inspect "$REG/app:$CIRCLE_SHA1"

workflows:
  images:
    jobs:
      - build-arch:
          matrix:
            parameters:
              arch: ["large", "arm.medium"]
      - join-manifest:
          requires: [build-arch]

Answer

Pipeline parameters are typed inputs declared in a top-level `parameters:` block (types `string`, `boolean`, `integer`, `enum`) and read anywhere in the config as `<< pipeline.parameters.name >>`. They are resolved at config-compile time, which is what makes them usable in places a shell variable can never appear: a workflow-level `when:`, a `resource_class`, an image tag, or a cache key. Three things can set them: a v2 API trigger, a scheduled pipeline, and a `continuation/continue` call from a `setup: true` config.

On a normal push they take their declared defaults, so a config using parameters still works for ordinary webhook builds. To trigger from the API, POST to `/api/v2/project/{project-slug}/pipeline` with a `Circle-Token` header, a `branch` or `tag`, and a `parameters` object. The API rejects anything not declared with a 400 and the message that the parameter is not in the config, and it also rejects parameters on a branch whose config does not declare them, which is why a rename must land on every active branch before you switch the caller.

Alongside parameters you get read-only pipeline values: `<< pipeline.number >>`, `<< pipeline.id >>`, `<< pipeline.git.branch >>`, `<< pipeline.git.revision >>`, and `<< pipeline.trigger_source >>`. Use the API trigger for release promotion from a chat bot or an internal deploy console.

version: 2.1

parameters:
  service:
    type: enum
    enum: ["api", "web", "worker"]
    default: "api"
  image-tag:
    type: string
    default: "latest"

jobs:
  deploy:
    docker: [{ image: cimg/aws:2024.03 }]
    steps:
      - run: ./deploy.sh << pipeline.parameters.service >> << pipeline.parameters.image-tag >>

workflows:
  manual-deploy:
    when:
      equal: ["api", << pipeline.parameters.service >>]
    jobs:
      - deploy:
          context: aws-prod

# Trigger it:
# curl -X POST https://circleci.com/api/v2/project/gh/acme/platform/pipeline \
#   -H "Circle-Token: $CIRCLE_TOKEN" -H 'Content-Type: application/json' \
#   -d '{"branch":"main","parameters":{"service":"api","image-tag":"1.8.2"}}'

Answer

CircleCI scans step output for the literal values of context and project environment variables and replaces matches with `****`. That is a safety net, not a control, and it fails in predictable ways. Masking only applies to values above a short length threshold, so a four-character token or a numeric account ID passes straight through.

It only matches verbatim, so `echo $TOKEN | base64`, `gzip`, JSON-encoding, or a value split across lines by `set -x` all escape the filter. It applies to the log stream only, so a secret written into an artifact, a JUnit XML file, or a core dump is stored and served unmasked. And `curl -v` prints the `Authorization` header your script assembled, which the masker never saw as a variable.

The concrete habits that matter: turn off `set -x` in any step that touches credentials, pass tokens to tools with `--password-stdin` or `--data @file` instead of on the command line where `ps` and the log both see them, never `env` or `printenv` in a debug step, review what `store_artifacts` uploads, and remember that anyone with project write access can open an SSH rerun and read every variable directly. The durable fix is short-lived OIDC credentials so a leaked value expires in minutes.

steps:
  - run:
      name: Registry login without exposing the token
      command: |
        set +x                       # never trace a credential step
        echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
  - run:
      name: API call without the token on the command line
      command: |
        printf '{"ref":"%s"}' "$CIRCLE_SHA1" > /tmp/body.json
        curl -sS -X POST https://internal.example.com/deploys \
          -H "Authorization: Bearer $DEPLOY_TOKEN" \
          --data @/tmp/body.json
  - run:
      name: Scrub before uploading anything
      when: always
      command: |
        grep -rIl "$DEPLOY_TOKEN" /tmp/logs 2>/dev/null | xargs -r rm -f
  - store_artifacts:
      path: /tmp/logs

Answer

Three separate limits exist and candidates usually know only one. Per step, `no_output_timeout` kills a `run` step that has produced no stdout or stderr for a given duration; the default is 10 minutes and the failure reads 'Too long with no output (exceeded 10m0s)'. It takes a Go duration string, so `no_output_timeout: 25m` or `1h30m`.

Per job, cloud jobs are capped at 5 hours regardless of output, and hitting that cap bills the full five hours. Per workflow there is no timeout at all, which is why an approval job left on hold can sit for weeks (harmlessly, since it consumes nothing) while a genuinely hung `terraform apply` keeps a large executor alive at full cost. The wrong fix is a heartbeat that prints a dot every 30 seconds, because it converts a 10-minute failure into a 5-hour one.

The right approach is to bound the operation itself with `timeout 20m ./long-thing.sh`, which gives you exit code 124 and a clear signal, and to raise `no_output_timeout` only for steps that are genuinely quiet, such as a large image push. Also enable auto-cancel of redundant workflows so a rapid series of pushes does not leave three obsolete pipelines running, and script bulk cancellation through `POST /api/v2/workflow/{id}/cancel` when an incident leaves work stuck.

steps:
  - run:
      name: Terraform apply, hard-bounded
      no_output_timeout: 30m
      command: |
        timeout --signal=TERM --kill-after=60s 25m terraform apply -auto-approve tfplan
        code=$?
        if [ $code -eq 124 ]; then
          echo "apply exceeded 25m, check for a held state lock"
          exit 1
        fi
        exit $code
  - run:
      name: Quiet image push needs a longer no-output window
      no_output_timeout: 20m
      command: docker push "$REG/app:$CIRCLE_SHA1"

Answer

Under the `docker` executor, every image after the first is a service container in the same network namespace as your primary, so Postgres is `localhost:5432` and Redis is `localhost:6379`, with no hostnames and no links. Each secondary image gets its own `environment:` block, which is where `POSTGRES_USER`, `POSTGRES_DB` and `POSTGRES_PASSWORD` go. The critical detail is that CircleCI starts all containers and then immediately runs your first step, with no readiness check.

Postgres typically needs a few seconds to initialise, so a test suite that connects on step one fails with 'connection refused' on cold runs and passes on warm ones, which is one of the most common sources of 'randomly failing' pipelines. Wait explicitly with `dockerize -wait tcp://localhost:5432 -timeout 1m` (present on the `cimg` convenience images) or a bounded `pg_isready` loop. Further gotchas: service containers have no persistent volume, so state is gone between jobs and cannot be cached; their logs do not appear in the CircleCI UI, so a service that crashes on a bad env var looks like a network problem; and the memory of all containers counts against the job's `resource_class`, so a `medium` running Postgres plus Elasticsearch will OOM. On the `machine` executor you would use Docker Compose with a healthcheck instead.

jobs:
  api-tests:
    docker:
      - image: cimg/node:20.10
        environment:
          DATABASE_URL: postgres://app:secret@localhost:5432/app_test
          REDIS_URL: redis://localhost:6379
      - image: cimg/postgres:16.2
        environment:
          POSTGRES_USER: app
          POSTGRES_DB: app_test
          POSTGRES_PASSWORD: secret
      - image: cimg/redis:7.2
    resource_class: large
    steps:
      - checkout
      - run:
          name: Wait for services
          command: |
            dockerize -wait tcp://localhost:5432 -wait tcp://localhost:6379 -timeout 90s
      - run: npm ci
      - run: npm run db:migrate && npm run test:api
      - store_test_results:
          path: test-results

Answer

Dynamic config lets a `setup: true` workflow generate the actual config at runtime and 'continue' execution with it. Use cases: (1) monorepo selective builds based on `git diff`, (2) computing matrix parameters from external systems (e.g., querying which services have unmerged migrations), (3) injecting feature-flag-driven jobs, (4) generating different pipelines per branch type without giant `when:` blocks. The flow: setup workflow runs → computes JSON parameters and/or a `continue_config.yml` → calls the `continuation/continue` command with the generated config.

The continuation runs as if it had been the original pipeline. This pattern replaced 'config generators' that wrote YAML in Python or Bash externally. Combined with the `path-filtering` Orb, this is how scaled monorepos (Coinbase, Stitch Fix, CRED) keep CI fast and cheap in 2026.

Mechanics and constraints: the setup config must declare `setup: true` at the top level AND the project must have dynamic config enabled in Advanced Settings, otherwise CircleCI runs the setup workflow as an ordinary pipeline and your jobs simply never appear. Exactly one continuation is allowed per pipeline; a second `continuation/continue` call fails with 'Cannot continue a pipeline that has already been continued', so branching logic has to be resolved into a single generated document. Parameters passed to the continuation must already be declared in the generated config's `parameters:` block, and they arrive as a JSON string, so a type mismatch surfaces as a compile error on the continue call rather than in your generator.

The generated config is compiled fresh, meaning it can pull in different Orb versions than the setup config used. Debug it by having the generator also `store_artifacts` the YAML it produced, then run `circleci config validate` on that artifact locally; without that, a broken generator gives you a red setup job and no visible pipeline at all. Keep the generator deterministic, since a generator that reads mutable external state makes reruns non-reproducible.

version: 2.1
setup: true
orbs:
  continuation: circleci/continuation@1.0.0

jobs:
  generate:
    docker: [{ image: cimg/python:3.12 }]
    steps:
      - checkout
      - run:
          name: Generate continue_config.yml
          command: python scripts/generate_pipeline.py > /tmp/continue.yml
      - continuation/continue:
          configuration_path: /tmp/continue.yml
          parameters: '{"deploy_envs":"staging,prod"}'

workflows:
  setup:
    jobs: [generate]

Answer

Compliance-heavy CI (SOC2, RBI guidelines for Indian fintech, PCI-DSS for card flows) has non-negotiable requirements: (1) **No long-lived secrets**, use OIDC for cloud auth, short-lived tokens everywhere. (2) **Audit trail**, every deploy logs who approved, what artifact was deployed, and the commit SHA + signed git tag. (3) **Approval gates**, production deploys require two-person sign-off via approval jobs gated by security-group-restricted contexts. (4) **Signed artifacts**, sign Docker images with Cosign/Sigstore in CI, verify at admission control (Kyverno/Gatekeeper) in Kubernetes. (5) **SBOM generation**, Syft on every build for supply chain audit. (6) **Self-hosted runners in your VPC** for any job touching production data, even hashed customer records shouldn't leave your network boundary. (7) **Branch protection**, `main` requires PR review, signed commits, and a green CircleCI status check. Razorpay's setup includes all of the above plus periodic third-party audit of the CircleCI config itself. Two things separate a real answer from a checklist.

First, name the enforcement point for each control, because a control that lives only in YAML is advisory: config policies (OPA/Rego, pushed with `circleci policy push`) reject non-compliant configs at compile time, context security groups gate credential access at schedule time, and Kyverno or Gatekeeper verifying a Cosign signature blocks an unsigned image at admission time. Only the last one survives an engineer editing `config.yml` on their own branch. Second, plan for the compromise case: CircleCI itself was breached in January 2023 and customers had to rotate every secret stored in the platform, which is the strongest practical argument for OIDC over stored keys, for a documented rotation runbook with an owner, and for storing artifact digests outside the CI provider. Retention is also a requirement, not a default: CircleCI's job logs and artifacts expire, so ship build metadata (commit SHA, approver, image digest, SBOM hash) into your own immutable store such as S3 Object Lock or a WORM bucket if your auditor asks for a seven-year trail.

jobs:
  build-signed:
    docker: [{ image: cimg/base:2024.03 }]
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Build and push by digest
          command: |
            docker build -t $REG/app:$CIRCLE_SHA1 .
            docker push $REG/app:$CIRCLE_SHA1
            DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' $REG/app:$CIRCLE_SHA1)
            echo "$DIGEST" > /tmp/digest.txt
      - run:
          name: Keyless sign with Cosign via CircleCI OIDC
          command: |
            cosign sign --yes --identity-token "$CIRCLE_OIDC_TOKEN_V2" "$(cat /tmp/digest.txt)"
      - run:
          name: Generate SBOM
          command: syft "$(cat /tmp/digest.txt)" -o spdx-json > /tmp/sbom.json
      - store_artifacts:
          path: /tmp/sbom.json

Answer

A self-hosted runner is a CircleCI agent you install on your own infrastructure (bare metal, VM, Kubernetes, or a developer laptop for niche cases). The agent registers with CircleCI cloud and polls for jobs targeted at its `resource_class`. Use them when: (1) you need access to private network resources (on-prem DBs, internal artifact registries), (2) you need specialised hardware (GPUs you already own, ARM bare metal), (3) your compliance regime forbids running CI on shared infrastructure (banking, healthcare), (4) you have idle capacity and want to offload steady-state load from credit-billed cloud.

Trade-offs: you operate the infrastructure (patching, scaling, monitoring); CircleCI provides the orchestration and UI. The 2026 Kubernetes runner deploys as a Helm chart with horizontal autoscaling on job queue depth. Pricing is per active runner per month, not per-minute, cheaper than cloud at high utilisation.

Operational details that only show up once you run them: the machine runner executes jobs as a local user on the host, NOT in a container, so there is no clean environment between jobs. Leftover files, a stale `node_modules`, a still-listening port from a crashed test server, or a `git` index lock will contaminate the next build, and 'works on cloud, fails on runner' is almost always this. Either use the container runner (the Kubernetes flavour, which does give you a fresh pod per job) or make the job explicitly clean its working directory.

Second, resource classes are namespaced (`my-org/internal-arm64`) and a job targeting a class with zero healthy runners does not error, it queues indefinitely, so alert on queue depth and runner heartbeat. Third, secrets still flow from CircleCI cloud to your runner, and the runner needs outbound HTTPS to CircleCI, so the network isolation story is 'no inbound' rather than 'air-gapped'. Fourth, `docker_layer_caching` and `setup_remote_docker` are cloud features that do not apply, you use the host's own daemon instead. Autoscale the Kubernetes runner on pending-job count, not CPU.

# Job targets the self-hosted resource class
jobs:
  build-on-prem:
    machine: true
    resource_class: my-org/internal-arm64
    steps:
      - checkout
      - run: ./build-arm.sh

Answer

Stepwise approach: (1) **Rerun with SSH**, CircleCI's 'Rerun job with SSH' opens an SSH tunnel into a re-spawned container that pauses after the failing step. You can reproduce manually with the exact env. (2) **Capture artifacts on failure**, `when: on_fail` blocks that save logs, core dumps, and `docker ps -a` output. (3) **Look for ordering bugs**, flaky tests usually mean shared mutable state (filesystem, port, DB). Run with `--randomize` to surface them. (4) **Resource starvation**, `medium` containers have 4 GB RAM; OOM-killed processes don't always print a message.

Inspect `dmesg` or bump to `large` and see if flakiness disappears. (5) **Network flakiness**, third-party services (DockerHub, npm) rate-limit unauthenticated pulls; authenticate and add retries. (6) **Test isolation**, split flaky tests into a separate job with `parallelism: 1` and run them last so they don't block the rest. (7) **Use `circleci-cli local execute`** to reproduce a single job on your workstation. The cardinal rule: never `if: failure() { retry }` to mask flakiness, it accumulates technical debt and hides regressions. Before any of that, quantify it: the Tests tab and the Insights 'flaky tests' view identify tests that passed and failed on the same commit, which converts 'CI is flaky' into a ranked list of five test names you can actually fix.

Two CircleCI-specific causes are worth naming. Parallelism plus shared state: shards that all write to the same S3 prefix, Redis keyspace, or database schema fail only when the timing overlaps, which is why the failure follows load rather than code. And cache poisoning: a partial-key `restore_cache` hit can hand a job a dependency tree built from a different lockfile, producing a failure that vanishes the moment someone bumps the cache prefix.

When you do reach for retries, scope them to the specific network call with `curl --retry 3 --retry-connrefused` or a bounded `until` loop, never to the whole job, and record every retry so the rate is visible. Rerun-with-SSH sessions stay open for about two hours and cost credits the entire time, so terminate them when you are done.

steps:
  - run:
      name: Wait for Postgres, bounded retry on the flaky part only
      command: |
        for i in $(seq 1 30); do
          nc -z localhost 5432 && break
          echo "waiting for db ($i/30)"; sleep 2
        done
        nc -z localhost 5432 || { echo "db never came up"; exit 1; }
  - run:
      name: Isolate DB state per parallel shard
      command: |
        export TEST_SCHEMA="ci_${CIRCLE_BUILD_NUM}_${CIRCLE_NODE_INDEX}"
        npm run test:integration
  - run:
      name: Collect diagnostics when it fails
      when: on_fail
      command: |
        free -m; dmesg | tail -50 || true
        cp -r /tmp/logs /tmp/diagnostics || true
  - store_artifacts:
      path: /tmp/diagnostics

Answer

Config policies are Rego (OPA) rules stored at the organisation level and evaluated by CircleCI at config-compile time, before any job is scheduled. That placement is the whole point: unlike a linting job inside the pipeline, a policy cannot be bypassed by editing `config.yml` on a branch, because the compile itself is rejected. You author policies in a directory, test them locally with `circleci policy test ./policies` and `circleci policy decide ./policies --input .circleci/config.yml`, then publish with `circleci policy push ./policies --owner-id <org-id>`.

Each rule is switched on with `enable_rule["name"]` and carries an enforcement level: `soft_fail` records a violation and lets the pipeline run, `hard_fail` blocks the compile. The sane rollout is to ship everything as `soft_fail` first, watch the decision logs for a couple of weeks to find the repos that will break, fix them, then promote to `hard_fail`. Typical rules for a platform team: ban community Orbs outside an allowlist, require that any job named `deploy-*` uses an approved context, forbid `resource_class: 2xlarge` outside a named set of projects, require `store_test_results` on test jobs, and reject floating Orb tags such as `@volatile`. The feature sits on CircleCI's higher-tier plans, and the practical limit is that policies see the compiled config, so they reason about the expanded result rather than your source YAML.

# policies/orbs.rego
package org

import data.circleci.config

policy_name["orb_governance"]

allowed := {"circleci/node", "circleci/aws-cli", "acme/deploy"}

# Set-valued rule: one reason string per violation
approved_orbs_only[reason] {
  some name
  config.orbs[name]
  not allowed[name]
  reason := sprintf("orb %q is not on the approved list", [name])
}

# A floating tag lets someone else change our pipeline with no commit here
approved_orbs_only[reason] {
  some name
  ref := config.orbs[name]
  endswith(ref, "volatile")
  reason := sprintf("orb %q must be pinned to an exact version", [name])
}

enable_rule["approved_orbs_only"]
hard_fail["approved_orbs_only"]

# Test locally, then ship org-wide:
#   circleci policy test ./policies
#   circleci policy decide ./policies --input .circleci/config.yml
#   circleci policy push ./policies --owner-id "$ORG_ID"

Answer

Start by internalising that CircleCI has two entirely separate substitution phases. Compile time resolves `<< parameters.x >>`, `<< pipeline.parameters.x >>`, `<< matrix.x >>` and Orb references, producing a static document. Run time resolves `$VAR` and `${VAR}` inside a shell.

Almost every confusing error is a phase mix-up. `circleci config process .circleci/config.yml` prints the phase-one output and answers most questions instantly: if the command you called is missing from the expanded document, either the Orb never imported (check the `orbs:` key and, for a private Orb, pass `--org-slug`), or you referenced `myorb/deploy` where the Orb actually exports `deploy-service`. 'Unexpected argument(s)' means you passed a parameter the command does not declare, usually after an Orb minor bump changed the signature; `circleci orb source acme/deploy@2.1.0` shows the real declaration. Parameters are scoped to the job or command that declares them, so a nested command cannot read its caller's parameters and produces 'variable not defined' rather than falling back.

A parameter with no default is mandatory and fails with 'Missing required argument'. Two structural gotchas: pipeline values are only available at compile time, so `echo << pipeline.git.branch >>` works but a shell script cannot read them later unless you export them into `$BASH_ENV`; and heavily parameterised monorepo configs can exceed CircleCI's compiled-config size ceiling, at which point you must move to dynamic config.

# 1. What does CircleCI actually see?
circleci config process .circleci/config.yml | less

# 2. Private orb? resolution needs the org
circleci config validate --org-slug gh/acme .circleci/config.yml

# 3. What does the orb really export?
circleci orb info acme/deploy
circleci orb source acme/deploy@2.1.0 | grep -A5 '^  deploy'

# 4. Compile-time vs run-time in one job
#    << >> is substituted before the shell exists; $VAR after.
#    steps:
#      - run:
#          name: Show both phases
#          command: |
#            echo "compiled branch: << pipeline.git.branch >>"
#            echo "runtime branch:  $CIRCLE_BRANCH"
#            echo 'export BUILD_REF=<< pipeline.git.revision >>' >> "$BASH_ENV"

Answer

Two attack surfaces. The fork PR surface: a pull request from a fork can rewrite `.circleci/config.yml`, so if the project builds fork PRs with secrets attached, an attacker's config runs with your credentials. Keep 'Build forked pull requests' off unless you need it, keep 'Pass secrets to builds from forked pull requests' off unconditionally, and never point fork PRs at self-hosted runners, which have network access to your internal estate and no fresh-machine guarantee between jobs.

If you must run untrusted code, run it in a project with no contexts, no deploy credentials, and `npm ci --ignore-scripts` so a malicious `postinstall` cannot execute. The dependency surface: Orbs are inlined at compile time from a registry you do not control, so a floating `@5` tag lets another party change your pipeline with no commit in your repo. Pin exact versions, review with `circleci orb source`, restrict imports with a config policy, and treat a community Orb with the same suspicion as an unaudited npm package. Beyond that, use OIDC so there is nothing long-lived to steal, restrict contexts by security group and branch, require signed commits and reviews on the branch that can deploy, and maintain a rotation runbook: the January 2023 CircleCI breach forced customers to rotate every stored secret, and the teams that recovered quickly were the ones who already knew what was stored where.

version: 2.1

orbs:
  node: circleci/node@5.2.0        # exact pin, never @5 or @volatile

jobs:
  untrusted-pr-check:
    docker: [{ image: cimg/node:20.10 }]
    # no context: attached, so no credentials and no OIDC token
    steps:
      - checkout
      - run: npm ci --ignore-scripts
      - run: npm run lint && npm run test:unit

  deploy:
    docker: [{ image: cimg/aws:2024.03 }]
    steps:
      - run: ./deploy.sh

workflows:
  ci:
    jobs:
      - untrusted-pr-check
      - deploy:
          requires: [untrusted-pr-check]
          context: aws-prod          # restricted to security group + main
          filters:
            branches:
              only: main

Answer

First separate queue time from run time, because they have disjoint causes and the Insights page reports both. If queue time grew, you are hitting plan concurrency, someone added a large matrix that saturates the org, or a self-hosted `resource_class` lost its runners and jobs are waiting on capacity that no longer exists. If run time grew, work through the job timeline top down.

Check 'Preparing environment': a jump here means the image changed and is no longer warm in CircleCI's cache, which happens when someone bumps `cimg/node:20.10` to a tag that is rarely pulled. Check the `restore_cache` step for 'No cache is found for key', the single most common cause of a sudden slowdown, usually triggered by a lockfile churn, a cache-prefix bump, or a key that accidentally includes `{{ epoch }}` or a branch name. Check whether `store_test_results` still uploads, because without timing data `--split-by=timings` degrades to filename splitting and one shard carries the slow tests while the others idle.

Then run the control experiment: rerun a pipeline from a commit that was fast last week. If it is now slow, the cause is environmental (image, registry, upstream package host, CircleCI itself) and the code is innocent. If it is still fast, bisect the commits in between. Pull per-job history from `/api/v2/insights/{project-slug}/workflows/{workflow}/jobs` to see exactly which job's p95 moved and when.

# Per-job duration and credit history, before vs after
curl -sS -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/acme/platform/workflows/ci/jobs?branch=main&reporting-window=last-30-days" \
  | jq -r '.items[] | [.name, .metrics.duration_metrics.p95, .metrics.total_credits_used] | @tsv' \
  | sort -k2 -nr | head

# Workflow-level queue time vs run time
curl -sS -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/acme/platform/workflows/ci?branch=main" \
  | jq '.items[0].metrics | {p95_duration: .duration_metrics.p95, median_queue: .median_credits_used}'

# Was it a cache miss? grep the job log for the tell
#   "No cache is found for key: v3-deps-..."

Answer

Measure before cutting: the Tests tab ranks slowest tests, and the job timeline shows how much of the 40 minutes is checkout, dependency install, build, and actual testing. In most suites the true test execution is under half of it. Then attack in order of leverage.

Split the monolith into unit, integration and end-to-end jobs so they run concurrently instead of serially, and build once, persisting artifacts to a workspace rather than rebuilding in every job. Cache the package manager store keyed on the lockfile so install drops from minutes to seconds. Apply `parallelism` with `circleci tests split --split-by=timings`, which requires `store_test_results` to be uploading, and verify shard balance afterwards rather than assuming it.

Right-size the executor for the bottleneck job: a Jest or pytest-xdist suite is CPU-bound and genuinely benefits from `large`, while a job that just waits on network does not. One trap deserves naming: inside a `docker` executor, `nproc` reports the HOST's CPU count, not your container's limit, so `--maxWorkers=$(nproc)` or `-n auto` spawns far too many workers and the suite gets slower through thrashing. Hard-code worker counts to match the resource class. Finally, accept the arithmetic: total wall clock is the longest path through the DAG, so once every job is under 10 minutes, further parallelism buys nothing and the remaining work is making individual tests faster.

jobs:
  build-once:
    docker: [{ image: cimg/node:20.10 }]
    resource_class: large
    steps:
      - checkout
      - restore_cache: { keys: ["v4-npm-{{ checksum \"package-lock.json\" }}"] }
      - run: npm ci
      - save_cache:
          key: v4-npm-{{ checksum "package-lock.json" }}
          paths: ["~/.npm"]
      - run: npm run build
      - persist_to_workspace:
          root: .
          paths: ["dist", "node_modules"]

  unit:
    docker: [{ image: cimg/node:20.10 }]
    resource_class: large        # 4 vCPU
    parallelism: 8
    steps:
      - checkout
      - attach_workspace: { at: . }
      - run:
          command: |
            FILES=$(circleci tests glob "src/**/*.test.ts" | circleci tests split --split-by=timings)
            # 4 vCPU, not $(nproc) which reports the host's core count
            npx jest --maxWorkers=4 --ci --reporters=default \
              --reporters=jest-junit $FILES
      - store_test_results: { path: test-results }

workflows:
  fast-ci:
    jobs:
      - build-once
      - unit: { requires: [build-once] }

Answer

Treat this as a multi-quarter programme, not a weekend port. Phases: (1) **Inventory**, catalog every Jenkinsfile, shared library, plugin, and credential. Identify pipelines that depend on Jenkins-specific features (Groovy shared libs, parameterised builds with non-trivial UI, agent label affinity). (2) **Pilot**, pick one greenfield service and rebuild its pipeline in CircleCI 2.1, leveraging Orbs to replace Jenkins shared libraries.

Validate cost, speed, and developer experience. (3) **Translate patterns**, Jenkins stages → CircleCI jobs; Jenkins parallel blocks → workflow fan-out; Jenkins shared libs → custom Orb in your namespace; Jenkins credentials → contexts with security groups; Jenkins agent labels → resource_class + self-hosted runners. (4) **Move secrets**, never paste them in chat; use the CircleCI API or Terraform provider to provision contexts. (5) **Run in parallel**, both Jenkins and CircleCI run on every PR for 2-4 weeks; compare outcomes, fix divergences. (6) **Cut over per service**, flip the merge requirement to CircleCI's status check; decommission the Jenkins job. (7) **Tear down**, only after every active project has migrated AND archived projects are confirmed dead. Common pitfalls: underestimating Jenkins plugins with no Orb equivalent (e.g., custom HTML report publishers), missing the fact that some teams stored business logic IN Jenkinsfile rather than the application repo, and credit-cost surprise when high-volume Jenkins jobs run on much cheaper self-hosted hardware.

# Jenkins declarative pipeline
# pipeline {
#   agent { label 'linux' }
#   stages {
#     stage('Test')   { parallel { stage('unit'){...} stage('lint'){...} } }
#     stage('Deploy') { when { branch 'main' }
#                       steps { withCredentials([...]) { sh './deploy.sh' } } }
#   }
# }

# CircleCI equivalent
version: 2.1

workflows:
  ci:
    jobs:
      - unit                       # Jenkins parallel block -> workflow fan-out
      - lint
      - deploy:
          requires: [unit, lint]   # Jenkins stage order -> requires:
          context: prod-deploy     # withCredentials -> restricted context
          filters:
            branches: { only: main }   # when { branch 'main' } -> filters

# agent { label 'linux' }  -> resource_class / self-hosted resource class
# Jenkins shared library   -> an orb published in your own namespace