GitLab Interview Questions and Answers

Answer

GitLab looks for a configuration file at the repository root, by default .gitlab-ci.yml, whenever an event that can create a pipeline occurs: a push to a branch or tag, a merge request event, a scheduled pipeline, an API or trigger token call, a manual run from the Pipelines page, or a trigger from a parent or upstream pipeline. The file path is configurable under Settings, CI/CD, General pipelines, and it can even live in another project, which is how compliance teams centralise configuration. The value is exposed at runtime as CI_CONFIG_PATH.

The minimum valid file is a single job with a script. There is no required stages block: if you omit it, GitLab uses the default stages build, test and deploy, plus the always-present hidden stages .pre and .post which run first and last regardless of where they are declared. If the YAML is invalid, GitLab does not create a pipeline at all, it records a failed pipeline with a yaml invalid message, so validate with the Pipeline Editor tab, the /-/ci/lint endpoint, or glab ci lint before pushing. A common interview follow-up is what happens when the file exists but every job is filtered out by rules: GitLab reports the pipeline as having no jobs and, on a merge request, blocks the merge if pipelines are required to succeed.

# .gitlab-ci.yml, smallest useful example
stages:
  - build
  - test

build-app:
  stage: build
  image: node:22-alpine
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

unit-tests:
  stage: test
  image: node:22-alpine
  script:
    - npm ci
    - npm test

Answer

The top-level stages list defines the order of execution phases in a pipeline. Each job declares which phase it belongs to with the stage keyword. Jobs in the same stage run in parallel, subject to available runner concurrency, and the next stage starts only when every job in the previous stage has finished successfully, or has finished with allow_failure set to true.

A job with no stage keyword defaults to test. .pre and .post are special: .pre always runs before every other stage and .post always runs after, no matter where you place them in the stages list, and you do not need to declare them at all. They are useful for a configuration sanity check at the start and for a notification or cleanup job at the end. Two behaviours catch people out.

First, if no job in a stage is created because every job was filtered out by rules, the stage simply does not appear, it does not block the pipeline. Second, .pre and .post do not run if the pipeline contains no other jobs. Stage-based ordering is also the slowest way to run a pipeline, because a fast unit-test job waits for the slowest job in the same stage. That is why interviewers usually follow up by asking about needs and directed acyclic graph pipelines, which break the stage barrier.

stages:
  - build
  - test
  - deploy

lint-config:
  stage: .pre
  script:
    - ./scripts/validate-env.sh

compile:
  stage: build
  script:
    - make build

notify:
  stage: .post
  when: always
  script:
    - ./scripts/notify-slack.sh $CI_PIPELINE_STATUS

Answer

A GitLab Runner is a separate agent process that polls the GitLab instance for jobs and executes them. GitLab itself never runs your build, it only schedules it. A runner is registered against an instance, a group, or a single project, and the registration produces a runner authentication token stored in config.toml.

Recent GitLab versions have moved away from the old registration-token flow toward creating the runner in the UI or API first and then using the resulting glrt- authentication token, so mention that if you are asked about registration. The executor decides where the script actually runs. The shell executor runs commands directly on the runner host, which is fast but leaks state between jobs and is a security risk on shared runners.

The docker executor starts a fresh container per job from the image keyword, which is the common default. The kubernetes executor creates a pod per job and is the standard choice for teams already running clusters, since it scales horizontally and gives you resource requests and limits per job. The docker-autoscaler executor, backed by fleeting plugins for AWS, GCP or Azure, spins up ephemeral cloud VMs on demand and has replaced the deprecated docker+machine executor. There are also custom, ssh, and virtualbox or parallels executors for niche cases such as macOS and iOS builds.

# /etc/gitlab-runner/config.toml
concurrent = 8
check_interval = 3

[[runners]]
  name = 'docker-runner-mumbai'
  url = 'https://gitlab.example.com/'
  token = 'glrt-REDACTED'
  executor = 'docker'
  [runners.docker]
    image = 'alpine:3.20'
    privileged = false
    volumes = ['/cache']
    pull_policy = ['always']
  [runners.cache]
    Type = 's3'
    Shared = true
    [runners.cache.s3]
      ServerAddress = 's3.ap-south-1.amazonaws.com'
      BucketName = 'gitlab-runner-cache'
      BucketLocation = 'ap-south-1'

Answer

before_script and script are concatenated into a single shell invocation on the runner. That matters: if any command in before_script exits non-zero, the job fails immediately and script never runs. Because they share one shell, an export or a cd in before_script is visible to script, which people rely on for things like activating a virtualenv. after_script is different.

It runs in a brand new shell, in a fresh execution context, so exported variables and working-directory changes from the earlier phase are gone. It runs whether the job passed, failed, was cancelled, or timed out, which makes it the right place for log collection, cleanup, or uploading a debug bundle. Historically the exit code of after_script was ignored, and in more recent versions a failing after_script can be surfaced in the job status, so read the release notes for the version your organisation runs before relying on either behaviour.

Two further details show up in interviews. The runner uses set -e style behaviour by default on shell executors, but a multi-line YAML block counts as one command, so a pipeline of commands joined with && or a script file gives you more predictable failure semantics. And you can hoist common setup into default: before_script so every job inherits it, while any job that declares its own before_script overrides the default rather than appending to it.

default:
  image: python:3.12-slim
  before_script:
    - python -m venv .venv
    - source .venv/bin/activate
    - pip install -r requirements.txt

pytest:
  stage: test
  script:
    - pytest -q --junitxml=report.xml
  after_script:
    # New shell: .venv is NOT active here
    - echo "job status is $CI_JOB_STATUS"
    - tar czf debug-logs.tgz logs/ || true
  artifacts:
    when: always
    reports:
      junit: report.xml

Answer

The ones you reach for constantly are CI_COMMIT_SHA and CI_COMMIT_SHORT_SHA for image tags, CI_COMMIT_REF_NAME and CI_COMMIT_REF_SLUG for branch-derived names such as review app hostnames, CI_DEFAULT_BRANCH so your rules do not hardcode main, CI_PIPELINE_SOURCE to distinguish a merge request pipeline from a scheduled one, CI_PROJECT_DIR for absolute paths, CI_PROJECT_PATH and CI_PROJECT_ID for API calls, CI_MERGE_REQUEST_IID and CI_MERGE_REQUEST_TARGET_BRANCH_NAME on merge request pipelines, CI_REGISTRY, CI_REGISTRY_IMAGE, CI_REGISTRY_USER and CI_REGISTRY_PASSWORD for the built-in container registry, and CI_JOB_TOKEN for authenticating back to the API. Precedence runs roughly from most specific to least: variables passed when triggering the pipeline manually, by API, or from a schedule win first, then project-level CI/CD variables, then group-level, then instance-level, then variables inherited through dotenv artifacts from upstream jobs, then job-level variables in the YAML, then global variables in the YAML, then deployment variables, and finally predefined variables. The practical takeaway is that a value set in the UI beats the same key set in .gitlab-ci.yml, which is exactly what surprises people when a local YAML default refuses to take effect. Note also that CI_COMMIT_REF_SLUG is lowercased, truncated to 63 characters and stripped of unsafe characters, which is why it is safe in DNS names while CI_COMMIT_REF_NAME is not.

build-image:
  stage: build
  image: docker:27
  services:
    - docker:27-dind
  variables:
    IMAGE: $CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker build -t "$IMAGE" .
    - docker push "$IMAGE"
    - |
      if [ "$CI_COMMIT_REF_NAME" = "$CI_DEFAULT_BRANCH" ]; then
        docker tag "$IMAGE" "$CI_REGISTRY_IMAGE/api:latest"
        docker push "$CI_REGISTRY_IMAGE/api:latest"
      fi

Answer

Artifacts are job outputs, cache is job input acceleration. Artifacts are uploaded to GitLab object storage at the end of a job, are visible and downloadable in the UI and API, are versioned per job, and are automatically fetched by downstream jobs in later stages or by jobs listed in needs. They are the correct mechanism for build output, test reports, coverage files and anything a human or a later stage must consume.

Cache is an optimisation stored by the runner, keyed by a string you control, and is explicitly not guaranteed to exist. Its purpose is to avoid re-downloading node_modules, the Maven .m2 directory, pip wheels or Go module downloads. If a cache is missing, the job must still succeed, just more slowly.

That contract is the single most common design mistake: teams cache build output, a runner is replaced, and suddenly deployments ship an empty directory. The storage layer also differs. Artifacts always travel through the GitLab instance, so large artifacts cost bandwidth and object storage on every job.

Cache lives on the runner host by default, which means it is useless across autoscaled runners unless you configure distributed cache in the runners.cache section of config.toml pointing at S3, GCS or Azure Blob. Control artifact lifetime with expire_in, use artifacts:when to keep logs from failed jobs, and keep the total artifact size small, because a multi-gigabyte artifact uploaded by fifty jobs is a real bill.

build:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull-push
  artifacts:
    name: 'dist-$CI_COMMIT_SHORT_SHA'
    paths:
      - dist/
    expire_in: 1 week
    when: on_success

Answer

only and except are the legacy filtering keywords. They are still supported for backward compatibility but they are not receiving new features, and GitLab documentation recommends rules for all new configuration. The reasons are practical. only and except cannot be combined with rules in the same job, they have confusing implicit defaults, they use separate syntax for refs, variables and changes, and there is no way to change when or allow_failure conditionally. rules is a single ordered list evaluated top to bottom; the first matching entry decides whether the job is added to the pipeline and with which attributes, and if nothing matches the job is not created.

Each entry can carry if, changes, exists, when, allow_failure, variables, needs and interruptible, so one rule can say run this job only on the default branch, as a manual job, with allow_failure true, and with a different DEPLOY_ENV value. Two migration gotchas matter. First, only:refs defaults to branches and tags, whereas rules has no implicit filter, so a naive translation can suddenly run jobs on merge request pipelines you did not expect. Second, only:changes ignores the rest of the pipeline context, while rules:changes on a non-merge-request pipeline compares against the previous commit, which is unreliable on a new branch, so pair it with compare_to.

# Legacy
deploy-old:
  only:
    refs:
      - main
  except:
    variables:
      - $SKIP_DEPLOY
  script: ./deploy.sh

# Modern equivalent, with more control
deploy:
  script: ./deploy.sh
  rules:
    - if: $SKIP_DEPLOY
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      variables:
        DEPLOY_ENV: production
    - if: $CI_COMMIT_BRANCH =~ /^release\//
      variables:
        DEPLOY_ENV: staging

Answer

image names the container the job script runs inside when using the docker or kubernetes executor. services lists additional containers started alongside it on the same network, typically a database, cache or Docker daemon. Each service is reachable by a hostname derived from the image name: postgres:16 is reachable as postgres, and redis:7-alpine as redis. You can override this with the alias field, which is essential when you run two versions of the same image or when the derived name is awkward.

Services start before the job script and the runner waits for them, but only for the container to start, not for the application inside it to be ready. That is why a job connecting to Postgres in the first second frequently fails with connection refused: the correct fix is a readiness loop such as pg_isready in before_script, or the runner variable that sets a service healthcheck wait time. Service configuration is passed through environment variables in the job, for example POSTGRES_PASSWORD and POSTGRES_DB for the official Postgres image.

Use entrypoint and command when the default entrypoint does not fit. Debugging services used to be painful because their logs were hidden; setting CI_DEBUG_SERVICES to true surfaces service container logs in the job output, at the cost of potentially exposing service environment variables, so keep it off by default.

integration-tests:
  image: golang:1.23
  services:
    - name: postgres:16-alpine
      alias: db
    - name: redis:7-alpine
      alias: cache
  variables:
    POSTGRES_DB: app_test
    POSTGRES_USER: app
    POSTGRES_PASSWORD: secret
    DATABASE_URL: 'postgres://app:secret@db:5432/app_test?sslmode=disable'
    REDIS_URL: 'redis://cache:6379'
  before_script:
    - until pg_isready -h db -U app; do sleep 1; done
  script:
    - go test ./... -tags=integration

Answer

Runners advertise a set of tags, and jobs request runners with the tags keyword. A runner picks up a job only if the runner's tag set is a superset of the job's tags, so a job tagged docker and linux needs a runner carrying at least both. A runner can also be configured to run untagged jobs; if that option is off, any job without a tags list will never be picked up by it.

Beyond tags, matching depends on scope, whether the runner is instance-wide, assigned to the group, or assigned to the project, on whether the runner is locked to a specific project, on whether it is paused, and on protected status: a runner marked as protected only runs jobs on protected branches and protected tags. A pipeline stuck in pending with the message that this job is stuck because no runners match almost always means one of four things: the tag list has a typo, the only matching runner is paused or offline, the runner does not accept untagged jobs and the job has no tags, or the job is on a protected branch and the runner is not protected. On GitLab.com, a fifth possibility is that shared runner compute minutes for the namespace are exhausted. Check Settings, CI/CD, Runners for the green availability dot, and check the job detail page which names the exact reason.

build-arm:
  tags:
    - docker
    - arm64
  script:
    - make build ARCH=arm64

# Runner side, config.toml
# [[runners]]
#   name = 'arm-fleet'
#   executor = 'docker'
#   # tags configured in the UI: docker, arm64, linux
#   # 'Run untagged jobs' unchecked, so untagged jobs are ignored

Answer

The when keyword accepts on_success, on_failure, always, manual, delayed and never. A manual job appears in the pipeline graph with a play button and does nothing until someone clicks it or triggers it through the API. The critical interaction is with allow_failure.

For a manual job, allow_failure defaults to true, which makes the job optional: the pipeline is reported as successful even if the manual job is never run. Set allow_failure to false and the job becomes a blocking manual job, which puts the pipeline into the manual status and prevents subsequent stages from running until it is played. That is the standard pattern for a production deployment gate.

Note that a blocking manual job also blocks merge trains and any merge request setting that requires a successful pipeline, which surprises teams the first time they add a gate. when: delayed with start_in creates a scheduled job that runs after a specified wait, useful for a canary soak period before the full rollout. allow_failure also supports exit_codes, so a job can be tolerated only when it exits with a specific code, for example an advisory linter that exits 2 for warnings but 1 for hard errors. Finally, permission matters: only users with the Developer role or above on a protected environment can run the corresponding manual deployment job.

deploy-prod:
  stage: deploy
  environment:
    name: production
    url: https://app.example.com
  script:
    - ./scripts/deploy.sh production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false

soak-check:
  stage: deploy
  needs: [deploy-prod]
  when: delayed
  start_in: 15 minutes
  script:
    - ./scripts/check-error-rate.sh

advisory-lint:
  script:
    - ./scripts/lint.sh
  allow_failure:
    exit_codes:
      - 2

Answer

needs turns a stage-ordered pipeline into a directed acyclic graph. A job with needs starts as soon as the jobs it names have completed, even if those jobs are in an earlier stage that has not finished as a whole, and even if jobs in intermediate stages are still running. On a wide pipeline this is the single biggest wall-clock saving available, because a fast frontend test no longer waits for a slow backend integration suite that happens to share a stage. needs also controls artifact download: by default a job with needs downloads artifacts only from the jobs it names, not from every job in earlier stages, which cuts network time significantly.

You can suppress that with needs entries of the form job plus artifacts set to false. There are constraints an interviewer will check. A job cannot need a job in a later stage.

The graph must be acyclic. There is a documented limit on the number of jobs a single job can list in needs, historically ten and raised to fifty in later versions, so extremely wide fan-in needs restructuring. An empty needs list, written as needs: [], removes all dependencies and lets a job start in the very first moments of the pipeline, which is the trick used for a fast lint or configuration check that should fail within thirty seconds instead of waiting for a build stage.

stages: [build, test, deploy]

lint:
  stage: test
  needs: []          # starts immediately, ignores stage order
  script: ./scripts/lint.sh

build-api:
  stage: build
  script: make api
  artifacts:
    paths: [bin/api]

build-web:
  stage: build
  script: make web
  artifacts:
    paths: [dist/]

test-api:
  stage: test
  needs: [build-api]  # does not wait for build-web
  script: ./bin/api --selftest

test-web:
  stage: test
  needs:
    - job: build-web
      artifacts: true
  script: npx playwright test

Answer

CI_PIPELINE_SOURCE tells you why the pipeline exists. Common values are push for a branch or tag push, merge_request_event for a merge request pipeline, schedule for a pipeline schedule, web for a manual run from the UI, api and trigger for programmatic runs, pipeline for a multi-project trigger, and parent_pipeline for a child pipeline. A branch pipeline runs against the commit on the source branch only.

A merge request pipeline is created for the merge request itself, exposes the CI_MERGE_REQUEST family of variables, and shows results in the merge request widget. GitLab also offers merged results pipelines on paid tiers, which run against a temporary commit representing the source branch merged into the target branch, so you test the code as it will exist after the merge rather than the code as it exists on the branch. That catches semantic conflicts, two branches that merge cleanly but break together.

The practical problem candidates must know is duplicate pipelines: if you do nothing, pushing to a branch that has an open merge request creates both a branch pipeline and a merge request pipeline, doubling runner spend and confusing the merge request widget. The fix is workflow rules that skip the branch pipeline when an open merge request exists, using CI_OPEN_MERGE_REQUESTS.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG
    - if: $CI_PIPELINE_SOURCE == "schedule"

smoke:
  script: ./scripts/smoke.sh
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      variables:
        TARGET: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Answer

Protected branches restrict who can push and merge, and they are also the security boundary for CI secrets. A CI/CD variable marked as protected is injected only into jobs running on a protected branch or a protected tag. That is the mechanism that stops a contributor from opening a branch, adding a job that echoes your production database password, and reading it from the job log.

If your deployment job suddenly reports an empty credential, the first thing to check is whether the branch is protected. Masking is a separate and weaker control: a masked variable has its value replaced with the string that indicates a masked value wherever it appears in the job log. Masking has requirements, the value must be a single line, at least eight characters, and free of most whitespace and special characters, which is why some base64 blobs refuse to mask.

Masking does not stop exfiltration, a job can always base64 the value and print it, so masking is a hygiene control against accidental leaks and not a defence against a malicious contributor. Variables can also be of type File, in which case GitLab writes the value into a temporary file and sets the variable to that file path, which is how you feed a kubeconfig, a service account JSON, or a PEM key to a CLI that expects a path. Finally, restrict variables by environment scope so a staging job cannot see production credentials.

# Settings > CI/CD > Variables
#   GCP_SA_KEY   type: File   protected: yes  masked: no   scope: production
#   SENTRY_TOKEN type: Var    protected: yes  masked: yes  scope: *

deploy-prod:
  stage: deploy
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
  script:
    # GCP_SA_KEY is a path because the variable type is File
    - gcloud auth activate-service-account --key-file "$GCP_SA_KEY"
    - gcloud run deploy api --image "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA"

Answer

GitLab parses structured reports uploaded through artifacts:reports. For unit tests, produce a JUnit XML file and declare it under artifacts:reports:junit. GitLab then shows a Test summary panel in the merge request that lists new failures, newly fixed tests and total counts, comparing the head pipeline against the target branch's latest pipeline.

Almost every test runner can emit JUnit XML: pytest with the junitxml flag, Jest with jest-junit, Go with gotestsum, Maven and Gradle natively, PHPUnit with the log-junit flag. Always pair the report with artifacts:when set to always, otherwise a failing job uploads nothing and the widget shows no failures at all, which is the exact moment you need it. For coverage there are two separate things.

The coverage keyword takes a regular expression applied to the job log and extracts a single percentage that GitLab stores as the pipeline coverage value, shown on the merge request and available as a badge. Separately, artifacts:reports:coverage_report with coverage_format set to cobertura or jacoco uploads a machine-readable report so GitLab can annotate changed lines in the merge request diff with covered or uncovered markers. The older cobertura key was replaced by coverage_report, so if you find cobertura in an old pipeline, that is a migration item. Paths inside the Cobertura file must be relative to the repository root or the diff annotation silently does nothing.

pytest:
  stage: test
  image: python:3.12
  script:
    - pip install -r requirements.txt pytest pytest-cov
    - pytest --junitxml=report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    when: always
    expire_in: 1 week
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Answer

A hidden job is any top-level key starting with a dot. GitLab never runs it, so it exists purely as a template. extends pulls a hidden job's configuration into a real job and performs a deep merge on maps: nested keys such as variables or artifacts are merged key by key, while arrays such as script are replaced wholesale, not concatenated. That last part is the detail interviewers probe, because people expect script to append. extends can chain through multiple levels and accepts a list to merge several templates, with later entries winning on conflict.

YAML anchors, written with an ampersand to define and an asterisk to reference, are a pure YAML feature evaluated before GitLab sees the file. They work but they cannot cross an include boundary, because each included file is parsed separately, which makes them a poor fit for shared configuration. !reference is GitLab's own tag and is the tool for surgical reuse: it injects a specific key from another job, including a hidden job in an included file, and it can be used inside an array so you can compose a script from several fragments. The practical rule used on most large GitLab estates is extends for whole-job inheritance, !reference for stitching script fragments together, default for image, tags and before_script that apply everywhere, and anchors only in single-file configurations.

.aws-auth:
  before_script:
    - aws sts get-caller-identity

.node-base:
  image: node:22-alpine
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
  variables:
    NPM_CONFIG_CACHE: .npm

test:
  extends: .node-base
  variables:
    NODE_ENV: test        # merged with NPM_CONFIG_CACHE, not replaced
  script:
    - !reference [.aws-auth, before_script]
    - npm ci
    - npm test

Answer

include has five forms. include:local pulls another YAML file from the same repository and the same commit, which is how monorepos split configuration per service. include:project pulls a file from another project on the same GitLab instance, with a ref that can be a branch, tag or commit SHA, and file that accepts a list. include:remote fetches a file over HTTPS from any public URL, evaluated at pipeline creation time. include:template pulls one of the templates that ship with GitLab, such as the SAST or Dependency Scanning job definitions. include:component references a versioned CI/CD component from the CI/CD Catalog and is the modern replacement for shared templates. All included files are merged into one configuration before the pipeline is created, and the Pipeline Editor's merged view is the only sane way to debug the result. The production gotcha is versioning. include:project with ref set to main means your pipeline changes the moment someone merges into a platform repository you do not control, which has caused real outages. Pin to a tag or a SHA, or use a component with a semantic version. include:remote is worse, since it is an unauthenticated network fetch that can fail or be tampered with, and it silently fails the pipeline creation if the endpoint is down. include also supports rules, so you can conditionally pull in a heavy security template only on the default branch.

include:
  - local: '/ci/jobs/build.yml'
  - project: 'platform/ci-templates'
    ref: 'v2.4.1'                 # pin to a tag, never to main
    file:
      - '/templates/docker.yml'
      - '/templates/deploy.yml'
  - template: 'Jobs/Secret-Detection.gitlab-ci.yml'
  - component: '$CI_SERVER_FQDN/platform/components/terraform@1.3.0'
    inputs:
      stage: validate
  - local: '/ci/jobs/nightly.yml'
    rules:
      - if: $CI_PIPELINE_SOURCE == "schedule"

Answer

environment registers the job as a deployment. GitLab then tracks, per environment, which commit is currently deployed, the full deployment history, who triggered each deployment, and it renders a View app button pointing at the url you supply. That record drives several other features: the merge request page shows which environments contain the change, the Deployments page and the Environments page become an audit trail, and the metric of deployment frequency for DORA reporting is derived from it.

Environments can be static, such as staging and production, or dynamic, where the name embeds a variable such as the branch slug, which is how review apps work. deployment_tier lets you classify an environment as production, staging, testing, development or other, which matters because DORA metrics and some compliance dashboards only count production tier deployments. Protected environments, available on paid tiers, restrict who can run a deployment job to a named list of users or groups, and this is enforced independently of branch protection, so a Developer who can push to main may still be unable to deploy to production. environment:action controls the semantics: start is the default, stop marks the environment as stopped, prepare records a deployment without changing the current version, and verify and access exist for read-only interactions. Interviewers often follow up by asking how you stop a review app, which leads into on_stop and auto_stop_in.

deploy-staging:
  stage: deploy
  script:
    - helm upgrade --install api ./chart --namespace staging
  environment:
    name: staging
    url: https://staging.example.com
    deployment_tier: staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  needs: [deploy-staging]
  script:
    - helm upgrade --install api ./chart --namespace prod
  environment:
    name: production
    url: https://app.example.com
    deployment_tier: production
  when: manual

Answer

The CODEOWNERS file, placed at the repository root, in .gitlab/ or in docs/, maps path patterns to users, groups or subgroups who own that code. GitLab uses gitignore-style patterns, so a trailing slash matches a directory and a leading slash anchors to the root. When a merge request touches a matching path, the owners are automatically added as eligible approvers and shown in the merge request approvals section.

On paid tiers you can make Code Owner approval required by enabling that setting on the protected branch, at which point the merge is blocked until an owner approves. GitLab's CODEOWNERS supports named sections written in square brackets, and a section marked as optional with a caret prefix is advisory only. You can also require more than one approval from a section by appending a number to the section heading.

Separately, approval rules configured under Settings, Merge requests define named rules with a required approval count and an eligible approver list, and they can be scoped so a rule only applies when specific files change. Two behaviours are frequently tested. Approvals can be configured to reset when new commits are pushed, which prevents approval then sneak-in-a-change. And the author of a merge request cannot approve their own merge request if the prevent-author-approval setting is enabled, which most regulated Indian banking clients require.

# CODEOWNERS
* @platform-team

[Backend][2]
/services/api/            @backend-leads @alice
/services/payments/       @payments-guild

[Infrastructure]
/terraform/               @sre-team
/.gitlab-ci.yml           @sre-team

^[Docs]
/docs/                    @tech-writers

# Section [Backend] requires 2 approvals.
# Section ^[Docs] is optional and never blocks the merge.

Answer

Cache design is about hit rate. A single global key means every job overwrites the same archive and jobs with different dependency sets keep invalidating each other. The right unit is the lockfile. cache:key:files takes up to two files and computes a SHA of their contents, so the key changes only when dependencies actually change, and you get a clean miss on a dependency bump and a hit on every other commit. cache:key:prefix lets you namespace that per job or per architecture.

In a monorepo, give each service its own cache entry with its own lockfile and its own paths, rather than one enormous cache containing every node_modules in the tree. Use policy deliberately: a job that only consumes dependencies should set policy to pull so it does not spend time re-uploading an unchanged archive, while a single designated job sets pull-push and is the only writer. fallback_keys, available in recent versions, lets a miss fall back to a previous key so a lockfile bump does not start from an empty cache, and the CACHE_FALLBACK_KEY variable does the same globally. Two operational points matter more than the YAML.

First, cache is per runner unless you configure distributed cache in runners.cache pointing at S3, GCS or Azure Blob; on autoscaled fleets local cache is close to useless. Second, measure. The job log prints whether the cache was restored and how long the archive took, and a cache that takes ninety seconds to download to save sixty seconds of npm install is a net loss.

.node-cache:
  cache:
    - key:
        prefix: 'npm-$CI_JOB_IMAGE'
        files:
          - services/api/package-lock.json
      paths:
        - services/api/node_modules/
      policy: pull
      fallback_keys:
        - 'npm-default'

warm-cache:
  extends: .node-cache
  cache:
    - key:
        prefix: 'npm-$CI_JOB_IMAGE'
        files: [services/api/package-lock.json]
      paths: [services/api/node_modules/]
      policy: pull-push
  script:
    - npm ci --prefix services/api

Answer

rules:changes asks GitLab which files changed and runs the job only if a listed path matched. The subtlety is what it compares against, and that depends on pipeline type. On a merge request pipeline, GitLab compares the source branch against the merge base with the target branch, which is what you want.

On a branch pipeline, it compares against the previous commit on that branch, and on the very first pipeline for a new branch, or on a scheduled or tag pipeline, there is no meaningful previous commit, so GitLab treats the rule as always true and the job runs. That is why a supposedly path-filtered deploy job fires on a scheduled nightly pipeline and deploys something nobody changed. There are three defences.

Use compare_to to name an explicit base such as the default branch, which makes the comparison deterministic on branch pipelines. Guard the rule with a pipeline source condition so changes filtering only applies where it is meaningful. And for genuinely complex monorepo routing, stop trying to express it in rules and generate a child pipeline from a script that computes the changed set itself with git diff. A related keyword is rules:exists, which checks for the presence of a file matching a glob in the repository rather than a change to it, useful for running a Dockerfile lint job only in directories that actually contain a Dockerfile.

deploy-payments:
  stage: deploy
  script: ./scripts/deploy.sh payments
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        compare_to: 'refs/heads/main'
        paths:
          - 'services/payments/**/*'
          - 'libs/ledger/**/*'

dockerfile-lint:
  script: hadolint services/*/Dockerfile
  rules:
    - exists:
        - 'services/*/Dockerfile'

Answer

A parent-child pipeline splits one logical pipeline into a parent that decides what to run and one or more children that do the work. You create a child with a trigger job pointing at another YAML file via trigger:include. The child runs as its own pipeline with its own graph, its own stages and its own concurrency, which keeps the parent graph readable and lets each team own its own file.

It is the standard answer for monorepos: the parent has one trigger job per service, each guarded by rules:changes, and only the affected services produce a child pipeline. By default a parent does not wait for its children and the child's status does not affect the parent; set strategy to depend so the trigger job mirrors the child's status and the parent fails when the child fails. The more powerful variant is dynamic child pipelines.

A job runs a script that inspects the repository, computes what needs building, writes a YAML file, and publishes it as an artifact; a second job then uses trigger:include with artifact and job to run that generated file. This is how large estates handle hundreds of services or matrix combinations that cannot be expressed statically. Two constraints to remember: a child pipeline cannot itself trigger a grandchild beyond the documented nesting limit, and variables do not automatically flow both ways, the parent's variables are inherited by the child but the child cannot push variables back up.

generate-config:
  stage: build
  image: python:3.12-slim
  script:
    - python ci/generate_pipeline.py > generated-ci.yml
  artifacts:
    paths: [generated-ci.yml]

run-children:
  stage: test
  needs: [generate-config]
  trigger:
    include:
      - artifact: generated-ci.yml
        job: generate-config
    strategy: depend

payments-child:
  stage: test
  trigger:
    include: 'services/payments/.gitlab-ci.yml'
    strategy: depend
  rules:
    - changes:
        paths: ['services/payments/**/*']
        compare_to: 'refs/heads/main'

Answer

A multi-project pipeline is triggered with trigger:project, which starts a pipeline in a different project on the same instance. Unlike a child pipeline it is a fully independent pipeline owned by the downstream project, running that project's own .gitlab-ci.yml at the branch given by trigger:branch. Add strategy: depend if the upstream job should wait and reflect the downstream result, otherwise the trigger job succeeds the instant the downstream pipeline is created.

Variables declared on the trigger job are passed as pipeline variables to the downstream pipeline, which means they sit at the top of the precedence order and override the downstream project's own definitions; that is powerful and occasionally dangerous. To send a specific set explicitly you can also use the pipeline trigger API with a trigger token. Downstream jobs can read CI_PIPELINE_SOURCE with the value pipeline to know they were triggered this way, and upstream identifiers are available through variables such as the upstream project path.

Artifacts move in the other direction: a downstream job can pull artifacts from a specific job in another project using needs:project with a ref and a job name, authenticated with CI_JOB_TOKEN. That token is the security boundary, and in recent GitLab versions the job token allowlist is enforced by default, so the downstream project must explicitly authorise the upstream project under Settings, CI/CD, Job token permissions. A large share of cross-project pipeline failures after a GitLab upgrade come from exactly that allowlist.

trigger-deploy-service:
  stage: deploy
  variables:
    UPSTREAM_IMAGE: $CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA
    ENVIRONMENT: staging
  trigger:
    project: 'infra/deployments'
    branch: main
    strategy: depend

# In the downstream project:
fetch-upstream-artifact:
  stage: build
  needs:
    - project: 'apps/api'
      job: build-binary
      ref: main
      artifacts: true
  script:
    - ls -la bin/

Answer

There are two related features. parallel set to an integer clones the job that many times and gives each clone CI_NODE_INDEX, from one up to CI_NODE_TOTAL. Your test runner is responsible for sharding: Playwright takes a shard argument, Jest takes shard, RSpec is usually split with knapsack or a hand-rolled splitter based on previous timings. Simply setting parallel to ten does nothing on its own if the script runs the whole suite ten times, which is a classic interview trap. parallel:matrix is different: it expands the job across the cartesian product of the variable lists you provide, creating one job per combination, each with those variables set.

That is the tool for cross-version and cross-architecture builds, for example three Python versions times two database backends. Each matrix entry is a separate list item, so you can also enumerate specific combinations instead of a full product, and you can list several values under one key inside a single entry to expand only that dimension. Job names in the UI include the variable values, which makes failures easy to locate.

Limits matter in practice: there is a documented maximum on the number of jobs a single matrix can generate, in the low hundreds, and every generated job consumes a runner slot, so a careless matrix can starve the rest of the pipeline. Combine matrix with needs to keep downstream jobs waiting only on the specific combination they consume.

e2e:
  stage: test
  parallel: 6
  script:
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    when: always
    reports:
      junit: results/junit.xml

build-matrix:
  stage: build
  image: python:$PY_VERSION
  parallel:
    matrix:
      - PY_VERSION: ['3.11', '3.12', '3.13']
        DB: ['postgres', 'mysql']
      - PY_VERSION: ['3.13']
        DB: ['sqlite']
  script:
    - pip install -r requirements.txt
    - pytest -k "$DB"

Answer

workflow:rules is evaluated once, before any job rules, and decides whether the pipeline is created at all. If no rule matches and there is no catch-all, no pipeline is created and the push shows no pipeline rather than a failed one. The classic problem it solves is double pipelines.

With default settings, a push to a branch that has an open merge request produces both a branch pipeline and a merge request pipeline, so runner minutes double and the merge request widget shows two results. The canonical fix is an ordered set of workflow rules: accept merge request pipelines, then explicitly refuse branch pipelines when CI_OPEN_MERGE_REQUESTS is set, then allow the default branch and tags. GitLab also ships a ready-made template for this under Workflows. workflow:rules can set variables globally, which is a neat way to compute a deployment target or an image tag once instead of repeating a rules block in every job. workflow:name, available in recent versions, renames the pipeline in the UI, which is genuinely useful when scheduled, tag and merge request pipelines all look alike in a busy list.

Two cautions. workflow rules apply to child pipelines too unless the child file has its own workflow block, which catches people whose children mysteriously fail to start. And auto-cancelling behaviour is separate: it is controlled by the interruptible keyword and the project setting for auto-cancelling redundant pipelines, not by workflow.

workflow:
  name: '$PIPELINE_LABEL'
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      variables:
        PIPELINE_LABEL: 'MR !$CI_MERGE_REQUEST_IID'
        DEPLOY_TARGET: 'review'
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      variables:
        PIPELINE_LABEL: 'main $CI_COMMIT_SHORT_SHA'
        DEPLOY_TARGET: 'staging'
    - if: $CI_COMMIT_TAG
      variables:
        PIPELINE_LABEL: 'release $CI_COMMIT_TAG'
        DEPLOY_TARGET: 'production'

Answer

Environment variables set in a job's shell die with the job, because each job runs in a fresh container. The supported handoff mechanism is artifacts:reports:dotenv. A job writes KEY=value lines into a file and declares that file as a dotenv report; GitLab parses it and injects those keys as environment variables into downstream jobs, specifically jobs in later stages, or the jobs that list the producing job in needs.

This is how you compute a semantic version once, or capture the URL of a freshly provisioned review environment, or record a Terraform output, and use it everywhere later. Rules worth knowing: the file must contain simple KEY=value pairs, values spanning multiple lines are not supported without quoting, there are limits on the number of variables and on file size, and variables inherited this way sit below manually supplied pipeline variables in the precedence order, so a UI override still wins. Inheritance is controllable with the dependencies keyword and with inherit:variables, so a job can opt out of receiving upstream dotenv variables entirely.

For cross-pipeline flow, a child or downstream pipeline can also consume dotenv variables from an upstream job when the trigger is configured with needs referencing that job. A frequent mistake is expecting a dotenv value to be available inside the producing job's own after_script, which it is not, because the report is processed after the job completes.

compute-version:
  stage: prepare
  script:
    - VERSION=$(git describe --tags --abbrev=0)-$CI_PIPELINE_IID
    - echo "APP_VERSION=$VERSION" >> build.env
    - echo "IMAGE_TAG=$CI_REGISTRY_IMAGE/api:$VERSION" >> build.env
  artifacts:
    reports:
      dotenv: build.env

build:
  stage: build
  needs: [compute-version]
  script:
    - docker build -t "$IMAGE_TAG" --build-arg VERSION="$APP_VERSION" .
    - docker push "$IMAGE_TAG"

deploy:
  stage: deploy
  needs: [compute-version, build]
  script:
    - helm upgrade --install api ./chart --set image.tag="$APP_VERSION"

Answer

Docker-in-Docker means running the docker:dind service alongside your job so the docker CLI has a daemon to talk to. It works and it is what most tutorials show, but it requires the runner to be configured with privileged set to true, which effectively gives every job on that runner root on the host. On a shared runner serving multiple teams that is an unacceptable blast radius, and it is the first thing a security-minded interviewer will push on.

Two configuration details always come up. Modern Docker images enable TLS by default, so you must either set DOCKER_TLS_CERTDIR to the shared certificate directory and point DOCKER_HOST at the TLS port, or explicitly disable TLS by setting DOCKER_TLS_CERTDIR to an empty string, which is only acceptable on an isolated runner. And dind starts with an empty layer cache on every job, so builds are slow unless you pull a previous image and use cache-from or BuildKit inline cache.

The alternative is a daemonless builder. Kaniko and Buildah build images from a Dockerfile inside an ordinary unprivileged container and push directly to a registry, which removes the privileged requirement entirely and is the standard choice on Kubernetes runners. Kaniko authenticates by writing a Docker config JSON containing the CI_JOB_TOKEN into the container. The trade-off is that some Dockerfile features and cache behaviours differ from the Docker daemon, so complex multi-stage builds occasionally need adjustment.

# Daemonless build with Kaniko, no privileged runner required
build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: ['']
  script:
    - mkdir -p /kaniko/.docker
    - |
      echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
        --context "$CI_PROJECT_DIR"
        --dockerfile "$CI_PROJECT_DIR/Dockerfile"
        --destination "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA"
        --cache=true
        --cache-repo "$CI_REGISTRY_IMAGE/cache"

# Docker-in-Docker variant, needs privileged = true on the runner
build-dind:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_HOST: 'tcp://docker:2376'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_TLS_VERIFY: '1'
  script:
    - docker build -t "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA" .

Answer

Inside a job, GitLab injects CI_REGISTRY for the registry host, CI_REGISTRY_IMAGE for the image path matching the project, and CI_REGISTRY_USER with CI_REGISTRY_PASSWORD, which is a short-lived credential derived from CI_JOB_TOKEN and valid only for the life of the job. Log in by piping the password to docker login with password-stdin, never as a command-line argument, because arguments land in the job log and in the process table. For pushing to another project's registry, CI_JOB_TOKEN gives read access to projects that have allowlisted yours, and for anything broader you need a deploy token or a project access token with the registry scopes.

Storage is the part people forget. Every pipeline pushing an image tagged with the commit SHA produces a new tag, and untagged layers left behind by overwritten tags accumulate. GitLab provides a cleanup policy per project under Packages and registries, configured with a regex for tags to keep, a regex for tags to remove, a count of most recent tags to always keep, and an age threshold.

It runs on a schedule and only deletes tags, with garbage collection reclaiming the underlying layers afterwards. On self-managed installs the registry garbage collector may need to be run explicitly. There is also protected container tags, so a cleanup policy or a careless delete cannot remove a release tag. A useful pattern is to push both a SHA tag for traceability and a semantic version tag for releases, and to have the cleanup policy keep anything matching the version pattern.

publish:
  stage: publish
  image: docker:27
  services: [docker:27-dind]
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker build
        --cache-from "$CI_REGISTRY_IMAGE/api:cache"
        --build-arg BUILDKIT_INLINE_CACHE=1
        -t "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA"
    - |
      if [ -n "$CI_COMMIT_TAG" ]; then
        docker tag "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA" "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_TAG"
        docker push "$CI_REGISTRY_IMAGE/api:$CI_COMMIT_TAG"
      fi

# Cleanup policy (Settings > Packages and registries):
#   keep the 10 most recent tags
#   keep tags matching ^v\d+\.\d+\.\d+$
#   remove tags matching .* older than 30 days

Answer

retry re-runs a failed job automatically, and crucially it can be scoped by failure reason with retry:when, accepting values such as runner_system_failure, stuck_or_timeout_failure, api_failure, scheduler_failure and runner_unsupported, plus script_failure. Retrying on infrastructure reasons is good hygiene. Retrying blindly on script_failure is usually a mistake, because it hides flaky tests, doubles the time to feedback and can double-apply a non-idempotent deployment.

Recent versions also support retry:exit_codes so you can retry only on a specific exit code, for example a known transient network error from a package mirror. timeout sets a per-job limit that overrides the project's default; the effective limit is also capped by the runner's own timeout setting, so a job asking for four hours on a runner configured with a one-hour maximum will still be killed at one hour, and the resulting failure message confuses people. interruptible marks a job as safe to cancel; combined with the project setting for auto-cancelling redundant pipelines, pushing a new commit cancels the older pipeline's still-running interruptible jobs, which is the single largest runner cost saving on busy repositories. The rule is simple: tests and builds are interruptible, deployments and anything that mutates external state are not. Once a non-interruptible job has started, GitLab will not auto-cancel the rest of that pipeline, which is a deliberate safety property people often mistake for a bug.

default:
  interruptible: true
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - api_failure

integration-tests:
  stage: test
  timeout: 25 minutes
  script: ./scripts/integration.sh

deploy-production:
  stage: deploy
  interruptible: false      # never auto-cancel a live deployment
  timeout: 1 hour
  retry: 0
  environment: production
  script: ./scripts/deploy.sh

Answer

resource_group serialises jobs so that only one job holding a given resource key runs at a time across the whole project. Without it, two merge requests merged a minute apart both start a deploy job, and two Terraform applies or two Helm upgrades race against the same state, which produces a locked state file at best and an inconsistent environment at worst. Adding resource_group with a name such as production makes GitLab queue the second job until the first has finished, without failing it.

The name can include variables, so resource_group set to the environment name gives you one lock per environment automatically, and in a review-app setup one lock per branch. Process mode controls queue behaviour and is configured through the API rather than YAML. unordered is the default and gives no ordering guarantee. oldest_first processes the queue in pipeline order, which is what you want for a sequential release train. newest_first runs the most recent job and cancels or skips the older queued ones, which suits a continuously deploying main branch where deploying an older commit after a newer one would be a regression. The important distinction interviewers look for is between resource_group and environment protection: resource_group is about concurrency, protected environments are about authorisation, and needs is about ordering within a single pipeline. They solve three different problems and a candidate who conflates them gets marked down.

terraform-apply:
  stage: deploy
  resource_group: 'tfstate-$TF_WORKSPACE'
  variables:
    TF_WORKSPACE: production
  script:
    - terraform init -backend-config=backend.hcl
    - terraform apply -auto-approve -input=false plan.tfplan
  environment:
    name: production

deploy-review:
  stage: deploy
  resource_group: $CI_ENVIRONMENT_SLUG   # one lock per review app
  environment:
    name: review/$CI_COMMIT_REF_SLUG
  script: ./scripts/deploy-review.sh

Answer

A review app is a full ephemeral copy of the application deployed per merge request so reviewers can click through the change. In GitLab this is a dynamic environment: the environment name embeds CI_COMMIT_REF_SLUG or the merge request IID, so each branch gets its own environment record and its own URL. GitLab exposes CI_ENVIRONMENT_SLUG, a sanitised and truncated form safe for DNS names and Kubernetes namespaces, and CI_ENVIRONMENT_URL, which is populated from the url field.

Teardown is the part that separates a working setup from a cloud bill. environment:on_stop names another job in the same pipeline that GitLab runs when the environment is stopped, and that job must carry environment with the same name plus action set to stop, and it must be when: manual so it is created but not run immediately. GitLab then triggers it automatically when the branch is deleted or the merge request is merged or closed, provided the stop job still exists in the pipeline. environment:auto_stop_in adds a time-based expiry such as three days, after which GitLab stops the environment on its own, which is the safety net for abandoned branches. Two common failures: the stop job cannot fetch artifacts it needs because the source branch is gone, so give it GIT_STRATEGY set to none and make it self-contained, and the stop job must not be filtered out by rules on the branch pipeline or GitLab will have nothing to run.

deploy-review:
  stage: deploy
  script:
    - helm upgrade --install "$CI_ENVIRONMENT_SLUG" ./chart
        --namespace "review-$CI_ENVIRONMENT_SLUG" --create-namespace
        --set ingress.host="$CI_ENVIRONMENT_SLUG.review.example.com"
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_ENVIRONMENT_SLUG.review.example.com
    on_stop: stop-review
    auto_stop_in: 3 days
    deployment_tier: development
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

stop-review:
  stage: deploy
  variables:
    GIT_STRATEGY: none
  script:
    - helm uninstall "$CI_ENVIRONMENT_SLUG" --namespace "review-$CI_ENVIRONMENT_SLUG"
    - kubectl delete namespace "review-$CI_ENVIRONMENT_SLUG" --ignore-not-found
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  when: manual
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual

Answer

Storing an AWS access key as a CI variable means a permanent credential sits in project settings, is copied into every deployment job, and has to be rotated by a human. OIDC removes it. GitLab can mint a short-lived, signed JSON Web Token per job using the id_tokens keyword, with an aud claim you choose.

You register GitLab as an OIDC identity provider in the cloud account once, create a role with a trust policy that accepts tokens from your GitLab instance, and constrain the trust to specific claims: the sub claim encodes the project path, the ref and the ref type, so you can allow only the main branch of one project to assume the production role. The job then exchanges the token for temporary credentials, with the AWS CLI that is sts assume-role-with-web-identity or simply setting AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN so the SDK does it automatically. The same pattern works for GCP workload identity federation, Azure workload identity, HashiCorp Vault's JWT auth method and Kubernetes.

This replaced the older CI_JOB_JWT and CI_JOB_JWT_V2 variables, which were deprecated and then removed, so if you see them in an existing pipeline that is a migration task. The security gain is real: tokens expire in minutes, they are scoped per job, and a leaked job log gives an attacker nothing usable. Interviewers at security-conscious organisations treat knowledge of the sub claim structure as the signal that you have actually implemented this rather than read about it.

deploy-aws:
  stage: deploy
  image: amazon/aws-cli:latest
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/gitlab-deploy-prod
    AWS_DEFAULT_REGION: ap-south-1
  script:
    - echo "$AWS_ID_TOKEN" > /tmp/web-identity-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity-token
    - aws sts get-caller-identity
    - aws s3 sync ./dist s3://prod-assets-bucket --delete
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# IAM trust condition, restrict by the sub claim:
#   gitlab.example.com:sub = 'project_path:apps/api:ref_type:branch:ref:main'

Answer

The secrets keyword tells the runner to fetch a secret from an external manager and expose it to the job, rather than storing the value in GitLab. The Vault integration works through Vault's JWT auth method: you declare an id_token with an aud matching what Vault expects, Vault validates the signature against GitLab's JWKS endpoint, checks bound claims such as the project path and the ref, and issues a short-lived Vault token bound to a role and policy. The runner then reads the requested path and either sets the value as an environment variable or, with file set to true, writes it to a temporary file and gives you the path.

Vault server address comes from the VAULT_SERVER_URL variable or the vault:server configuration, and the engine block lets you specify the KV version and mount path, which matters because kv-v2 stores data under a data sub-path that trips up people migrating from kv-v1. Recent GitLab versions extended the same keyword to other providers including Azure Key Vault, Google Secret Manager, Akeyless and AWS Secrets Manager, so the pattern generalises. The advantages over CI variables are meaningful: secrets are centrally rotated, access is auditable in the secret manager's own log, and revocation is immediate rather than requiring you to find every project that copied the value. The trade-off is a hard dependency on the secret manager being reachable from the runner network, so plan for what a Vault outage does to your deployment path.

deploy:
  stage: deploy
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.com
  variables:
    VAULT_SERVER_URL: https://vault.example.com
    VAULT_AUTH_ROLE: gitlab-prod-deploy
  secrets:
    DB_PASSWORD:
      vault:
        engine:
          name: kv-v2
          path: kv
        path: apps/api/production
        field: db_password
      token: $VAULT_ID_TOKEN
    TLS_KEY:
      vault: apps/api/production/tls@kv
      file: true
      token: $VAULT_ID_TOKEN
  script:
    - psql "postgres://api:$DB_PASSWORD@db.internal/app" -c 'select 1'
    - nginx -c /etc/nginx/nginx.conf -g "ssl_certificate_key $TLS_KEY;"

Answer

GitLab ships job templates that you pull in with include:template. SAST is a dispatcher: it detects the languages in the repository and enables the matching analyzer jobs, historically a set of language-specific containers wrapping tools such as Semgrep, and in recent versions consolidated heavily onto a Semgrep-based analyzer for most languages. Secret Detection scans commits for credential patterns and, on the default branch, can scan the full history.

Dependency Scanning parses lockfiles to build a dependency graph and matches it against the advisory database, and newer versions produce a CycloneDX software bill of materials alongside the findings. Container Scanning inspects a built image, DAST exercises a running application, and IaC scanning covers Terraform and Kubernetes manifests. All of them write artifacts:reports entries in a standard security report format, which is how findings reach the merge request widget and, on Ultimate, the vulnerability report and security dashboard.

On the free tier you still get the merge request widget for some scanners and the JSON artifact for all of them, so you can gate on results with your own script. Tuning is done with variables rather than by editing the template: SAST_EXCLUDED_PATHS to skip vendored code, SAST_EXCLUDED_ANALYZERS to disable noisy ones, SECURE_LOG_LEVEL for debugging, and DS_EXCLUDED_PATHS for dependency scanning. Override a template job by redefining a job with the same name after the include, which merges rather than replaces. The common production complaint is pipeline time, so most teams run the heavy scanners only on the default branch and on a nightly schedule.

include:
  - template: 'Jobs/SAST.gitlab-ci.yml'
  - template: 'Jobs/Secret-Detection.gitlab-ci.yml'
  - template: 'Jobs/Dependency-Scanning.gitlab-ci.yml'
  - template: 'Jobs/Container-Scanning.gitlab-ci.yml'

variables:
  SAST_EXCLUDED_PATHS: 'spec, test, tests, tmp, vendor, node_modules'
  SECRET_DETECTION_HISTORIC_SCAN: 'false'
  DS_EXCLUDED_PATHS: 'qa, fixtures'
  CS_IMAGE: '$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA'

# Override a template job: same name, merged on top of the template
semgrep-sast:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  allow_failure: false

Answer

By default, a job downloads the artifacts of every job in all preceding stages. On a wide pipeline with large build outputs that becomes minutes of pure network time per job, repeated across dozens of jobs. There are two keywords to control it. dependencies is the older mechanism: give it an explicit list of job names and the job downloads artifacts from only those jobs, and give it an empty list to download nothing at all.

It does not change ordering, only artifact transfer. needs both changes ordering and scopes artifact download, and each entry can carry artifacts set to false so you can depend on a job for timing without paying for its output. The rule interviewers check is what happens when you combine them: if a job has both, dependencies must be a subset of the jobs listed in needs, otherwise the configuration is invalid. There is also a correctness trap.

If a job listed in dependencies did not run, because rules filtered it out, the depending job fails with a message about a missing dependency unless the artifact was optional. On a large pipeline the fastest cheap win is to add an empty dependencies list to every job that does not consume artifacts, typically linters, security scanners and notification jobs. Combine that with expire_in and with artifacts:exclude to strip files you never need, such as source maps or intermediate object files, and job start times drop noticeably.

build:
  stage: build
  script: make all
  artifacts:
    paths: [dist/, build/]
    exclude:
      - 'build/**/*.o'
      - 'dist/**/*.map'
    expire_in: 3 days

lint:
  stage: test
  dependencies: []        # downloads nothing, starts fast
  script: ./scripts/lint.sh

package:
  stage: package
  needs:
    - job: build
      artifacts: true
    - job: lint
      artifacts: false    # ordering only, no download
  script: ./scripts/package.sh dist/

Answer

Before your script runs, the runner prepares a working copy, and on a repository with years of history that step can dominate a short job. GIT_STRATEGY controls it. clone removes the working directory and clones fresh, which is the safest and slowest. fetch reuses the existing checkout on that runner and fetches only new objects, which is dramatically faster on persistent runners and is the default for the docker executor when a cached volume exists. none skips fetching entirely and expects the working directory to already contain what you need, which is exactly right for a deployment job that only consumes artifacts, or for an environment stop job whose branch no longer exists. empty, available in recent versions, clears the directory without fetching. GIT_DEPTH sets a shallow clone depth; the default on GitLab.com is a small number such as twenty, and a shallow clone breaks anything that walks history, so a job running git describe for versioning, a full-history secret scan, or SonarQube blame analysis must set GIT_DEPTH to zero.

GIT_SUBMODULE_STRATEGY takes none, normal or recursive, and pairs with GIT_SUBMODULE_DEPTH and GIT_SUBMODULE_PATHS to fetch only the submodules a job needs; submodule URLs should be relative in .gitmodules so CI_JOB_TOKEN authentication works without extra credentials. GIT_CLEAN_FLAGS controls the git clean invocation between jobs, and setting it to none is a known way to keep an expensive build directory around at the cost of correctness.

variables:
  GIT_STRATEGY: fetch
  GIT_DEPTH: '20'
  GIT_SUBMODULE_STRATEGY: none

version:
  stage: prepare
  variables:
    GIT_DEPTH: '0'          # full history for git describe
  script:
    - git describe --tags --long

build-with-submodules:
  variables:
    GIT_SUBMODULE_STRATEGY: recursive
    GIT_SUBMODULE_DEPTH: '1'
    GIT_SUBMODULE_PATHS: 'vendor/protos vendor/sdk'
  script: make build

deploy:
  variables:
    GIT_STRATEGY: none      # only needs artifacts, never the source
  needs: [build-with-submodules]
  script: ./scripts/deploy.sh

Answer

The release keyword creates a GitLab Release object attached to a tag, with a name, description, and optional asset links pointing at artifacts, packages or external URLs. Historically the job had to run an image containing release-cli, because the keyword is implemented by that tool, and the canonical image is the release-cli image published by GitLab; newer versions have been moving toward the glab CLI, so check what your instance expects. The job must run on a tag pipeline, since a release is tied to a tag, which in practice means a rule on CI_COMMIT_TAG.

You can also have the job create the tag itself with the tag_name and ref fields, which is how teams cut releases from a manual pipeline on the default branch. Release descriptions are commonly generated rather than hand-written: GitLab's changelog API builds notes from commits following Conventional Commits trailers, and a release job can call that endpoint or read a CHANGELOG file produced earlier in the pipeline. Assets are links, not uploads, so the usual pattern is to publish a binary to the Package Registry generic packages endpoint, or push an image to the Container Registry, then reference that URL in assets:links.

Milestones can be attached so the release page lists closed issues. For Indian teams shipping to enterprise clients, the release object plus the attached SBOM and signed artifacts is often the audit evidence, so this question tends to expand into supply-chain territory.

upload-binary:
  stage: package
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - |
      curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file dist/api \
        "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/api/$CI_COMMIT_TAG/api-linux-amd64"

create-release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  needs: [upload-binary]
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - echo 'creating release'
  release:
    tag_name: $CI_COMMIT_TAG
    name: 'Release $CI_COMMIT_TAG'
    description: './CHANGELOG.md'
    assets:
      links:
        - name: 'api-linux-amd64'
          url: '$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/api/$CI_COMMIT_TAG/api-linux-amd64'
        - name: 'container image'
          url: 'https://$CI_REGISTRY_IMAGE/api:$CI_COMMIT_TAG'

Answer

The old answer was the docker+machine executor driving Docker Machine to create cloud VMs. Docker Machine is unmaintained and GitLab deprecated that executor, so the current architecture is the GitLab Runner Autoscaler, which separates the runner from the instance provider. The docker-autoscaler executor, or the instance executor for jobs that need the whole VM, talks to a fleeting plugin, and there are plugins for AWS Auto Scaling groups, Google Compute Engine and Azure scale sets.

Configuration lives in config.toml: the autoscaler block sets capacity_per_instance, max_use_count so an instance is recycled after N jobs, max_instances, and one or more autoscaling policies with periods and idle_count, so you can hold warm capacity during Indian business hours and scale to zero overnight, which is the single biggest cost lever on a self-managed fleet. The Kubernetes executor is the other mainstream answer: one pod per job, with resource requests and limits, node selectors and tolerations for spot or preemptible pools, and pod cleanup handled by the runner. Its pitfalls are the pod pending timeout when the cluster cannot schedule, ephemeral storage limits when a build writes a large artifact, and the helper image needing registry access in air-gapped environments. Whichever you pick, the operational essentials are the same: cap concurrent globally and per runner, use spot or preemptible capacity for interruptible jobs only, put the cache in object storage in the same region as the runners so transfer is free and fast, and monitor queue wait time rather than just job duration, because queue time is what developers actually feel.

# config.toml, AWS autoscaling with the fleeting plugin
concurrent = 100

[[runners]]
  name = 'autoscaler-apsouth1'
  url = 'https://gitlab.example.com/'
  token = 'glrt-REDACTED'
  executor = 'docker-autoscaler'
  [runners.docker]
    image = 'alpine:3.20'
  [runners.autoscaler]
    plugin = 'aws'
    capacity_per_instance = 2
    max_use_count = 20
    max_instances = 50
    [runners.autoscaler.plugin_config]
      name = 'gitlab-runner-asg'
      profile = 'default'
    [[runners.autoscaler.policy]]
      periods = ['* 9-20 * * mon-sat']
      timezone = 'Asia/Kolkata'
      idle_count = 6
      idle_time = '20m'
    [[runners.autoscaler.policy]]
      periods = ['* * * * *']
      idle_count = 0

Answer

There are three levels. A merge request pipeline tests the source branch alone. A merged results pipeline tests a temporary commit created by merging the source branch into the current target branch, so it catches semantic conflicts between two branches that merge cleanly at the text level.

A merge train goes further: it queues merge requests and builds each one against the target branch plus every merge request ahead of it in the train, so what is tested is exactly the state that will exist after merging. That is the only way to guarantee a green default branch under high merge throughput. The failure modes are what interviewers dig into.

If a merge request in the train fails, it is removed and every merge request behind it is rebuilt, so a single flaky test can invalidate a long queue and burn a large amount of compute. Long pipelines make trains impractical, because the train advances at the speed of the slowest pipeline, so teams that adopt trains usually first split their pipeline with needs and move slow suites to a post-merge or nightly run. Blocking manual jobs stall the train, since the train waits for a pipeline it cannot complete without human input.

Skipped pipelines and pipelines that create no jobs also confuse the train. Practical mitigations: enable auto-cancel on redundant train pipelines, mark train jobs interruptible, keep the merge-train pipeline lean by gating heavy scanners on the default branch instead, and quarantine flaky tests aggressively, because with trains the cost of flakiness is multiplied by the queue length.

# Keep the merge train pipeline lean and predictable
.train-only: &train-only
  if: $CI_MERGE_REQUEST_EVENT_TYPE == "merge_train"

unit:
  stage: test
  interruptible: true
  script: npm test -- --runInBand=false
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

heavy-e2e:
  stage: test
  interruptible: true
  script: npx playwright test
  rules:
    - *train-only            # full suite only inside the train
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

manual-gate:
  stage: deploy
  when: manual
  allow_failure: false
  rules:
    # never inside a train, a blocking manual job stalls the queue
    - if: $CI_MERGE_REQUEST_EVENT_TYPE == "merge_train"
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Answer

A CI/CD component is a reusable, versioned unit of pipeline configuration published from a component project and consumed with include:component. It went generally available during the GitLab 17 series and is the intended successor to the pattern of a central templates repository consumed with include:project. The mechanics: a component project contains a templates directory, each component is either a single YAML file or a directory with a template.yml, and the project is published to the CI/CD Catalog by marking it as a catalog resource and creating a release for a tag.

Consumers then reference it by path and version, and semantic version ranges are supported so you can pin to a major line and still pick up patches. The feature that makes components genuinely better than raw includes is the spec:inputs header. A component declares its inputs with types, defaults, allowed option lists and descriptions, and GitLab validates them at pipeline creation, so a typo in an input name fails fast with a clear message instead of silently producing a broken job.

Inputs are interpolated with the dollar-brace-square syntax at configuration merge time, which is different from CI variables that are expanded at job runtime, and confusing the two is the most common bug when writing a first component. Practically, components let a platform team ship a versioned, tested, documented pipeline library, and let application teams upgrade on their own schedule instead of being broken by a merge into a shared main branch.

# Component project: templates/docker-build/template.yml
spec:
  inputs:
    stage:
      default: build
    image_name:
      description: 'Image path without a tag'
    dockerfile:
      default: Dockerfile
    push:
      type: boolean
      default: true
    runner_tag:
      options: ['docker', 'docker-arm64']
      default: docker
---
docker-build:
  stage: $[[ inputs.stage ]]
  tags: [ '$[[ inputs.runner_tag ]]' ]
  image: gcr.io/kaniko-project/executor:debug
  script:
    - /kaniko/executor --dockerfile '$[[ inputs.dockerfile ]]'
        --destination '$[[ inputs.image_name ]]:$CI_COMMIT_SHORT_SHA'
        --no-push=$[[ inputs.push ]]

# Consumer project: .gitlab-ci.yml
include:
  - component: $CI_SERVER_FQDN/platform/ci-components/docker-build@~latest
    inputs:
      image_name: $CI_REGISTRY_IMAGE/api
      runner_tag: docker-arm64

Answer

Start with data, not guesses. The pipeline detail page and the CI/CD Analytics view show duration trends, and the job list shows per-job duration and queued duration separately. Queued duration rising means a runner capacity problem, not a code problem, and the fix is fleet sizing or tag routing, not YAML.

If job duration rose, open a slow job and read the sections in the log: the runner prints timings for the prepare, get sources, restore cache, download artifacts, script and upload artifacts phases. Very often the regression is in one of the non-script phases. Get sources slow means the repository grew or GIT_STRATEGY changed from fetch to clone; fix with shallow depth and persistent fetch.

Restore cache or upload artifacts slow means the archive grew; check whether someone added a directory to cache paths, enable the fastzip feature flag on the runner, and tune ARTIFACT_COMPRESSION_LEVEL and CACHE_COMPRESSION_LEVEL to fast when the objects are already compressed. Download artifacts slow means the job is pulling artifacts it does not need, so add needs with artifacts false or an empty dependencies list. If the script itself is slow, bisect inside it by timing steps and look for a dependency install that lost its cache hit, usually because a lockfile-derived cache key changed or a fallback key is missing.

Structurally, convert stage ordering to needs so the critical path is real work rather than waiting, split slow suites with parallel, and move heavy scanners off merge request pipelines onto the default branch and a nightly schedule. Only then reach for CI_DEBUG_TRACE, and remember it prints every variable including masked ones, so never enable it on a protected branch pipeline.

# Runner-side speed-ups
variables:
  FF_USE_FASTZIP: 'true'
  ARTIFACT_COMPRESSION_LEVEL: 'fast'
  CACHE_COMPRESSION_LEVEL: 'fast'
  TRANSFER_METER_FREQUENCY: '5s'
  GIT_STRATEGY: fetch
  GIT_DEPTH: '20'

# Fail fast: cheap checks with no dependencies start at t=0
lint:
  stage: test
  needs: []
  dependencies: []
  interruptible: true
  script: ./scripts/lint.sh

# Heavy scanners off the merge request path
dependency-scanning:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# Last resort, never on a protected branch
# variables:
#   CI_DEBUG_TRACE: 'true'

Answer

The naive approach, telling teams to include a template, fails the audit immediately because any developer can delete the include line. GitLab's answer on the Ultimate tier is policy enforcement defined at the group or subgroup level and stored in a dedicated security policy project. Scan execution policies inject specified scanner jobs, such as Secret Detection or Dependency Scanning, into every pipeline in scope on a schedule or on every pipeline, and the injected jobs are not editable from the project's own configuration.

Pipeline execution policies go further and let you enforce arbitrary custom CI jobs, either injected into the developer's pipeline or overriding it entirely, which is how a platform team mandates a licence check or an internal provenance step. Merge request approval policies then block merging when a scan finds a vulnerability above a severity threshold, requiring approval from a named security group rather than the author's own team. This model replaced the older compliance framework pipeline approach, where a compliance framework label pointed at a pipeline configuration file in another project, so if you find that in an existing estate, treat it as a migration item.

Complementary controls belong in the same answer: protected branches with Code Owner approval on the CI configuration path, push rules to reject unsigned commits or secrets in commit messages, protected environments to restrict who deploys, restricting who can edit CI/CD variables, and locking down the CI_JOB_TOKEN allowlist so one project cannot use its job token to read another. For BFSI clients in India this bundle is usually the actual deliverable, so being able to name the pieces matters more than YAML trivia.

# .gitlab/security-policies/policy.yml in the security policy project
scan_execution_policy:
  - name: Mandatory secret and dependency scanning
    description: Runs on every pipeline in the group
    enabled: true
    rules:
      - type: pipeline
        branches: ['*']
    actions:
      - scan: secret_detection
      - scan: dependency_scanning

approval_policy:
  - name: Block criticals
    enabled: true
    rules:
      - type: scan_finding
        branches: ['main']
        scanners: [dependency_scanning, container_scanning]
        severity_levels: [critical, high]
        vulnerability_states: [newly_detected]
        vulnerabilities_allowed: 0
    actions:
      - type: require_approval
        approvals_required: 1
        role_approvers: [maintainer]

Answer

A GitLab instance is a set of cooperating services, and Omnibus hides that until it starts hurting. Puma serves the Rails web and API workload. Workhorse sits in front of Puma and handles large uploads, downloads, Git HTTP traffic and long-polling so Puma workers are not tied up on slow clients.

Sidekiq runs background jobs, everything from pipeline creation and webhook delivery to repository housekeeping, split across queues. Gitaly owns all repository storage and is the only component that touches the disk where repositories live; Praefect adds replication and failover across multiple Gitaly nodes, and recent versions have been developing a Raft-based clustering approach, so state your version when discussing it. PostgreSQL holds the relational data, Redis handles caching, sessions, and the Sidekiq queues, and object storage holds artifacts, LFS, uploads, packages and the container registry.

GitLab publishes reference architectures sized by users, and following one is a better answer than improvising. Under load the first bottleneck is usually Sidekiq queue latency, visible as pipelines that take a minute to appear after a push and webhooks that fire late; the fix is more Sidekiq processes with queue selectors so slow queues cannot starve pipeline creation. The second is Gitaly CPU and disk on a monorepo, fixed by moving hot repositories to their own storage, enabling pack-objects cache, and controlling shallow clone depth in CI.

Puma comes third. Always move artifacts and LFS to object storage rather than local disk, and never run CI runners on the GitLab application nodes, which is the mistake most small self-managed installs make.

# /etc/gitlab/gitlab.rb, a few load-relevant settings
puma['worker_processes'] = 8
puma['per_worker_max_memory_mb'] = 1400

sidekiq['max_concurrency'] = 20
sidekiq['queue_groups'] = [
  'pipeline_default,pipeline_processing',
  'urgent_other,default,mailers'
]

gitaly['configuration'] = {
  pack_objects_cache: { enabled: true }
}

gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['connection'] = {
  provider: 'AWS',
  region: 'ap-south-1'
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'

Answer

The goal is that a change to one service costs one service's worth of pipeline. Layer the solution. First, routing: a parent pipeline whose only job is to decide what changed, using rules:changes with compare_to against the default branch for the simple cases, and a script computing the affected set with git diff plus a dependency graph for the real ones, because path globs cannot express that a change in libs/ledger affects payments and billing but not search.

Tools such as Nx, Turborepo, Bazel or Pants can emit that affected list, and the pipeline just consumes it. Second, generation: have that job write a YAML file with one trigger block per affected service and publish it as an artifact, then run it with trigger:include from artifact, so the graph is exactly as wide as the change. Third, isolation: each service owns its own child configuration file with its own CODEOWNERS entry, so teams change their pipeline without touching a shared file.

Fourth, caching: per-service cache keys derived from that service's lockfile, plus a remote build cache from the build tool itself, which usually beats GitLab cache for compiled languages. Fifth, protect the default branch differently from merge requests: merge requests run the affected subset, the default branch runs a broader set, and a nightly scheduled pipeline runs everything so drift is caught within a day. Sixth, keep git cheap with shallow fetch and sparse checkout where the tooling allows it, because on a large monorepo the checkout itself becomes a measurable share of every job.

detect-affected:
  stage: prepare
  image: node:22-alpine
  variables:
    GIT_DEPTH: '0'
  script:
    - npx nx show projects --affected --base=origin/$CI_DEFAULT_BRANCH --json > affected.json
    - node ci/render-children.js affected.json > children.yml
    - cat children.yml
  artifacts:
    paths: [children.yml]

run-affected:
  stage: build
  needs: [detect-affected]
  trigger:
    include:
      - artifact: children.yml
        job: detect-affected
    strategy: depend

nightly-full:
  stage: build
  trigger:
    include: 'ci/full-matrix.yml'
    strategy: depend
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Answer

The legacy approach was storing a kubeconfig or a service account token as a CI variable and running kubectl from the job. It works, but it means a long-lived cluster-admin credential lives in project settings and the cluster API server must be reachable from the runner network, which is a problem for private clusters. The modern approach is the GitLab agent for Kubernetes: agentk runs inside the cluster and holds an outbound connection to the agent server, KAS, on the GitLab side.

No inbound firewall rule is needed and no cluster credential is stored in GitLab. The agent is configured by a file in the repository under .gitlab/agents, and the ci_access section grants named projects or groups permission to use the agent's tunnel from CI jobs. A job then runs kubectl config use-context with the agent path and talks to the cluster through the tunnel, with the agent enforcing impersonation rules so a CI job can be limited to a specific service account and namespace.

That is push-based CD. The pull-based alternative is GitOps, where a controller in the cluster reconciles from a Git repository. GitLab deprecated the agent's own built-in GitOps feature in favour of Flux, and provides integration so GitLab shows Flux reconciliation status against the environment.

Choose the CI tunnel when you want the pipeline to own ordering, run migrations before a rollout, and report deployment status into environments. Choose Flux when you want drift correction, cluster state that survives GitLab being unavailable, and a strict separation between the repository of record and the pipeline.

# .gitlab/agents/prod-cluster/config.yaml
ci_access:
  projects:
    - id: apps/api
    - id: apps/web
  groups:
    - id: platform

# .gitlab-ci.yml
deploy-k8s:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
    url: https://app.example.com
    deployment_tier: production
  resource_group: production
  script:
    - kubectl config get-contexts
    - kubectl config use-context platform/infra:prod-cluster
    - kubectl -n prod set image deploy/api api=$CI_REGISTRY_IMAGE/api:$CI_COMMIT_SHORT_SHA
    - kubectl -n prod rollout status deploy/api --timeout=180s
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false

Answer

Work backwards from the question an auditor asks: for this artifact running in production, which commit produced it, on which runner, from which pipeline, with which dependencies, and can any of that be forged. GitLab gives you pieces for each. For provenance, the runner can emit an in-toto style attestation for job artifacts when the artifacts metadata generation variable is enabled, recording the job, the pipeline, the commit and the inputs; that file is your SLSA-style provenance and should be attached to the release.

For dependency transparency, Dependency Scanning produces a CycloneDX software bill of materials per project, which you keep as a release asset rather than only as a transient artifact. For integrity of inputs, enforce signed commits with push rules so unsigned or unverified commits are rejected, and use protected tags so a release tag cannot be moved after the fact. For integrity of outputs, sign images with cosign using an OIDC identity from id_tokens rather than a stored key, and enable immutable or protected container tags so a published tag cannot be overwritten.

For the pipeline itself, pin every include to a tag or SHA, pin container images by digest rather than by a floating tag, and lock down the CI_JOB_TOKEN allowlist so a compromised low-trust project cannot reach into a high-trust one. Finally, enforce all of it with a pipeline execution policy at group level rather than trusting each project's YAML, and make the release job the only place with permission to push to the production registry path.

variables:
  RUNNER_GENERATE_ARTIFACTS_METADATA: 'true'

sign-and-attest:
  stage: publish
  image: gcr.io/projectsigstore/cosign:v2.4.1
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
  variables:
    IMAGE: $CI_REGISTRY_IMAGE/api@$IMAGE_DIGEST
  script:
    - cosign sign --yes "$IMAGE"
    - cosign attest --yes --type cyclonedx --predicate gl-sbom.cdx.json "$IMAGE"
  artifacts:
    paths: [gl-sbom.cdx.json]
  rules:
    - if: $CI_COMMIT_TAG

verify-before-deploy:
  stage: deploy
  image: gcr.io/projectsigstore/cosign:v2.4.1
  script:
    - cosign verify --certificate-identity-regexp '.*' --certificate-oidc-issuer "$CI_SERVER_URL" "$IMAGE"