ArgoCD Interview Questions and Answers

Answer

GitOps is an operating model for cloud-native systems where Git is the single source of truth for both application and infrastructure state. The four principles (formalised by the OpenGitOps working group) are: (1) **Declarative**, the desired system state is described declaratively, not via imperative scripts. (2) **Versioned and immutable**, that state is stored in Git, so every change is auditable, reviewable, and revertible. (3) **Pulled automatically**, software agents (like ArgoCD) pull approved changes from Git into the cluster. (4) **Continuously reconciled**, agents continuously observe actual state and correct any drift back to the Git-declared state. ArgoCD is essentially the reference implementation of these four principles for Kubernetes.

Map each principle onto something concrete so the answer doesn't sound theoretical: declarative is the `Application` CRD plus the Helm or Kustomize render behind it; versioned and immutable is `spec.source.targetRevision` pinned to a tag or commit SHA instead of a floating `HEAD`; pulled automatically is the application-controller reconciling every `timeout.reconciliation` (180s by default in `argocd-cm`) or instantly on a Git webhook; continuously reconciled is `syncPolicy.automated.selfHeal: true`, which reverts a `kubectl edit` inside one reconcile cycle. Interviewers usually probe two edges. First, a Jenkins job that checks out Git and runs `kubectl apply` is not GitOps: it satisfies principles 1 and 2 but nothing pulls and nothing reconciles, so drift is invisible until an incident.

Second, secrets break the naive reading of principle 2, so you commit ciphertext (a SealedSecret) or a reference (an ExternalSecret) and keep plaintext in a vault. A strong candidate also names the limits: GitOps describes desired state, not runtime state such as HPA-adjusted replica counts or `status` subresources, which is exactly why `ignoreDifferences` exists.

Answer

ArgoCD is a Kubernetes-native, declarative GitOps continuous delivery tool. A traditional CI/CD pipeline (Jenkins, GitLab CI) pushes changes to the cluster, the pipeline holds kubectl credentials and runs `kubectl apply`. ArgoCD inverts that: the cluster pulls.

You commit a manifest change to Git, ArgoCD's controller (running inside the cluster) notices the diff and applies it. Benefits: no long-lived cluster credentials in your CI system, drift is detected and corrected automatically, rollbacks are a `git revert`, and audit history matches your Git log. ArgoCD does CD only, your CI system still builds images and bumps tags in the GitOps repo.

The full loop in production looks like: CI builds and pushes `payments-api:v1.42.3` to ECR, then opens a commit (or a PR) against the manifests repo changing `image.tag`; ArgoCD's repo-server re-renders, the application-controller diffs, and the new ReplicaSet rolls out. Be precise about the credential inversion, because that is the point interviewers press on: in the push model your CI runner holds a kubeconfig with cluster-edit rights, and a compromised runner owns production. In the pull model ArgoCD holds those rights inside the cluster and the CI runner only needs write access to a Git repo.

Two honest caveats to raise yourself: `git revert` only restores manifests, so a rollback still fails if the previous image tag has been garbage-collected from the registry, and ArgoCD has no notion of build, test or artifact promotion. PreSync and PostSync hooks are Kubernetes Jobs, not a pipeline engine, so multi-stage approval gates still live in your CI tool or in branch protection on the GitOps repo.

Answer

ArgoCD runs as a set of stateless pods inside a Kubernetes cluster (usually a dedicated 'argocd' namespace). The four core components are: (1) **API Server**, gRPC/REST endpoint that powers the UI, CLI, and webhooks; handles auth and RBAC. (2) **Repo Server**, clones Git repos and generates the final Kubernetes manifests (running `helm template`, `kustomize build`, or just reading YAML). It's stateless and horizontally scalable. (3) **Application Controller**, the reconciliation loop.

It compares the desired state (manifests from repo-server) against the live state (from the K8s API) and applies diffs. Sharded by cluster for scale. (4) **Redis**, caches manifests and Application state so the repo-server doesn't re-template on every reconciliation. Optionally, an **ApplicationSet Controller** generates Application CRs from templates, and **Dex** handles SSO.

Two details that separate a reader from an operator. The application-controller ships as a StatefulSet rather than a Deployment precisely because shard identity comes from the ordinal in the pod name, paired with the `ARGOCD_CONTROLLER_REPLICAS` environment variable; get that env var out of sync with the replica count and some clusters simply stop reconciling with no obvious error. And configuration lives in four well-known objects, not in flags you can guess: `argocd-cm` (repos, resource customizations, SSO), `argocd-rbac-cm` (policy CSV), `argocd-cmd-params-cm` (component flags such as `server.insecure` and `application.namespaces`), and the `argocd-secret` Secret (signing key, webhook secrets, local accounts). Know the failure signature of each component: repo-server down gives every Application a `ComparisonError` about failed manifest generation while workloads keep running; controller down means no syncs and no drift correction, again with workloads untouched; api-server down kills the UI, CLI and webhooks but reconciliation continues; and Redis is a disposable cache, so deleting the pod costs one slow re-render, not data.

Answer

The Application is ArgoCD's primary Custom Resource Definition. It tells ArgoCD: 'sync this Git path into this cluster + namespace, with these options'. The spec has three required sections, `source` (where the manifests live), `destination` (which cluster and namespace to deploy into), and `project` (which AppProject governs RBAC and allowed sources/destinations).

It can be created via the UI, CLI (`argocd app create`), or directly as a YAML committed to Git, the last form is the GitOps-native way and enables app-of-apps. Details worth naming: `destination` accepts either `server` (the cluster API URL, `https://kubernetes.default.svc` for the local cluster) or `name` (the friendly name from the cluster Secret), never both, and mixing them across a fleet is a common source of 'cluster not found' errors after a cluster is re-registered. `project` is mandatory; the built-in `default` project permits every repo, destination and resource kind, which is why the first hardening task on any real install is to stop using it. The `status` block is written by the controller, not by you: it carries `sync.revision` (the exact SHA deployed), `health.status`, `operationState` for the last sync, and a `history` list capped by `spec.revisionHistoryLimit` (10 by default) that powers `argocd app rollback`.

A frequent follow-up is what happens when you `kubectl delete application payments-api`. Without the `resources-finalizer.argocd.argoproj.io` finalizer on the metadata, only the Application object disappears and every Deployment, Service and Ingress it created is orphaned in the cluster; with the finalizer, ArgoCD cascades the delete first.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/manifests
    targetRevision: main
    path: services/payments/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Answer

The UI renders each Application as a tree of Kubernetes resources rooted at the top-level objects in the manifest. For each node you see: **health status** (Healthy/Degraded/Progressing/Suspended/Missing), **sync status** (Synced/OutOfSync/Unknown), a live YAML diff between Git and cluster state, recent events, and pod logs. The tree-view is invaluable for debugging, you can click any resource to see exactly which container is failing or which field drifted.

Production teams in India (Razorpay, Swiggy) often grant developers read-only UI access so they can self-serve debugging without cluster credentials. Understand where the tree comes from, because it is not just 'what is in Git'. ArgoCD applies the manifests it renders, then walks `ownerReferences` from those roots, which is why a Deployment you committed expands into a ReplicaSet and Pods that appear nowhere in your repo, and why an operator-created StatefulSet shows up under its CR.

Resources with no owner chain back to a managed root are invisible unless the AppProject enables `orphanedResources` monitoring. Access to the richer panels is gated separately: viewing container logs needs an explicit `logs, get` grant in `argocd-rbac-cm` (recent releases removed the old blanket toggle that let any read-only user tail logs), and the terminal tab needs both the `exec` feature enabled in `argocd-cmd-params-cm` and an `exec, create` policy, which most regulated teams deliberately leave off. The CLI equivalents are `argocd app get payments-api -o tree` for the same hierarchy and `argocd app diff payments-api` for the same diff, both useful when you want the answer inside a script rather than a browser.

Answer

These are two independent dimensions. **Sync status** answers 'does the cluster match Git?' OutOfSync means Git declared something the cluster doesn't have (or vice versa). **Health status** answers 'is the resource actually working?' A Deployment is Healthy when all replicas are ready; Degraded if not.

You can be Synced + Degraded (Git matches cluster, but the pods are crashlooping) or OutOfSync + Healthy (someone made a kubectl change to the live state, Git hasn't caught up but the app still runs fine). The combination determines what action to take: OutOfSync triggers a sync; Degraded triggers an alert. There is a third value on each axis that trips people up.

Sync status `Unknown` almost always means the repo-server could not render manifests at all (bad `targetRevision`, missing Helm dependency, expired repo credentials), so there is nothing to compare against; the Application's `conditions` will carry a `ComparisonError` with the underlying message. Health `Missing` means the object is declared in Git but absent from the cluster, and health `Progressing` is not a timeout-free state: `argocd app wait --health --timeout 600` is how CI decides a rollout actually landed. Also note that `Synced` is a comparison result, not proof the last sync operation succeeded.

The operation's own outcome lives in `status.operationState.phase` (Running, Succeeded, Failed, Error), and a PostSync hook Job can fail while the resources it followed are already Synced and Healthy. ArgoCD Notifications exposes these as separate triggers for exactly this reason: `on-sync-failed` fires on the operation, `on-health-degraded` fires on the health axis, and alerting on only one of them leaves a real class of incidents silent.

Answer

An automated sync policy makes ArgoCD apply Git changes to the cluster without manual approval. Two flags fine-tune the behaviour: **prune: true**, delete cluster resources that have been removed from Git. Without it, ArgoCD never deletes anything, which gradually leaks orphaned resources. **selfHeal: true**, if someone makes a kubectl change to a managed resource, ArgoCD overwrites it back to the Git-declared state.

Without it, ArgoCD only reacts to Git changes, not live-state drift. Most production setups enable both; non-production sometimes leaves `prune` off to avoid surprises. Behaviour details a senior interviewer will dig into.

Self-heal is debounced (`--self-heal-timeout-seconds`, 5s by default) and newer releases back it off exponentially, otherwise ArgoCD and a mutating controller can fight in a tight loop and burn API server quota. Automated sync does not retry a failed sync unless you add a `retry` block with `limit` and `backoff`, so a transient webhook timeout can leave an app OutOfSync until the next Git push. The guard rail everyone should know is `syncPolicy.automated.allowEmpty`, which defaults to false: if a refactor accidentally makes the source path render zero manifests, ArgoCD refuses to prune the whole application rather than deleting production.

It does not protect you from a legitimate render that drops one Deployment, so per-resource opt-outs matter too, via the annotation `argocd.argoproj.io/sync-options: Prune=false` on resources that must never be deleted, `PruneLast=true` to prune only after everything else applies cleanly, and `PrunePropagationPolicy=foreground` so dependent objects go before their owners. Recent releases also support `Prune=confirm`, which pauses and requires a human click in the UI before destructive deletes proceed.

syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground

Answer

Four native types, plus a plugin escape hatch: (1) **Plain YAML/JSON**, ArgoCD just `kubectl applies` the files in the path. (2) **Helm**, ArgoCD detects a `Chart.yaml` and runs `helm template` to generate manifests (it does NOT use `helm install`, Tiller is dead, and Helm releases are not tracked). (3) **Kustomize**, ArgoCD detects `kustomization.yaml` and runs `kustomize build`. (4) **Jsonnet**, supported via the repo-server. For anything else (cdk8s, Pulumi rendered to YAML, custom tooling), you write a **Config Management Plugin (CMP)** that ArgoCD invokes to produce manifests. In production, most teams in India use Helm for off-the-shelf charts and Kustomize for their own services.

The Helm answer has a follow-up almost every time: because ArgoCD templates instead of installing, there is no Helm release Secret in the namespace, so `helm list -n payments` returns nothing and `helm rollback` does not exist. Rollback is `argocd app rollback` or a `git revert`. Helm lifecycle hooks are not silently dropped either, ArgoCD translates `helm.sh/hook: pre-install` and friends into its own PreSync and PostSync phases, and `helm.sh/hook: test` becomes a PostSync Job.

CRDs are included by default but can be suppressed with `helm.skipCrds: true` when a separate Application owns them. On the Kustomize side, remote Helm charts inside a `kustomization.yaml` need `kustomize build --enable-helm`, which ArgoCD only passes if you set `kustomize.buildOptions: --enable-helm` in `argocd-cm`; without it you get 'must specify --enable-helm' in the repo-server logs. Plain directories accept `directory.recurse: true` plus `include`/`exclude` globs. CMP v1 (inline plugins in `argocd-cm`) was deprecated during the 2.x line and removed, so any custom tooling today must run as a sidecar CMP v2 container on the repo-server pod.

Answer

Add a repository credential as a Kubernetes Secret in the argocd namespace with the label `argocd.argoproj.io/secret-type: repository`. Three credential types: SSH (private key), HTTPS (username + token), and GitHub App (installation ID + private key, preferred for GitHub because it has finer-grained permissions and no per-user token rotation). The UI and CLI also let you add repos, but for GitOps purity the secret itself should be managed declaratively (via Sealed Secrets or External Secrets, never plain Git).

At more than a handful of repos, switch to `argocd.argoproj.io/secret-type: repo-creds`, where `url` is a prefix such as `https://github.com/example` and any Application whose `repoURL` starts with it inherits those credentials, so onboarding a new repo needs no new Secret. Scope a credential to a single tenant by setting the `project` key in the Secret, which stops another team's AppProject from borrowing it. The GitHub App form uses `githubAppID`, `githubAppInstallationID` and `githubAppPrivateKey`; SSH uses `sshPrivateKey`; self-hosted GitLab or Bitbucket behind a private CA needs `tlsClientCertData`/`tlsClientCertKey` or, as a last resort, `insecure: "true"`.

Two failure signatures are worth memorising because they show up in every debugging round. `ssh: handshake failed: knownhosts: key is unknown` means the host key is missing from the `argocd-ssh-known-hosts-cm` ConfigMap, not that your key is wrong. `authentication required` on a repo that works locally usually means a fine-grained GitHub token without the Contents: Read scope. Helm OCI registries are a separate entry with `type: helm` and `enableOCI: "true"`.

apiVersion: v1
kind: Secret
metadata:
  name: repo-manifests
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://github.com/example/manifests
  password: ghp_xxxxxxxxxxxx
  username: not-used

Answer

Three ways to interact with ArgoCD: (1) **UI**, best for ad-hoc inspection, troubleshooting, viewing diffs and logs. (2) **argocd CLI**, talks to the API server over gRPC; needed for some operations like rotating SSO tokens and triggering syncs from scripts. (3) **kubectl**, talks directly to the K8s API to manipulate the Application/AppProject CRDs. This is the GitOps-native path: your repo contains Application YAMLs, and CI applies them with kubectl (or ArgoCD itself, in an app-of-apps). Rule of thumb: kubectl for declarative state, argocd CLI for imperative operations, UI for humans.

The security consequence is the part interviewers actually want. ArgoCD's RBAC layer in `argocd-rbac-cm` only governs traffic through the API server, so a user who has `edit` on Application CRs in the `argocd` namespace can create and sync applications while completely bypassing that policy file. AppProject constraints still bite (the controller refuses a destination or repo the project does not allow, and surfaces it as a `ComparisonError`), but 'who may sync prod' is not enforced on the kubectl path.

That is why platform teams keep direct write access to the `argocd` namespace limited to CI and manage everything else through the API server or through Git. Useful CLI specifics: `argocd login --core` skips the API server entirely and drives the CRDs via your kubeconfig, which is handy in a cluster-admin break-glass session; `argocd app diff --local ./overlays/prod` diffs an uncommitted working copy against the live cluster; and `argocd account generate-token --account ci` issues a scoped, non-human token instead of sharing the local `admin` account, which should be disabled once SSO is live.

Answer

Three options: (1) Click 'Sync' in the UI, useful for debugging and one-off operations. (2) `argocd app sync <name>`, same effect via CLI, scriptable. (3) Patch the Application's `operation` field via kubectl. You can also trigger a sync from a Git webhook (configured in the repo settings), ArgoCD will react to pushes within seconds instead of waiting for the default 3-minute polling interval. For production, webhooks are essential: 3-minute lag between merge and deploy feels broken in 2026.

Know the flags, because 'click Sync' is a junior answer. `argocd app sync payments-api --prune --dry-run` shows what a prune would remove without touching the cluster. `--resource apps:Deployment:payments-api` syncs one object out of a large app, which is how you recover a single broken Deployment without re-applying 200 manifests. `--revision v1.42.3` syncs a specific tag or SHA that is behind `targetRevision`, useful for a controlled roll-forward, and it leaves the app OutOfSync afterwards by design. `--local ./overlays/prod` applies your working copy, deliberately breaking GitOps, so it belongs in a dev cluster only. Pair a scripted sync with `argocd app wait payments-api --health --sync --timeout 600` so CI fails loudly instead of returning the instant the operation is accepted. When a sync hangs (a PreSync migration Job that never finishes, a hook waiting on a missing PVC), `argocd app terminate-op payments-api` cancels the in-flight operation; the underlying Job survives, so delete it too or the next sync inherits the same stuck hook.

Answer

ArgoCD polls Git every **3 minutes** by default (`timeout.reconciliation` in the argocd-cm ConfigMap). For production responsiveness, set up a **Git webhook** instead, GitHub, GitLab, and Bitbucket are all supported. With webhooks, ArgoCD reacts to a `git push` in seconds.

You can also lower the polling interval if webhooks aren't possible (e.g. an air-gapped cluster), but going below 1 minute starts hammering your Git provider and the repo-server's clone cache. The mechanics: `argocd-server` exposes `/api/webhook`, and the shared secret goes into the `argocd-secret` Secret under `webhook.github.secret` (or `webhook.gitlab.secret`, `webhook.bitbucketserver.secret`). Receiving a push does not sync anything directly, it invalidates the manifest cache for every Application whose `repoURL` and `targetRevision` match the payload, and those apps then reconcile immediately.

That detail explains the classic monorepo complaint: one commit touching a single service invalidates all 400 Applications pointing at that repo, producing a thundering herd against the repo-server. Mitigations are `spec.source.path`-aware webhook filtering at the provider, more repo-server replicas, and `timeout.reconciliation.jitter` to spread the periodic pass so every app does not wake at the same instant. Also keep polling as a floor rather than turning it off: webhooks are fire-and-forget, and a dropped delivery during a provider incident would otherwise leave the cluster silently behind Git. Air-gapped clusters that cannot receive inbound webhooks are the one case where lowering `timeout.reconciliation` to 60s is the right call.

# argocd-cm ConfigMap
data:
  timeout.reconciliation: 180s
  timeout.reconciliation.jitter: 60s
---
apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
stringData:
  webhook.github.secret: shhh-very-secret
# GitHub webhook URL: https://argocd.example.com/api/webhook
# Content type: application/json, event: push

Answer

Two supported paths. The raw manifest install is `kubectl create namespace argocd` followed by `kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml`. The same repo ships `ha/install.yaml`, which gives you multiple api-server and repo-server replicas plus a Redis HA setup with anti-affinity, and that is what any shared or production instance should use.

The other path is the `argo/argo-cd` Helm chart, preferred when you intend to self-manage ArgoCD with ArgoCD, because the values file is itself a committable artifact. Note that the CRDs are cluster-scoped even though the workloads are namespaced, so two ArgoCD installs on one cluster share `Application` and `AppProject` definitions. For the first login the username is `admin` and the password is autogenerated into a Secret: read it with `kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d`.

Port-forward `svc/argocd-server` and run `argocd login localhost:8080`. Two gotchas to mention: argocd-server terminates TLS itself, so behind an ingress that also terminates TLS you get a redirect loop unless you set `server.insecure: "true"` in `argocd-cmd-params-cm` or configure gRPC passthrough. And `argocd-initial-admin-secret` is deleted the moment you change the admin password, so it is a bootstrap credential, not a stored one. Hardening step: wire SSO, set `admin.enabled: "false"` in `argocd-cm`, and issue per-automation tokens instead.

kubectl create namespace argocd
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

# initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d

kubectl port-forward svc/argocd-server -n argocd 8080:443
argocd login localhost:8080 --username admin
argocd account update-password

Answer

`spec.source.targetRevision` tells the repo-server which revision to render. For a Git source it accepts a branch name (`main`), a tag (`v1.42.3`), a full or short commit SHA, and `HEAD`, which resolves to the repository's default branch. For a Helm repository source it is the chart version and accepts semver ranges such as `1.2.*` or `>=1.2.0 <2.0.0`.

The interesting part is the trade-off. A branch gives you continuous delivery, merge to main and the cluster follows, but the Application manifest no longer records what is running; the only record is `status.sync.revision`, the SHA the controller resolved at sync time. A pinned tag or SHA gives you reproducibility and a real audit trail at the cost of an extra commit per release, which is why teams pair it with a CI step or Renovate to bump the pin.

A semver range on a third-party chart is the risky middle ground: an upstream maintainer publishing 1.2.9 can change your production without any commit in your repo, so regulated teams pin exact chart versions. One more interaction worth naming: `argocd app rollback` replays an entry from `status.history`, but if `targetRevision` is a moving branch the next reconciliation drifts forward again, so a real rollback means reverting the branch in Git, not just clicking rollback.

# Git source pinned to an immutable tag
source:
  repoURL: https://github.com/example/manifests
  targetRevision: v1.42.3
  path: services/payments/overlays/prod
---
# Helm repository source, targetRevision is the chart version
source:
  repoURL: https://charts.bitnami.com/bitnami
  chart: redis
  targetRevision: 20.1.4

Answer

`CreateNamespace=true` makes ArgoCD create `spec.destination.namespace` if it does not exist, instead of failing the sync with `namespaces "payments" not found`. It creates a bare namespace, so if you need labels on it (Istio injection, Pod Security Admission levels) use `spec.syncPolicy.managedNamespaceMetadata`, which lets ArgoCD own those labels without you committing a Namespace object that then fights other tooling. The other options you should be able to name and justify: `PrunePropagationPolicy=foreground|background|orphan` decides what Kubernetes does with dependents when a pruned owner is deleted. `PruneLast=true` defers all deletes until every apply has succeeded, so a half-failed sync does not remove the old object before the new one exists. `ApplyOutOfSyncOnly=true` skips resources already in sync, which measurably cuts sync time on applications with hundreds of objects. `ServerSideApply=true` switches from client-side apply to server-side apply with field managers, and is the standard fix for `metadata.annotations: Too long: must have at most 262144 bytes` on large CRDs. `RespectIgnoreDifferences=true` makes the apply itself honour your `ignoreDifferences` rules rather than only the diff view. `Validate=false` disables client-side schema validation for CRs the client cannot resolve. `FailOnSharedResource=true` aborts instead of quietly stealing a resource another Application already tracks.

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    managedNamespaceMetadata:
      labels:
        istio-injection: enabled
        pod-security.kubernetes.io/enforce: restricted
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - RespectIgnoreDifferences=true
      - PruneLast=true
      - ApplyOutOfSyncOnly=true

Answer

An AppProject is ArgoCD's multi-tenancy primitive, it groups Applications and constrains what they're allowed to do. Without AppProjects, any Application can pull from any Git repo and deploy to any cluster/namespace, which is unacceptable at platform scale. A well-designed AppProject restricts: **sourceRepos** (which Git URLs are allowed), **destinations** (which cluster + namespace pairs), **clusterResourceWhitelist / namespaceResourceBlacklist** (which kinds of resources can be created, e.g. team 'payments' can't create ClusterRoles), and **roles** (RBAC for who can sync, override, or delete Applications in this project).

Razorpay's platform team typically gives each product team an AppProject scoped to their namespaces; the team controls what they deploy without being able to escape into kube-system or other teams' spaces. A few operational points that show real usage. Constraints are enforced by the application-controller at reconcile time, not only by the UI, so an Application created straight through kubectl with a disallowed `repoURL` lands in a `ComparisonError` reading 'application repo ... is not permitted in project'.

Destinations support wildcards in the namespace (`payments-*`) and, in recent releases, a `name` field matching the cluster Secret instead of a raw API URL, which survives cluster re-registration. `clusterResourceWhitelist: []` is not a no-op, an empty list denies all cluster-scoped kinds, which is exactly what you want for a product team and exactly what breaks their cert-manager ClusterIssuer if they own one. Project roles issue JWT tokens (`argocd proj role create-token payments developer`) for CI systems that must sync only their own applications. And `orphanedResources: { warn: true }` surfaces objects living in the project's namespaces that no Application owns, which is how you find the leftovers from the pre-GitOps era. Sync windows and `signatureKeys` for GPG-verified commits also attach here, making AppProject the single place where tenancy, guardrails and change control meet.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example/payments-manifests
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments-*
  clusterResourceWhitelist: []  # no cluster-scoped resources
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
  roles:
    - name: developer
      policies:
        - p, proj:payments:developer, applications, sync, payments/*, allow
      groups:
        - okta:payments-developers

Answer

App-of-apps is a bootstrap pattern where one parent Application's Git source contains nothing but other Application CRDs. ArgoCD syncs the parent, which creates the children, which sync their own workloads. Why bother?

You get a single root that you can revert to spin down everything, and you can bootstrap a new cluster by `kubectl apply`-ing just the root Application. Trade-off: when a child Application is OutOfSync, the parent shows Healthy/Synced (it only cares that the child CR exists, not its state), so monitoring needs to look at children, not just the root. In 2026, **ApplicationSets** have largely replaced app-of-apps for fanning out to many clusters; app-of-apps is still useful for bootstrapping a single cluster's platform layer (cert-manager, ingress, monitoring, etc.).

Two mechanics decide whether the pattern behaves. Ordering: children are just resources to the parent, so put `argocd.argoproj.io/sync-wave: "-2"` on cert-manager, `"-1"` on the ingress controller and `"0"` on workloads, otherwise everything applies at once and half the children go Degraded while waiting on CRDs that do not exist yet. Deletion: whether removing a child Application file from Git actually removes its workloads depends on `prune` on the parent plus the `resources-finalizer.argocd.argoproj.io` finalizer on the child.

Without the finalizer, the child CR is pruned and its Deployments are orphaned in the cluster, quietly consuming nodes forever. On the monitoring point, the usual fix is a health check customization for `argoproj.io_Application` in `argocd-cm` so a child's Degraded state propagates up to the parent, which restores the single-pane view people expected in the first place. The pattern also composes with ApplicationSets rather than competing: the root is often an ApplicationSet with a Git directory generator over `./apps/*`, so adding a platform component is one new folder rather than a new hand-written Application.

# parent Application's Git path contains:
# ./apps/cert-manager.yaml
# ./apps/ingress-nginx.yaml
# ./apps/prometheus.yaml
# ./apps/argocd-self.yaml
# Each of those files is itself an Application CR.

Answer

An ApplicationSet is a controller that templates out many Applications from a single declaration. It's the answer to 'I have one service that needs to deploy to 30 clusters' or 'each team has a deployment in each environment, I don't want to maintain 80 YAML files'. ApplicationSets use **generators** to produce parameters, and a **template** that renders an Application per parameter set.

Generator types: **List** (hard-coded list), **Cluster** (one Application per ArgoCD-registered cluster), **Git** (one per folder or file in a repo), **Matrix** (combine two generators, e.g. cluster × app), **Pull Request** (preview environments for open PRs), and **SCM Provider** (one per repo in a GitHub org). At Swiggy- or Zomato-scale infra, ApplicationSets are essential, no platform team manually maintains 1000+ Application YAMLs. Beyond the generator list, three things come up in interviews.

Templating: set `goTemplate: true` and use `{{.name}}` style expressions with sprig functions rather than the older fasttemplate `{{name}}` syntax, which cannot handle conditionals or default values; `goTemplateOptions: ["missingkey=error"]` turns a typo into a hard failure instead of a silently empty field. Deletion safety: by default the controller deletes Applications when they fall out of the generator's output, which means removing a cluster from the Cluster generator tears down its workloads. Guard it with `syncPolicy.applicationsSync: create-update` (never delete) or `preserveResourcesOnDeletion: true` when you want the Application removed but its resources left running.

Rollout control: a naive ApplicationSet updates all 30 clusters the instant the template changes, which is a fleet-wide blast radius. Progressive syncs (`strategy.type: RollingSync` with `steps` matched by label) roll the change canary cluster first, then the rest, and it is still gated behind an enable flag on the applicationset-controller, so check your version before promising it in production. `argocd appset generate -f appset.yaml` renders the output locally, which belongs in CI as a review artifact.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
spec:
  generators:
    - clusters: {}   # one Application per registered cluster
  template:
    metadata:
      name: '{{name}}-monitoring'
    spec:
      project: platform
      source:
        repoURL: https://github.com/example/addons
        targetRevision: main
        path: monitoring
      destination:
        server: '{{server}}'
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Answer

ArgoCD lets you order sync operations with **sync waves** and lifecycle **hooks**. Sync waves: annotate resources with `argocd.argoproj.io/sync-wave: "-1"` (or any integer), ArgoCD applies waves in ascending order, waiting for each wave to be Healthy before starting the next. Lower numbers go first; negatives are valid.

Hooks: annotate a resource with `argocd.argoproj.io/hook: <PreSync|Sync|PostSync|SyncFail>` to run it at a specific phase. Hooks are usually Jobs that do schema migrations (PreSync), smoke tests (PostSync), or alerting on failure (SyncFail). Hook deletion policies (`HookSucceeded`, `HookFailed`, `BeforeHookCreation`) decide what happens to the Job afterward.

Real-world pattern: PreSync Job runs `alembic upgrade head`, application Deployment in wave 0, PostSync Job hits `/healthz` to confirm the new version is live. The details that decide whether this works at 3am. Waves apply within a phase, and phases always run in order PreSync, Sync, PostSync, so a wave `-5` resource in the Sync phase still runs after every PreSync hook.

ArgoCD waits for each wave to report Healthy before starting the next, which means a resource kind with no health check (a bare ConfigMap, or a CRD without a Lua customization) is treated as instantly Healthy and provides no ordering guarantee at all. There is also a hard-coded delay of a couple of seconds between waves, so an application split into 20 waves pays a real latency cost on every sync. On hooks, the single most common outage is a PreSync migration Job whose name is static: Kubernetes Jobs are immutable, so the second sync fails with 'field is immutable' unless you set `hook-delete-policy: BeforeHookCreation` or generate a unique name. `HookSucceeded` deletes the Job right after success, which destroys the logs you will want during a postmortem, so many teams prefer `BeforeHookCreation` and keep the previous run visible. Finally, a failed PreSync hook aborts the whole sync and the app stays on the old revision, which is usually the correct behaviour for schema migrations but surprises people expecting a partial apply.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrations
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: payments-api:latest
          command: ["alembic", "upgrade", "head"]
      restartPolicy: Never

Answer

For built-in kinds (Deployment, StatefulSet, Service, Ingress, Job), ArgoCD ships health checks written in Lua. They inspect `status` and return `Healthy`, `Progressing`, `Degraded`, or `Suspended`. For custom CRDs (cert-manager Certificate, Argo Rollouts Rollout, Postgres CRs), ArgoCD looks for a Lua script under `resource.customizations.health.<group>_<kind>`.

You write a small Lua function that takes the resource object and returns a status. This is critical for any operator-based deployment, without a custom health check, ArgoCD considers the CR Healthy the moment it's accepted by the API server, which hides real failures. Practical points.

The ConfigMap key encodes group and kind with an underscore separator, `resource.customizations.health.<group>_<kind>`, and a group containing dots keeps them (`cert-manager.io_Certificate`), which is a frequent copy-paste error that silently leaves the default behaviour in place. The Lua sandbox has no network and no `os` access, and scripts that call unavailable libraries fail closed, so the resource shows Unknown rather than crashing the controller. Test before shipping: the repo has a health-check test harness, and `argocd admin settings resource-overrides health ./my-cr.yaml --argocd-cm-path ./argocd-cm.yaml` evaluates your script against a real object locally, which beats discovering a syntax error during a production sync.

Argo Rollouts, cert-manager, ExternalSecrets, Crossplane and the common database operators ship health checks in ArgoCD's bundled `resource_customizations` directory already, so check upstream before writing your own. A related knob is `resource.customizations.ignoreResourceUpdates`, which tells the controller not to re-reconcile on pure `status` churn from a chatty operator, a real CPU saving on large instances. Health also feeds `argocd app wait --health`, so a missing custom check makes your CI green before the dependency is actually ready.

# argocd-cm ConfigMap
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" and c.status == "True" then
          hs.status = "Healthy"; hs.message = c.message; return hs
        end
        if c.type == "Ready" and c.status == "False" then
          hs.status = "Degraded"; hs.message = c.message; return hs
        end
      end
    end
    hs.status = "Progressing"; hs.message = "Waiting for issuer"
    return hs

Answer

Three production-grade options, all of which keep ciphertext in Git and resolve to plain Secrets at sync time: (1) **Sealed Secrets** (Bitnami), encrypt with a controller-specific public key, commit the SealedSecret CR. Simple, no external dependencies, but you have to manage the controller's key. (2) **SOPS** (Mozilla), encrypt files with a KMS key (AWS KMS, GCP KMS, age). Use KSOPS or Helm Secrets as the ArgoCD plugin to decrypt at template time.

Great for multi-cloud teams. (3) **External Secrets Operator**, commit an ExternalSecret CR that references a real secret in AWS Secrets Manager / GCP Secret Manager / HashiCorp Vault. ESO syncs the value into a real K8s Secret. This is the dominant pattern in India at Razorpay-, Swiggy-, Zomato-scale because secrets rotation happens in the vault and pods pick it up without a redeploy.

Know the failure modes, not just the names. Sealed Secrets encrypts against one controller's private key, so restoring a cluster without restoring the `sealed-secrets-key` Secret makes every SealedSecret in Git permanently undecryptable, and the ciphertext is bound to namespace and name unless you set the `cluster-wide` or `namespace-wide` scope annotation. SOPS with KSOPS runs inside the repo-server as a sidecar plugin, which means the repo-server pod needs IRSA or workload identity for the KMS key, and a decrypt failure surfaces as a manifest generation error rather than anything secret-shaped.

External Secrets keeps the plaintext out of Git entirely but introduces a refresh interval (`spec.refreshInterval`), so a rotated value reaches the Secret on that cadence and running pods still need a restart unless you use Reloader or mount via a CSI driver. One point that impresses: whichever option you choose, tell ArgoCD not to render the diff, because a Secret's data appears base64 in the UI. Use `argocd.argoproj.io/compare-options: IgnoreExtraneous` where appropriate and the `hide-secret-data` behaviour in `argocd-cm`, and keep `logs, get` RBAC tight so nobody reads the value out of a controller log line.

Answer

ArgoCD has its own RBAC layer on top of K8s RBAC, because end-users don't touch the K8s API directly through ArgoCD. Policies are written in a Casbin-like CSV in the `argocd-rbac-cm` ConfigMap, with statements like `p, role:dev, applications, sync, payments/*, allow`. Roles map to **groups** that come from your IDP.

ArgoCD ships Dex as a bundled OIDC broker, or you can connect directly to Okta / Google / Azure AD / GitHub. SSO config goes in `argocd-cm` (`oidc.config`), group claims become group names in the policy file. Best practice: write policies against groups, never individual users, and use `g, <group>, <project-role>` to scope a group to one AppProject's role.

Specifics that matter in a review. The policy line is `p, <subject>, <resource>, <action>, <object>, <effect>` where object is `<project>/<application>`, so `payments/*` means every app in the payments project, not every app whose name starts with payments. Resources include `applications`, `applicationsets`, `projects`, `repositories`, `clusters`, `logs`, `exec` and `accounts`; recent major releases enforce `logs` and fine-grained sub-resource actions (`applications, update/*`, `applications, delete/*`) by default rather than folding them into a blanket update, which is a genuine upgrade-time breakage if your CSV predates it.

Deny rules exist and always win over allows, useful for carving one namespace out of a broad grant. `policy.default: role:readonly` is safer than an empty default, but `role:''` (deny by default) is the right setting for a regulated environment. On the IDP side, the group claim must actually be in the token: with Okta or Entra ID you often have to request the `groups` scope and add `requestedScopes` and `requestedIDTokenClaims` to `oidc.config`, otherwise every SSO user silently lands on the default role and the usual complaint is 'SSO works but I cannot see any applications'. Debug with `argocd account can-i sync applications payments/payments-api`.

# argocd-rbac-cm ConfigMap
data:
  policy.default: role:readonly
  policy.csv: |
    p, role:platform-admin, *, *, *, allow
    p, role:dev, applications, get, */*, allow
    p, role:dev, applications, sync, payments/*, allow
    g, okta:platform-team, role:platform-admin
    g, okta:payments-team, role:dev

Answer

Sync windows let you whitelist or blacklist sync activity by time. They're configured on an AppProject and can be **allow** (default deny outside the window) or **deny** (default allow except in the window). Use cases: freeze production deployments during high-traffic events (Black Friday, IPL final), restrict syncs to business hours so the platform team is around if something breaks, prevent any deploys to the prod cluster between Friday evening and Monday morning.

You can also set `manualSync: true` inside a window so manual syncs are still allowed while automated syncs are paused, useful for emergency hotfixes during a freeze. The mechanics deserve care because a misconfigured window is a silent outage of your delivery pipeline. `schedule` is standard cron in the controller's timezone, which defaults to UTC, so an India-based team writing `0 17 * * 5` gets a Friday 22:30 IST freeze rather than 17:00 IST unless they set `timeZone: Asia/Kolkata` on the window (supported in recent releases) or do the offset arithmetic themselves. `duration` is a Go duration string (`63h`, `2h30m`), not a cron end time, and overlapping windows resolve conservatively: if any deny window matches, syncs are blocked regardless of an allow window, and if any allow window exists then time outside every allow window is blocked. Selection is by `applications`, `namespaces` or `clusters` glob, so you can freeze only the prod cluster while dev keeps flowing. Two operational notes: an automated sync suppressed by a window is not queued, it simply does not happen, and the app sits OutOfSync until the window closes or someone syncs manually; and `argocd proj windows list <project>` plus the yellow banner on the Application page are how you confirm a freeze is actually active before a release call.

# AppProject snippet
spec:
  syncWindows:
    - kind: deny
      schedule: "0 17 * * 5"  # Friday 17:00
      duration: 63h            # until Monday 08:00
      applications:
        - "*"
      manualSync: true

Answer

ArgoCD itself does atomic sync, it applies the manifest and the Deployment rolls out via the standard K8s rolling update. For real blue/green or canary, you pair ArgoCD with **Argo Rollouts**, which is the sister project from the same team. Argo Rollouts replaces Deployment with a Rollout CR that supports blue/green (parallel old + new, switch service selector after analysis) and canary (gradually shift traffic 10% → 25% → 50% → 100%, with optional analysis at each step).

The analysis step queries Prometheus/Datadog/New Relic, if error rate or p99 latency violates thresholds, Rollouts auto-rolls back. Combined: ArgoCD syncs the Rollout manifest from Git, Argo Rollouts handles progressive traffic shifting. This is the standard pattern at platform teams in India for any service that handles user-visible traffic.

The integration detail that trips candidates up: ArgoCD does not understand a Rollout out of the box beyond the bundled health check, and during a paused canary the Rollout reports `Paused`, which ArgoCD renders as Suspended, not Degraded. If `selfHeal` is on and a human uses `kubectl argo rollouts promote` or `abort`, that live change is drift, and ArgoCD can undo the abort by re-applying the Git state, restarting the very canary you just killed. The fix is to keep promotion decisions declarative (analysis-driven, not manual) or to add the Rollout's `spec.paused`-adjacent fields to `ignoreDifferences`.

Traffic shaping also needs a provider: with plain Kubernetes Services, Rollouts can only shift traffic by replica ratio, which is coarse at low replica counts; real percentage control needs the `trafficRouting` block backed by Istio, NGINX ingress, ALB or a Gateway API implementation. Blue/green needs `activeService` and `previewService` plus `autoPromotionEnabled: false` if you want a human gate. And AnalysisTemplates fail closed only if you configure them to: set `failureLimit` and `inconclusiveLimit`, otherwise a Prometheus outage returns inconclusive and the rollout stalls halfway rather than aborting.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

Answer

Scale points and fixes: (1) **Application controller sharding**, by default one controller handles all clusters; set `controller.replicas` and `ARGOCD_CONTROLLER_REPLICAS` so each replica owns a subset of clusters. Pin a heavy cluster to its own shard via the `argocd.argoproj.io/secret-type: cluster` Secret. (2) **Repo server replicas**, stateless, just bump replicas. Make sure the Git clone cache (`/tmp/_argocd-repo`) has enough disk. (3) **Redis**, single-node Redis becomes a bottleneck around 1000 Applications; switch to Redis HA chart or AWS ElastiCache. (4) **Reconciliation interval**, bump from 3 minutes to 10+ if you have webhooks.

Less polling = less load. (5) **K8s API client-side throttling**, increase `--kubectl-parallelism-limit` if you see 'Waiting for sync' lag. Razorpay-scale: 3-5 controller shards, 5-8 repo-server replicas, Redis HA. Add the tuning knobs and the metrics that tell you which one to turn. `controller.status.processors` (20 by default) and `controller.operation.processors` (10) in `argocd-cmd-params-cm` govern how many apps are compared and synced concurrently; raising them without raising `controller.kubectl.parallelism.limit` just moves the queue.

Sharding algorithm matters at fleet scale: the legacy modulo hash reshuffles every cluster when a replica count changes, while the round-robin and consistent-hashing strategies (`ARGOCD_CONTROLLER_SHARDING_ALGORITHM`) keep assignments stable, and newer releases can rebalance dynamically. On the repo-server, `reposerver.parallelism.limit` caps concurrent manifest generations and `ARGOCD_EXEC_TIMEOUT` (90s by default) is what you raise when a large Helm chart dies with 'context deadline exceeded'. Watch `argocd_app_reconcile_bucket` for reconciliation latency, `argocd_git_request_total` for how often you are actually hitting Git versus the cache, and `workqueue_depth` on the controller: a steadily rising queue depth is the unambiguous signal that you need another shard. The often-missed win is cutting work rather than adding compute, via `ignoreResourceUpdates` for chatty operators and by excluding noisy kinds (Events, EndpointSlices) with `resource.exclusions` in `argocd-cm`.

Answer

Three related but distinct operations: **Refresh**, re-evaluate the current Git revision against live state, recompute Sync/Health status. Useful if you suspect a stale cache but didn't push anything. **Hard refresh**, same as refresh but bypasses ArgoCD's manifest cache, forcing the repo-server to re-template (`helm template`/`kustomize build`). Use when you've changed a remote Helm chart version or a Kustomize remote base. **Sync**, actually apply the diff to the cluster.

So: refresh updates the perception of state; sync changes the state. A common production gotcha: bumping a remote dependency in a Helm chart's `Chart.lock` and seeing 'no diff', fix is a hard refresh, not a regular sync. What is actually happening underneath: the repo-server caches generated manifests in Redis keyed by repo URL, resolved revision and the render inputs it knows about.

A plain refresh re-resolves the revision and re-runs the comparison, but if the resolved SHA is unchanged it happily reuses the cached render. A hard refresh (`argocd app get <name> --hard-refresh`, or the dropdown next to Refresh in the UI) drops that cache entry and forces `helm template` or `kustomize build` to run again. Anything the cache key cannot see needs a hard refresh: a floating tag on a remote chart, a `kustomize` remote base pointing at a branch, a values file pulled from another repo, or a change you just made to a Config Management Plugin.

You can automate it with `timeout.hard.reconciliation` in `argocd-cm`, which is disabled by default; setting it to something like `24h` catches slow-moving external drift without paying the re-render cost every cycle. Two more distinctions worth stating: refresh and hard refresh never mutate the cluster, they only update ArgoCD's view, and neither one is needed after a normal Git push because the webhook already invalidates the entry for that repo and revision.

Answer

Three idiomatic patterns: (1) **Multiple values files**, one per environment (`values-dev.yaml`, `values-prod.yaml`), passed via `helm.valueFiles` in the Application spec. Cleanest, easiest to diff. (2) **Inline values**, the Application has `helm.values: |` with overrides. Useful for small per-env tweaks (image tag, replica count). (3) **Helm + Kustomize together**, ArgoCD can render Helm output and then apply Kustomize patches via the `kustomize.helmCharts` feature.

Powerful but harder to reason about. Avoid the anti-pattern of one Helm chart with a big switch-case template; that gets ugly fast. For 5+ environments, ApplicationSet with a List or Git generator + valueFiles is the cleanest at scale.

Precision points an interviewer will check. Ordering is significant: `valueFiles` are merged left to right, so the environment file must come after the base or your overrides get silently overwritten, and a missing file fails the render unless you set `ignoreMissingValueFiles: true`. `helm.parameters` maps to `--set` and therefore forces string coercion on things like `image.tag: 1.42` becoming a number, which is why `helm.parametersFileParameters`-style typed alternatives and plain values files are safer for tags; `helm.fileParameters` maps to `--set-file` for injecting whole files such as a TLS chain. When the values file lives in a different repository from the chart (very common with third-party charts), use the multiple-sources form: one source is the chart, another is your config repo with `ref: values`, and `valueFiles: [$values/envs/prod.yaml]` resolves across them.

Also remember `helm.releaseName`, because ArgoCD defaults the release name to the Application name and templates that embed `.Release.Name` will rename every object if you rename the app, producing a full delete-and-recreate on the next sync. For per-environment structural differences rather than value differences, Kustomize overlays are the better tool.

spec:
  source:
    repoURL: https://github.com/example/charts
    targetRevision: main
    path: charts/payments-api
    helm:
      valueFiles:
        - values.yaml
        - values-prod.yaml
      parameters:
        - name: image.tag
          value: v1.42.3

Answer

Some fields get mutated after ArgoCD applies them, HPA managing replica count, Istio injecting sidecar containers, CNI plugins adding annotations. Without configuration, ArgoCD sees a permanent diff and either reports OutOfSync forever or fights the mutating controller with self-heal. Fix it with `ignoreDifferences` on the Application or AppProject.

You specify `group`, `kind`, and `jsonPointers` or `jqPathExpressions` to ignore. Common patterns: ignore `/spec/replicas` for HPA-managed Deployments, ignore Istio sidecar annotations, ignore CNI-added annotations on Pods. The trap that catches most teams: by default `ignoreDifferences` only affects the diff calculation, so the app reports Synced while the next sync still applies your Git value and stomps the HPA back to 3 replicas.

To make the apply itself respect the rule you need `syncOptions: [RespectIgnoreDifferences=true]`. `jqPathExpressions` handles the cases JSON pointers cannot, such as removing one container's resources from a list by name, and `managedFieldsManagers: ["kube-controller-manager"]` is the cleanest option of all: instead of naming fields, you tell ArgoCD to ignore whatever a given field manager owns, which is exactly right for HPA and for webhook-injected sidecars. Rules can be set per Application or globally in `argocd-cm` under `resource.customizations.ignoreDifferences.<group>_<kind>`, and the global form is what you want for a fleet-wide annoyance. Related knobs: `argocd.argoproj.io/compare-options: IgnoreExtraneous` on a resource you create but do not want tracked, and `resource.compareoptions: ignoreAggregatedRoles: true` for ClusterRoles whose rules are filled in by aggregation. Server-side diff, the default in recent major releases, removes a whole class of these false diffs because it asks the API server what would actually change instead of comparing against a last-applied annotation.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP

Answer

ArgoCD stamps every object it applies so it can find it again on the next reconciliation, and which stamp it uses is set by `application.resourceTrackingMethod` in `argocd-cm`. Three values exist. `label` (the historical default) writes `app.kubernetes.io/instance: <app-name>`. `annotation` writes `argocd.argoproj.io/tracking-id: <app>:<group>/<Kind>:<namespace>/<name>`. `annotation+label` writes both, so third-party tooling that keys off the well-known label keeps working while ArgoCD uses the richer identity. This is not trivia, it explains a whole family of bugs.

Labels are capped at 63 characters, so long Application names truncate and two applications can end up claiming the same object and fighting over it every cycle. `app.kubernetes.io/instance` is also a standard Helm label, so a chart that sets it itself can make ArgoCD adopt resources it never created. The annotation carries group, kind and namespace, so it correctly detects a resource that moved namespaces or changed ownership. If you must stay on labels, at least move the key with `application.instanceLabelKey: argocd.argoproj.io/instance` to stop colliding with Helm.

Switching methods has a migration cost: existing objects still carry the old stamp, so applications show extraneous or orphaned resources until each one is re-synced, which is why you do it in a window and then sync everything. The tell-tale symptom pointing here is resources going untracked right after an Application rename.

# argocd-cm ConfigMap
data:
  application.resourceTrackingMethod: annotation+label
  application.instanceLabelKey: argocd.argoproj.io/instance

# verify what ArgoCD stamped on a live object
# kubectl -n payments get deploy payments-api -o json \
#   | jq '.metadata.annotations["argocd.argoproj.io/tracking-id"]'
# -> "payments-api:apps/Deployment:payments/payments-api"

Answer

The legacy diff was computed in the controller using a three-way merge between your Git manifest, the live object, and the `kubectl.kubernetes.io/last-applied-configuration` annotation. That annotation is a full copy of the manifest, which doubles object size and is the direct cause of `metadata.annotations: Too long: must have at most 262144 bytes` on large CRDs. It is also blind: the controller cannot know what a mutating admission webhook or the API server's own defaulting will do to the object, so you get permanent phantom diffs such as an Istio sidecar showing as extraneous, `creationTimestamp: null`, or defaulted `resources` blocks reappearing every cycle.

Server-side diff instead sends the manifest to the API server as a dry-run apply and diffs the returned object against live state. Defaults, mutations and webhook injections are already baked into that response, so the diff shows what would genuinely change, and it composes with server-side apply so ArgoCD only claims the fields it manages. Recent major releases enable it by default; on older versions you turn it on per app with the annotation `argocd.argoproj.io/compare-options: ServerSideDiff=true` or globally with `controller.diff.server.side: "true"` in `argocd-cmd-params-cm`. Costs to acknowledge: each comparison becomes an API call, so a large fleet adds real API server load, the controller's service account needs apply-equivalent permissions for the dry run, and the first sync after switching can show one large diff while field ownership is rewritten.

# per-application opt-in
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
spec:
  syncPolicy:
    syncOptions:
      - ServerSideApply=true
---
# global, argocd-cmd-params-cm
data:
  controller.diff.server.side: "true"

Answer

`spec.sources` (plural, replacing the singular `spec.source`) lets one Application render from more than one repository. The dominant use case is an upstream Helm chart with your own values file: source one is the chart repo with `chart: redis` and `targetRevision: 20.1.4`, source two is your config repo carrying `ref: values` and no `path`, and the chart source refers to it as `$values/envs/prod/redis.yaml` inside `helm.valueFiles`. Before this landed, teams either vendored upstream charts into their own repo or maintained a wrapper chart with a `dependencies` block, both of which mean a commit dance for every upstream bump.

Rules worth knowing: only a source with a `ref` can be referenced, a ref-only source should not set `path` because it exists to be read rather than rendered, and the `$values` syntax is honoured in Helm value file paths, not in arbitrary fields. Sources that do render are concatenated, so two sources emitting the same object produce a duplicate-resource sync failure rather than a merge. Operationally, `status.sync.revisions` becomes a list with one entry per source, the UI shows a revision per source instead of a single SHA, and `argocd app sync --revision` needs `--source-position` to disambiguate.

Webhooks fire per source, so a push to either repo refreshes the app. Finally, the AppProject's `sourceRepos` must permit every repo involved, which is the usual cause of a 'not permitted in project' error right after splitting values out.

spec:
  project: platform
  sources:
    - repoURL: https://charts.bitnami.com/bitnami
      chart: redis
      targetRevision: 20.1.4
      helm:
        valueFiles:
          - $values/envs/prod/redis.yaml
    - repoURL: https://github.com/example/manifests
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: cache

Answer

The notifications controller ships with ArgoCD and reads `argocd-notifications-cm` plus `argocd-notifications-secret`. Three pieces compose an alert. A **service** holds delivery config (`service.slack`, `service.webhook`, `service.email`), with the token referenced from the Secret as `$slack-token`.

A **template** is the message body, rendered as a Go template over the Application object, so you have `{{.app.metadata.name}}`, `{{.app.status.sync.status}}`, `{{.app.status.operationState.syncResult.revision}}` and `{{.app.status.operationState.message}}` available. A **trigger** is a boolean expression over app state plus an `oncePer` key. You then subscribe with an annotation on the Application, `notifications.argoproj.io/subscribe.on-sync-failed.slack: platform-alerts`, or set default subscriptions in the ConfigMap so you are not annotating three hundred apps by hand.

The triggers worth wiring are `on-sync-failed`, `on-health-degraded`, `on-deployed`, and `on-sync-status-unknown`, which is the one people forget: it catches repo-server and credential breakage where no sync operation ever starts, so `on-sync-failed` stays silent. `oncePer: app.status.sync.revision` is the difference between a channel people read and one everyone mutes, because without it a persistently Degraded application notifies on every reconciliation. Two gotchas: a wrong Secret key reference fails quietly except for a line in the notifications-controller log, so test with a deliberate failure; and these alerts describe ArgoCD's view of the world, so they complement Prometheus SLO alerts on the workload rather than replacing them.

# argocd-notifications-cm
data:
  service.slack: |
    token: $slack-token
  template.app-sync-failed: |
    message: |
      :x: {{.app.metadata.name}} sync failed at {{.app.status.sync.revision}}
      {{.app.status.operationState.message}}
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      oncePer: app.status.sync.revision
      send: [app-sync-failed]
---
# on the Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: platform-alerts

Answer

A Config Management Plugin lets the repo-server render manifests with a tool ArgoCD does not support natively: cdk8s, an internal templating binary, Terraform output converted to YAML, or Helm wrapped in a custom decryption step such as KSOPS. Since the inline `configManagementPlugins` field in `argocd-cm` (CMP v1) was deprecated and then removed during the 2.x line, the only supported form is a sidecar container on the `argocd-repo-server` pod that speaks the CMP gRPC protocol over a shared socket. The setup: build a sidecar from your tool's image, mount `plugin.yaml` at `/home/argocd/cmp-server/config/plugin.yaml` from a ConfigMap, and share the `var-files`/`tmp` volumes so the sidecar can see the repo checkout the repo-server made. `plugin.yaml` declares `generate.command`, which must print valid YAML to stdout, and either a `discover` block (`fileName` or `find.glob`) for automatic detection or nothing at all if apps will name the plugin explicitly via `spec.source.plugin.name`.

Production failure modes worth naming: the sidecar runs as UID 999 and without the right volume mounts sees an empty directory, so 'no manifests generated' is usually a mount problem; the generate command inherits `ARGOCD_EXEC_TIMEOUT` (90s by default) and slow renders die with 'context deadline exceeded'; plugin output is cached like any other render, so iterating on plugin logic needs a hard refresh; and each sidecar adds memory per repo-server replica. Parameters come through `spec.source.plugin.env`, and plugin code is trusted code running next to your Git credentials.

# plugin.yaml mounted into the sidecar
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
  name: cdk8s
spec:
  version: v1.0
  init:
    command: ["sh", "-c", "npm ci"]
  generate:
    command: ["sh", "-c", "npx cdk8s synth -o - 2>/dev/null"]
  discover:
    fileName: "cdk8s.yaml"
---
# Application side
spec:
  source:
    plugin:
      name: cdk8s-v1.0
      env:
        - name: ENVIRONMENT
          value: prod

Answer

GitOps moves the blast radius into the merge, so the tests belong in CI on the manifests repo. A serious pipeline layers four checks. First, render exactly what the repo-server would render: `helm template` with the same value files, `kustomize build` with the same build options, and `argocd appset generate -f appset.yaml` for ApplicationSets, so a broken Go template fails in CI instead of generating forty malformed Applications.

Second, schema validation: pipe the rendered output into `kubeconform -strict -summary` with an extra `-schema-location` pointing at a CRD catalogue, so custom resources are checked too. This is what catches the error class where `spec.tempalte` renders fine and applies fine as an ignored unknown field. Third, policy: run conftest/OPA or Kyverno in CLI mode over the rendered YAML for organisational rules such as resource limits present, no `:latest` tags, no privileged pods, and an image registry allow-list.

Fourth, and highest value, diff against reality: `argocd app diff payments-api --local ./overlays/prod`, or `--revision <pr-sha> --server-side-generate` so the render happens in the repo-server rather than locally, posted as a comment on the pull request. Reviewers then see the effect on the cluster rather than a template change. Around that, use a PR-generator ApplicationSet for ephemeral preview namespaces and treat sync waves plus a PostSync smoke-test Job as the last gate. The honest caveat: none of this proves the workload is correct at runtime, so SLO alerting still owns that.

# CI: render, validate schema, enforce policy, diff against prod
helm template payments ./charts/payments-api \
  -f ./charts/payments-api/values.yaml \
  -f ./envs/prod/values.yaml > /tmp/rendered.yaml

kubeconform -strict -summary -ignore-missing-schemas \
  -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  /tmp/rendered.yaml

conftest test /tmp/rendered.yaml --policy ./policy

argocd app diff payments-api \
  --revision "$PR_SHA" --server-side-generate || true

Answer

Each component exposes Prometheus metrics on its own port: application-controller on 8082, api-server on 8083, repo-server on 8084, plus the applicationset and notifications controllers. Add ServiceMonitors for all of them and build alerts on four layers. Fleet state: `argocd_app_info` is a gauge labelled with `sync_status`, `health_status`, `project` and `dest_server`, so `count by (health_status) (argocd_app_info)` is your dashboard tile and a Degraded app persisting past fifteen minutes is a real page. `argocd_app_sync_total{phase="Failed"}` gives you a deployment failure rate you can trend per team.

Controller throughput: `argocd_app_reconcile_bucket` is a histogram of reconciliation duration, and when its p95 approaches `timeout.reconciliation` the controller can no longer complete a full pass, which means drift correction silently lags rather than erroring. Confirm with `workqueue_depth`, and fix with another shard or fewer watched resources. Git and cache health: `argocd_git_request_total` split by `request_type` shows whether you are serving from cache or fetching on every reconcile, and a sudden fetch spike usually means a cache key is churning.

Kubernetes pressure: a rising `argocd_kubectl_exec_pending` means you are at the parallelism limit or the target API server is throttling. Then alert on the unglamorous things that take ArgoCD down: TLS certificate expiry on the argocd-server ingress, Redis availability (a Redis outage presents as mass ComparisonErrors, not as a Redis alert), and the notifications-controller itself, because a dead notifier means every other alert stops arriving.

# p95 reconciliation latency per controller shard
histogram_quantile(0.95,
  sum(rate(argocd_app_reconcile_bucket[5m])) by (le, namespace)
)

# apps unhealthy for 15m (paging alert)
argocd_app_info{health_status="Degraded"} == 1

# sync failure rate by project
sum by (project) (
  rate(argocd_app_sync_total{phase="Failed"}[30m])
)

# controller falling behind
workqueue_depth{name="app_reconciliation_queue"} > 50

Answer

Two competing patterns and you pick based on blast radius tolerance: (1) **Hub-and-spoke**, one central ArgoCD instance manages all clusters. Operational simplicity (one UI, one RBAC layer), but a central failure or network partition affects everything, and you pay egress to reach remote clusters. Scale via controller sharding and ApplicationSets with a Cluster generator. (2) **ArgoCD per cluster**, each cluster runs its own ArgoCD that pulls from a per-cluster slice of the GitOps repo.

No single point of failure, lower egress, but operating 50 ArgoCDs is a job in itself. The hybrid Razorpay-style answer: one **management cluster** running a 'top-level' ArgoCD that bootstraps a per-cluster ArgoCD via ApplicationSet (one Application per spoke cluster, target manifest is the ArgoCD Helm chart). Each spoke ArgoCD then manages its own workloads from a region-specific path in Git.

Authentication via a shared OIDC, RBAC via AppProjects per team. Observability: ArgoCD Notifications + Prometheus metrics from every instance scraped into a central Thanos. Disaster recovery: the whole platform can be rebuilt by `kubectl apply`-ing the bootstrap Application into a fresh management cluster.

Two implementation details decide whether the hub option survives contact with reality. Cluster registration: `argocd cluster add` creates an `argocd-manager` ServiceAccount with cluster-admin on the spoke and stores a bearer token in a Secret labelled `argocd.argoproj.io/secret-type: cluster`. Long-lived tokens across fifty clusters are an audit finding waiting to happen, so on EKS use the `awsAuthConfig` form (the controller assumes a role and mints short-lived credentials) and on GKE use workload identity instead.

Shard placement: label heavy clusters with `argocd.argoproj.io/shard` in that same Secret to pin them, otherwise a replica count change reshuffles ownership across the fleet and every application reconciles at once. Also budget for the network: the hub controller holds watches against every spoke API server, so a flapping VPN or transit gateway shows up as intermittent Unknown status rather than as a network alert, and cross-region watch traffic is a real line item on the bill.

Answer

End-to-end flow: (1) **PR preview**, ApplicationSet with a Pull Request generator creates an Application per open PR, deploying to `preview-pr-<num>` namespace with a dynamic ingress. Comment on the PR with the preview URL via ArgoCD Notifications + GitHub webhook. (2) **Merge to main**, CI bumps the image tag in the production Kustomize overlay and pushes; ArgoCD picks it up via webhook within seconds. (3) **Canary**, the Application's manifest is an Argo Rollouts Rollout, not a Deployment. Rollouts shifts traffic 10% → 25% → 50% → 100% with pauses. (4) **Analysis**, each step has an AnalysisRun querying Prometheus for `success_rate{deployment="payments-api"}` and `histogram_quantile(0.99, ...) < 500ms`. (5) **Automated rollback**, if the AnalysisRun fails the threshold, Rollouts auto-aborts and routes 100% traffic back to the stable replicaset; ArgoCD reports the Rollout as Degraded; Notifications alerts Slack. (6) **Audit**, every step is in the Git log + the ArgoCD/Rollouts events.

This is exactly the shape used in production at India's K8s-native fintechs in 2026. Three things make or break this in practice. Preview environments leak money unless the PR generator is allowed to delete: leave `preserveResourcesOnDeletion` at false, put a ResourceQuota and LimitRange on the preview namespaces, and use `requeueAfterSeconds` plus a label filter (`argocd-preview` on the PR) so only opted-in PRs spin up infrastructure.

Image promotion must be immutable: CI writes a digest or an immutable tag, never `latest`, otherwise the canary and the stable ReplicaSet can resolve to different images with the same tag and the analysis result is meaningless. And the abort path needs testing, not assuming: deliberately ship a bad build to staging once a quarter and confirm the AnalysisRun fails, Rollouts shifts traffic back to stable, and Notifications actually posts to the channel. Cover the interaction with ArgoCD too, because a rollback that only happens inside Rollouts leaves Git claiming the broken revision is desired; either follow the abort with a revert commit or accept that the next reconcile re-attempts the same canary.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-previews
spec:
  goTemplate: true
  generators:
    - pullRequest:
        github:
          owner: example
          repo: payments-api
          labels: [argocd-preview]
        requeueAfterSeconds: 120
  template:
    metadata:
      name: 'payments-pr-{{.number}}'
    spec:
      project: previews
      source:
        repoURL: https://github.com/example/payments-api
        targetRevision: '{{.head_sha}}'
        path: deploy/preview
      destination:
        server: https://kubernetes.default.svc
        namespace: 'preview-pr-{{.number}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

Answer

Synced+Healthy means 'cluster state matches Git and resources report Healthy', NOT 'the application works'. The disconnect points to one of: (1) **Health checks lie**, the default Deployment health check just verifies pods are Ready. If your readiness probe is `tcpSocket` on port 8080, the pod is Ready as soon as the socket binds, even if dependencies (DB, downstream) are broken.

Fix: tighten readiness probes, add a custom Lua health check that calls `/healthz`. (2) **Stale image tag in Git**, Git says `v1.42.3` but `v1.42.3` itself is broken; ArgoCD did its job. Fix: roll forward or revert. (3) **Drift outside ArgoCD's scope**, someone updated a Secret directly that isn't managed by ArgoCD (e.g. external-secrets disabled), or a ConfigMap is mounted with `subPath` and pods need a restart. ArgoCD won't catch this. (4) **Network policy / Istio**, manifests synced fine, but a NetworkPolicy or Istio AuthorizationPolicy is denying traffic.

ArgoCD reports the policy as Healthy because it applied cleanly. (5) **Argo Rollouts mid-canary**, the new revision is broken but only 10% of traffic hits it, so most requests succeed; ArgoCD shows the Rollout as Healthy (Paused) during analysis. Approach: don't trust 'Synced+Healthy' alone, pair ArgoCD signals with application-level SLOs (success rate, latency, error rate) before declaring victory.

Answer

Pattern called 'ArgoCD manages ArgoCD'. Phases: (1) **Phase 0: install**, `kubectl apply -f https://...` the ArgoCD install manifest, or `helm install`. This is the only imperative step; everything after is declarative. (2) **Phase 1: self-manage**, commit the ArgoCD Helm values to your GitOps repo at, say, `bootstrap/argocd/`.

Create an Application called `argocd` whose source is that path and whose destination is the argocd namespace. ArgoCD now manages itself. (3) **Phase 2: bootstrap apps**, commit a root 'app-of-apps' or ApplicationSet that creates Applications for cert-manager, ingress, monitoring, external-secrets, and so on. Apply that one Application, everything else cascades.

After Phase 2, the entire cluster (including ArgoCD's own config, RBAC, projects) is in Git. Disaster recovery is a fresh K8s cluster + two `kubectl apply` commands. Gotcha: upgrading ArgoCD itself, when ArgoCD syncs a change to its own Deployment, the controller pod restarts mid-sync.

Mitigate by setting `syncOptions: ServerSideApply=true` and `Replace=false` so the operation completes idempotently on the next reconciliation. A few more things the self-management pattern demands. Never put `prune: true` and a wildcard path on the self-managing Application without care, because a bad render of the ArgoCD chart can prune ArgoCD's own Deployment and leave you with no controller to fix it; that is the one application where many teams keep automated sync off and require a manual click.

ArgoCD's CRDs are large enough to hit the client-side annotation size limit, so server-side apply is not optional for the self-app. Version skew is the other hazard: the controller applying a manifest that changes its own image tag will restart mid-operation, so upgrade during a quiet window and confirm `argocd version` reports matching client and server afterwards. Finally, back up what is not in Git, chiefly the cluster Secrets and any locally created accounts, with `argocd admin export -n argocd > argocd-backup.yaml`, because a rebuilt management cluster with no cluster credentials cannot reach a single spoke.

Answer

ArgoCD is genuinely well-suited to audit-heavy environments (which is why Indian fintechs under RBI scrutiny like Razorpay adopted it early) because GitOps gives you audit-by-default. Concrete controls: (1) **Immutable history**, Git commits are signed (`git commit -S`), and ArgoCD verifies signatures via the `gnupg` integration before syncing. (2) **Two-person rule**, disable direct pushes to the GitOps repo; everything goes through PR with required reviewers (CODEOWNERS, branch protection). The PR author cannot approve their own change. (3) **Segregation of duties**, SSO groups for developers can sync to dev/staging but not prod; only the platform team's group is mapped to the AppProject role with prod sync rights. (4) **Sync windows**, production sync windows allow only manual sync, requiring a human to click the button, captured in the ArgoCD audit log. (5) **Audit log**, argocd-server emits structured logs of every API call (who, what, when); ship to a SIEM with retention per your compliance requirement. (6) **Drift evidence**, the SelfHeal log line is your evidence that nobody is making out-of-band changes; review weekly. (7) **Rollback**, every deploy can be tied back to a Git commit, the PR review, the JIRA ticket, and the approver, closing the audit loop. Combined with Vault-backed secrets (rotation logged in Vault) and Argo Rollouts analysis traces, you have end-to-end evidence for any prod change.

Answer

Start by separating what is recoverable from Git and what is not, because most of ArgoCD is stateless. Applications, AppProjects and ApplicationSets are CRs that should already live in your GitOps repo through app-of-apps or an ApplicationSet, so they rebuild themselves. Redis is a pure cache and needs no backup.

What will actually block a restore is the state that never reaches Git: the cluster Secrets labelled `argocd.argoproj.io/secret-type: cluster` holding bearer tokens or IAM config for every spoke, repository and repo-creds Secrets, the server signing key inside `argocd-secret` (lose it and every issued API token and session is invalidated), locally created accounts and their tokens, and any GPG keys used for commit verification. The supported tooling is `argocd admin export -n argocd > backup.yaml` and `argocd admin import -n argocd - < backup.yaml`. Treat that dump as a live credential store: it contains cluster access in base64, so it belongs in an encrypted bucket with tight IAM, never in Git and never in CI logs.

Run it as a CronJob off the argocd-server image with a ServiceAccount scoped to those Secrets, and rehearse the restore, because `import` will prune resources that exist in the cluster but not in the dump unless you tell it otherwise. The stronger design makes the backup redundant: declare cluster registration and repo credentials as ExternalSecrets sourced from Vault and keep the signing key in your secret manager, so recovery is install, apply the bootstrap Application, and let it reconcile.

# scheduled export (run inside the cluster, write to encrypted storage)
argocd admin export -n argocd > /backup/argocd-$(date +%F).yaml

# what you must not lose, listed explicitly
kubectl -n argocd get secret \
  -l argocd.argoproj.io/secret-type=cluster
kubectl -n argocd get secret \
  -l argocd.argoproj.io/secret-type=repository
kubectl -n argocd get secret argocd-secret \
  -o jsonpath='{.data.server\.secretkey}'

# restore into a fresh install
argocd admin import -n argocd - < /backup/argocd-2026-08-01.yaml

Answer

Split the symptom into three, because each has a different root cause. Sync stuck in Running: read `status.operationState`, which names the phase and the resource being waited on. Typical causes are a PreSync hook Job that never completes (a migration blocked on a database lock), a sync wave whose resources have no health check so ArgoCD marks them Healthy instantly and moves on while the real dependency is not ready, or a custom Lua health check that can never return anything but Progressing. `argocd app terminate-op payments-api` cancels the in-flight operation, but the hook Job survives, so delete it or the next sync inherits the same block.

Add a `retry` block with backoff so transient failures self-resolve. Application stuck Deleting: this is the `resources-finalizer.argocd.argoproj.io` finalizer waiting on children that cannot delete, most often a namespace stuck Terminating on its own finalizer, a PVC still attached to a pod, or an admission webhook whose backing service no longer exists so the API server cannot complete the call. Diagnose the real blocker first with `kubectl get ns payments -o json | jq .status.conditions`; stripping the finalizer off the Application orphans workloads rather than deleting them, so it is a knowing choice, not a default. Stuck OutOfSync with an empty diff: hard refresh, then read the actual error from the app's conditions and the repo-server log, since `ComparisonError` carries the render failure verbatim and is far faster to read than clicking through the UI.

# what is the operation actually waiting on?
argocd app get payments-api -o json \
  | jq '.status.operationState | {phase, message, startedAt}'

# render or permission errors surface as conditions
kubectl -n argocd get app payments-api \
  -o jsonpath='{.status.conditions}' | jq

# cancel a hung sync, then clear the hook it left behind
argocd app terminate-op payments-api
kubectl -n payments delete job db-migrations

# last resort on a stuck delete: orphans the workloads
kubectl -n argocd patch app payments-api --type merge \
  -p '{"metadata":{"finalizers":null}}'

Answer

Reason through the paths rather than reciting a checklist. Identity: once SSO works, set `admin.enabled: "false"` in `argocd-cm`, because the local admin is a shared credential with full rights and no MFA; issue per-system tokens with `argocd account generate-token --account ci` and set `policy.default` to deny instead of `role:readonly`. Blast radius: the application-controller holds cluster-admin by design, so whoever can create an Application can deploy anything anywhere.

The real controls are AppProjects constraining `sourceRepos`, `destinations` and `clusterResourceWhitelist`, plus tight Kubernetes RBAC on Application CRs in the `argocd` namespace, since the kubectl path bypasses `argocd-rbac-cm` entirely. Supply chain: the repo-server executes Helm, Kustomize and any Config Management Plugin against repository content, and the historical vulnerability class here is escaping the checkout directory to read other tenants' values files or credentials, so patch promptly, run the repo-server with a read-only root filesystem and minimal mounts, and require GPG-signed commits via `signatureKeys` on production projects. Data plane: Redis stores rendered manifests, so write access to it means injecting workloads without touching Git; confirm the `argocd-redis` auth Secret is in place on recent versions and add a NetworkPolicy limiting Redis to ArgoCD pods. Exposure: keep the UI behind SSO and IP allow-listing rather than on the open internet, leave the terminal `exec` feature disabled, and grant `logs, get` deliberately, since container logs routinely leak secrets that your RBAC otherwise protects.

# argocd-cm
data:
  admin.enabled: "false"
---
# argocd-rbac-cm: deny by default
data:
  policy.default: ""
  policy.csv: |
    p, role:platform-admin, *, *, *, allow
    g, okta:platform-team, role:platform-admin
---
# AppProject: only GPG-signed commits may sync to prod
spec:
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23
---
# NetworkPolicy: only ArgoCD talks to Redis
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-redis
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: [argocd]

Answer

Two approaches with a genuine trade-off, and interviewers want the reasoning, not a preference. CI write-back is the mainstream choice: after the image is pushed, the pipeline edits the tag in the manifests repo (`kustomize edit set image payments-api=example/payments-api@sha256:...` or a `yq` in-place edit) and commits, usually directly for dev and through a pull request for production. Git keeps describing reality, review and audit stay intact, and rollback is a revert.

The costs are real: every service pipeline needs write access to the GitOps repo, and you must break the loop where a bot commit retriggers the build, via a bot-author filter or a skip token in the message. Argo CD Image Updater is a separate controller that polls registries and updates images itself, driven by annotations such as `argocd-image-updater.argoproj.io/image-list` plus an update strategy (`semver`, `newest-build`, `digest`, `alphabetical`). Its two write-back methods differ enormously: `argocd` patches the live Application through the API and is not GitOps at all, since Git no longer describes the cluster, while `git` commits back to the repo or into a `.argocd-source-<app>.yaml` override file.

Sensible policy: Image Updater with git write-back for dev and staging where speed matters, CI write-back with a PR gate for production, digests rather than mutable tags in prod, and registry credentials for the updater scoped read-only. Either way, exclude bot commits from change-approval metrics so your audit story stays honest.

# Image Updater: dev, git write-back, semver within a minor
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=example/payments-api
    argocd-image-updater.argoproj.io/api.update-strategy: semver
    argocd-image-updater.argoproj.io/api.allow-tags: regexp:^1\\.42\\.[0-9]+$
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
---
# CI write-back for prod: immutable digest, PR gated
# kustomize edit set image \
#   payments-api=example/payments-api@sha256:9f2c...
# git commit -m 'chore(prod): payments-api 1.42.3' && gh pr create

Answer

Diagnose before tuning. The symptoms are syncs that take minutes, `ComparisonError` messages containing 'context deadline exceeded', and repo-server pods getting OOMKilled; the confirming metrics are `argocd_git_request_total{request_type="fetch"}` climbing and `argocd_app_reconcile` p95 creeping toward `timeout.reconciliation`. The mechanism is specific to monorepos: one push invalidates the manifest cache for every Application whose `repoURL` matches, regardless of whether that service's path changed, so 400 apps each queue a manifest generation against the same commit and each needs a checkout.

Fixes in order of payoff. Add repo-server replicas first, since it is stateless, and only then raise `reposerver.parallelism.limit`, otherwise you are just reshaping the queue. Size the checkout volume properly, because concurrent clones of a large repo fill the default emptyDir and produce clone errors that look like Git problems.

Raise `ARGOCD_EXEC_TIMEOUT` above 90s only when a render is legitimately slow, never to paper over a loop. Set `timeout.reconciliation.jitter` so the periodic pass does not stampede on the same second. Cut controller-side churn with `resource.exclusions` for Events and EndpointSlices and `ignoreResourceUpdates` for chatty operators.

The structural answer, worth stating explicitly at the end: the durable fix is a smaller unit of work, which means splitting the GitOps monorepo per tenant or per platform layer. That also improves the RBAC story, because repo credentials and branch protection can then differ per team.

# argocd-cmd-params-cm
data:
  reposerver.parallelism.limit: "16"
  controller.status.processors: "50"
  controller.operation.processors: "25"
  controller.kubectl.parallelism.limit: "40"
---
# argocd-cm
data:
  timeout.reconciliation: 300s
  timeout.reconciliation.jitter: 60s
  resource.exclusions: |
    - apiGroups: [""]
      kinds: ["Event"]
      clusters: ["*"]
    - apiGroups: ["discovery.k8s.io"]
      kinds: ["EndpointSlice"]
      clusters: ["*"]
---
# repo-server: more replicas, bigger checkout volume
# kubectl -n argocd scale deploy argocd-repo-server --replicas=8
# env: ARGOCD_EXEC_TIMEOUT=180s