GitOps with Argo CD on Kubernetes: why we stopped applying manifests by hand from the CI pipeline

For a good while our Kubernetes deployment was just another step in the CI pipeline: the job would build, run the tests, and finish by running kubectl apply -f against the cluster using credentials stored as a CI secret. That worked fine until someone ran kubectl edit at three in the morning to fix an incident. That change landed directly on the live resources, and no later pipeline run touched that resource again, because the next commit didn't modify it. The cluster and the repository started drifting apart, and there was no automatic way to know by how much.

That's the structural problem with the push-based CI/CD model: the pipeline pushes changes to the cluster when something in the repository changes, but nothing watches the cluster the rest of the time. If the live state gets modified outside the pipeline - by hand, by an operator scaling a deployment, by some other tool touching the same namespace - that drift just sits there until someone notices it, or until an unrelated change to the same resource overwrites it without anyone realizing it also wiped out the emergency patch.

Pull-based instead of push-based

Argo CD flips the direction of the flow. Instead of the CI pipeline pushing manifests into the cluster, a controller running inside the cluster itself periodically polls a git repository, compares what it finds there against the actual state of the resources, and applies the difference. The CI pipeline no longer holds write credentials for the production cluster; its only job is to build the image, run the tests, and update the manifest repository with the reference to the new image. The thing that actually applies the changes is the controller inside the cluster, not an external job.

The central piece is the Application resource, which declares where the desired state comes from and which cluster namespace it maps to:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: catalog-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/tuurt/catalog-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: catalog
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true is the actual difference from a traditional pipeline: if someone edits the resource by hand inside the cluster, on the next reconciliation cycle Argo CD detects that the live state no longer matches what's declared in git and reverts the manual change, with nobody having to ask for it. prune: true does the symmetrical thing for deletions: if a manifest is removed from the repository, the corresponding resource is deleted from the cluster instead of being left orphaned.

Drift detection as a side effect of how it works, not a separate feature

Drift detection isn't a feature Argo CD bolts onto the sync process, it's a direct consequence of how it reconciles: on every cycle, it computes the diff between the manifest in git and the live state of the resource using the same logic it would use to decide what to apply. That diff is visible in the UI and from the command line even when selfHeal is turned off and Argo CD isn't correcting anything automatically:

argocd app diff catalog-service

With selfHeal off, that command is an auditing tool: it shows exactly what changed in the cluster since the last sync without reverting anything, which is useful in environments where a human should review the drift before deciding whether to correct it or whether the manual change should actually be folded back into the repository as a new commit. With selfHeal on, that same diff is what triggers the automatic correction. It's the same mechanism with a different decision about what to do with the result, not two separate features.

Keeping the manifest repository separate from the code repository

One decision we made early on was to keep the Kubernetes manifest repository separate from the application code repository. The code repository's CI pipeline, once it finishes building an image, makes an automatic commit to the manifest repository updating the image tag:

# Last step of the CI pipeline, after pushing the image
git clone https://github.com/tuurt/catalog-manifests.git
cd catalog-manifests
yq -i ".spec.template.spec.containers[0].image = \"registry.tuurt.com/catalog:$IMAGE_TAG\"" \
  overlays/production/deployment-patch.yaml
git commit -am "Bump catalog-service to $IMAGE_TAG"
git push

The separation avoids a concrete problem: if the manifest lives in the same repository as the code, every application commit - including changes that don't touch the deployment at all, like a test tweak - becomes part of the history Argo CD watches, and the audit diff fills up with noise unrelated to infrastructure changes. With a separate repository, the manifest history contains exclusively changes that actually modify what runs in the cluster, which matters the day someone needs to go back through that history to understand when something specific changed.

What didn't work: sync waves without explicit dependencies

For an Application with several components - the deployment, a database migration job, a configmap both of them read - we initially let Argo CD apply everything in whatever order it found the files in the manifest directory, assuming alphabetical file order would roughly match the logical order of application. It didn't: the deployment got applied before the migration job on more than one sync, and the pod started up against a database schema that didn't have the migration applied yet, with the predictable result that it failed to start until Argo CD retried after the job finished.

The fix was to use explicit sync waves through annotations instead of relying on the implicit ordering of files:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

The migration job sits in wave -1 and the deployment in the default wave, which is 0. Argo CD waits for every resource in a wave to finish - for a Job, to complete, not just to be created - before applying the next one. The order stopped depending on an implicit file-naming convention and became something declared where anyone reading the manifest can see it.

What changed with this model

The CI pipeline no longer needs write credentials on the production cluster, only write credentials on the manifest repository, which shrinks what a compromised pipeline could do directly against production. Drift between what's declared and what's actually running stopped being a question with no clear answer and became a diff you can pull up at any moment, whether it gets corrected automatically or not. And with the manifest repository split off from the code, it works as the real record of what was deployed to the cluster and when, instead of depending on reconstructing that history from CI pipeline logs that rotate and get lost over time.