How GitOps works, Argo CD, Flux and drift

Almost every cluster starts out being changed by hand, one command at a time. Someone runs kubectl apply from a laptop, or a build pipeline runs helm upgrade as its final step, and the application appears where it is supposed to be. That approach carries a team a surprisingly long way, and it starts to hurt once several people are deploying to several clusters.

The problem becomes, what is running in production right now, and does it match what you believe is running there. Who changed that memory limit, when did they change it, and did anybody review the change first. Somebody fixed an outage at two in the morning by editing a live object, nobody recorded what they did, and the next deployment overwrote the fix along with the problem it solved.

GitOps answers those questions by moving the source of truth out of the cluster and into a version-controlled repository. You write the desired state of the whole system into that repository, an agent reads it and applies it to the cluster, and the repository becomes the only sanctioned route for making a change. The cluster ends up matching whatever the repository says, and everything else in this primer follows from that one move.

GitOps describes a set of principles that several tools implement in materially different ways, so the word covers Argo CD, Flux, and Portainer's Git integration alike, and chapter four is where their real differences live. It also sits alongside continuous integration rather than replacing it, because CI builds and tests and publishes artifacts while GitOps takes a declared state and makes it real. Most teams run both, with a commit as the handoff between them.

The loop you already know

A controller compares the spec you declared against the status it observes, then acts to close the gap, continuously and without being asked, which is the mechanism chapter one of the Kubernetes primer covered. Chapter eleven then showed you that same loop handed outward to operators, watching your own custom object types instead of the built-in ones.

GitOps is that loop pointed one level further out again. An agent compares an entire repository against an entire cluster, closes whatever gap it finds, and then does it again. It is level-triggered in exactly the way the core controllers are, so it looks at the state of the world as it finds it rather than depending on having seen every event along the way. An agent that was restarted or briefly disconnected simply looks at reality when it wakes up and corrects whatever gap is there.

The repository becomes the authoritative record of intent, which matters more than any feature you will read about later, so asking the cluster what is running answers a different question from asking what should be running. When the two disagree, the repository is correct by definition, and reconciliation is the thing that resolves the disagreement.

Rolling back a bad change becomes reverting a commit and letting the agent do the rest, which turns your recovery path into an operation your team already performs several times a day for unrelated reasons.

The four principles

The OpenGitOps project, a working group under the CNCF, published a short statement of what GitOps requires. A system managed by GitOps expresses its desired state declaratively. That desired state is stored in a way that enforces immutability and versioning and retains a complete version history. Software agents automatically pull the desired state from the source. And software agents continuously reconcile, observing actual state and attempting to apply the desired state.

Read the second principle closely, because it describes the properties the storage has to have rather than naming Git specifically. Git is the usual answer and the reason the model carries that name, and an OCI registry holding versioned artifacts, each one addressed by a hash of its own content, satisfies the same requirement. Several implementations accept OCI artifacts as a first-class source for that reason, which matters in air-gapped sites where a registry is already how artifacts reach the network.

The fourth principle is where the implementations diverge most, and most arguments about what counts as real GitOps turn out to be arguments about one word in it, continuously. Hold that word loosely in your head until chapter four deals with it properly.

The push model is the one most teams arrive from, and it is easy to understand. A pipeline finishes building and testing, and its last stage runs kubectl apply or helm upgrade against the target cluster. It is quick to set up, works perfectly well at small scale, and its limitations take a while to surface. Every pipeline runner that can deploy holds cluster credentials and has a network path into your API server, so your CI system becomes part of the cluster's attack surface. Deployment happens only while the pipeline is running, so nothing is watching the cluster in between. And the record of what was deployed lives in build logs that rotate away on their own schedule.

The pull model turns that direction around, so an agent runs inside the cluster, or beside it in a management plane, and fetches the desired state from the repository itself. Credentials now point outward from the cluster rather than inward toward it, your API server no longer needs to be reachable from CI at all, drift gets corrected between deployments as well as during them, and the record of intent is commit history that does not expire.

Your CI system builds the image, runs the tests, scans it, signs it, and pushes it to a registry, and then writes a commit that updates the image reference in your deployment repository, and GitOps takes over from that commit onward. The handoff being a commit rather than an API call is what makes this preferable to having the pipeline call a deployment webhook directly, because your pipeline stops needing network reachability to the delivery platform and the repository stays the unconditional record of what was asked for.

A reconciliation cycle is five steps, and every implementation performs some version of all of them. The agent fetches the source at a specified reference, then renders that source into plain Kubernetes objects, which means running a Kustomize build, templating a Helm chart, or doing nothing at all when the repository already holds raw manifests. Chapter 6 covers what Kustomize and Helm each do, and the point here is that the agent always ends up holding plain objects whichever format it started from. It compares those rendered objects against what is live in the cluster, applies whatever differences it finds, and reports the outcome so you can see what happened.

What a sync does and does not tell you

A successful sync tells you less than it appears to, because applying an object successfully means the API server accepted it and nothing beyond that. Implementations that assess workload health separately from sync status are handing you two genuinely different signals, and the health one is the readiness and liveness territory from chapter six of the Kubernetes primer. Watch both at the moment you are deciding whether a release worked.

Ordering

The cluster will not work out the ordering on your behalf. Custom resource definitions have to exist before the custom resources that use them, and that follows directly from what chapter eleven said about a CRD being only a shape until a controller acts on it. Cert-manager has to be running before an ingress asks it for a certificate, and a database operator has to be installed before you declare a database. Argo CD expresses this with sync waves, Flux with dependency declarations between its Kustomization objects, and Helm with chart hooks, and the underlying requirement is identical in all three.

Pruning

Pruning means that removing a resource from the repository removes it from the cluster, and that behavior is what makes the repository authoritative rather than merely additive. Read the scoping semantics of your own implementation before you change it.

What starts a cycle

Polling means the agent asks the repository at a fixed interval whether anything has changed. It is predictable, it works when the repository has no route back to the agent, and its interval sets both your worst-case deployment latency and a multiplier on load against the repository and the cluster API. Webhooks mean the repository notifies the agent the moment a push lands, which brings latency down to almost nothing and fails silently whenever a delivery goes missing. Configure both of them, with webhooks for latency and polling as the backstop that catches whatever a webhook missed, because choosing only one leaves you either slow or occasionally wrong and running the two together costs almost nothing.

Webhooks are unavailable wherever the agent cannot be reached from the repository, and that covers more real environments than you would expect. Air-gapped clusters, remote industrial or retail sites behind network address translation, and management connections opened on demand rather than held open all fall into that category. Polling is the only mechanism available in those places, so the interval stops being a default you accept and becomes a number you choose deliberately and write down.

The implementations diverge on a single axis that carries two questions. Where does the reconciler run, and how continuously does it observe the cluster. Both questions have defensible answers, so this is a design decision rather than a ranking.

Argo CD

Argo CD runs as a set of controllers inside a cluster, principally an application controller, a repository server that fetches and renders sources, and an API server sitting behind its interface. It reconciles continuously, and it can target either the cluster it runs in or remote clusters registered to it, so one instance can serve a whole fleet, or you can run one per cluster, or one per group of clusters. With automated sync, pruning, and self-healing turned on together, an out-of-band change to a managed resource is reverted within seconds. ApplicationSets generate applications from a template plus a generator, so adding a labeled cluster or a new directory produces deployments without anyone writing another manifest by hand.

Flux

Flux runs as a set of narrower controllers, each with a single job. A source controller acquires artifacts from repositories, OCI registries, Helm repositories, and buckets, then serves them internally to the others. A Kustomize controller and a Helm controller apply what it fetched, and a notification controller handles events in both directions. Two further controllers, installed optionally, scan registries for new image versions and write updated tags back into the repository, which is the one situation where a GitOps agent legitimately commits to Git. The standard deployment bootstraps Flux into each cluster, and a hub-and-spoke arrangement is supported too, with one cluster's controllers reconciling the others through their kubeconfigs, the credential files that say how to reach a cluster and as whom. Reconciliation is continuous either way, and the source controller's local artifact cache keeps drift correction working through a repository outage.

Portainer

Portainer runs its reconciliation engine in its own control plane rather than inside the managed clusters. Those clusters never poll the repository and never hold repository credentials, and they are engaged at the point an update is applied to them. Reconciliation is event-driven, triggered by deployment events and by polling intervals, rather than being a loop that watches live cluster state between cycles. An option to re-apply the full declared state on every cycle regardless of detected change is available where convergence matters more than the API load it costs.

The tradeoff, from both sides

Continuous in-cluster reconciliation gives you the fastest drift correction available and satisfies that fourth principle in its strongest form, and it costs you an agent plus read credentials in every cluster, per-cluster configuration to maintain and upgrade, and real scaling work as the fleet grows, with component tuning becoming a platform task of its own at large application counts. Centralized event-driven reconciliation gives you one credential store, one audit stream, nothing to bootstrap or upgrade in each cluster, and no cluster-side dependency on the repository, and it costs you the window between cycles, during which the cluster can be wrong with nothing observing it.

Per-cluster agents keep each cluster independent, so a cluster carries on reconciling from its cached source even when the management plane is unreachable. Hub and centralized models concentrate delivery for the whole fleet into one place, which becomes a single point of failure for delivery even though the running workloads carry on unaffected.

These approaches coexist far more often than a side-by-side comparison implies. Running Argo CD or Flux for continuous repository-driven delivery while a separate operational control plane handles access control, fleet visibility, policy, and day-two workflows is a common arrangement in mature platform teams, and neither layer displaces the other.

Drift is the condition where live state has stopped matching declared state. It happens because somebody edited a live object during an incident, because a mutating admission webhook injected a sidecar, because the horizontal pod autoscaler from chapter ten adjusted a replica count you also declared, or because a controller wrote defaults into fields you left empty.

Fields written by other controllers will always differ from what you declared, so a naive full comparison reports permanent drift that nobody can ever clear, and everyone learns to ignore the signal. Every workable implementation therefore needs a way to disregard fields it does not own, and server-side apply with field ownership is what makes that tractable instead of a hand-maintained ignore list.

You can detect real drift and alert, leaving a human to decide what happens next, or you can detect it and correct it automatically. Automatic correction is the safer choice, and it has one consequence you need to plan for in advance. If an engineer applies an emergency fix by hand during an outage, an agent with self-healing enabled will revert that fix, possibly within seconds, and the outage resumes with the added confusion of nobody understanding why. The procedure for that has to exist in advance, and it has four steps. Suspend reconciliation for the affected workload, apply the fix, commit the fix, and then resume reconciliation, making sure whoever is on call knows the sequence without having to look it up.

GitOps describes your cluster while an engineer can still apply manifests to production by hand, and it governs your cluster once that route is closed. Closing it means cluster-level role-based access control of the kind chapter fourteen described, admission policy that rejects changes arriving outside the sanctioned path, and removing the console and form-based deployment options that let somebody bypass the repository without intending to.

The two categories

Cluster configuration is everything defining the governance baseline, which means namespace definitions and labels, network policies including the default-deny starting position, resource quotas and limit ranges, roles and bindings, ingress classes, storage classes, disruption budgets, and admission control policies. Application configuration is the workloads themselves, so deployments, statefulsets, daemonsets, services, application ingress rules, autoscalers, volume claims, config maps, and references to secrets.

Every item in that first list appeared in chapter fourteen of the Kubernetes primer as something stock Kubernetes deliberately leaves for you to fill in. Putting them under the same repository control as your applications is how the filling-in becomes repeatable rather than something one person did once by hand and then left.

Cluster configuration lands first, so that applications deploy into namespaces which already have their quota, network policy, and access bindings in place, and getting that order backward causes real problems. That sequencing only works when the governance objects live in the same system as the workloads they govern.

Separating code from configuration

Separating application source code from deployment configuration pays back faster than anything else in this chapter. Source lives in one repository and deployment configuration lives in another. That separation stops a build from being triggered by the commit that updated an image tag, which is otherwise a loop you will build by accident and then spend an afternoon diagnosing. It also lets repository permissions differ, so a wide group can write code while a narrower group approves what actually runs in production, and it keeps deployment history legible because the log of the deployment repository is a log of deployments.

Separating environments

Branch-per-environment gives each environment its own branch inside one repository, so developers commit to a development branch, promotion to staging is a merge, and promotion to production is a merge into main. Its strength is that promotion becomes a pull request, so every promotion arrives with a review gate and leaves a merge commit behind as its record. Its weakness is divergence, because a branch receiving an environment-specific fix accumulates differences, and every later merge grows more expensive and more likely to carry something you did not intend.

Directory-per-environment keeps one branch and separates environments into directory trees, so production sits alongside staging and development in the same commit graph. Promotion becomes a change within that branch, usually an overlay edit or a values file update. Its strength is that nothing diverges and the whole system is visible in one tree at one revision. Its weakness is that a commit straight into the production directory goes live immediately, unless branch protection and required review are standing in the way.

Directory-per-environment is the more common modern choice, because its weakness is fixable with repository controls you should have configured anyway, while branch divergence never goes away.

Manifest formats

Raw Kubernetes YAML is easy to read, and it duplicates heavily as soon as you have more than one environment. Kustomize handles that with a base directory of common definitions and per-environment overlay directories that patch it, working by patching rather than templating, and it already ships inside kubectl. It suits cases where the differences between your environments are few and structural.

Helm suits the opposite case, where the differences are numerous and parameterized, and you will need it regardless because almost all third-party software ships as a chart. The Helm primer covers how charts and values actually work, and the pattern here is one values file per environment sitting alongside the chart. Pin the chart version rather than letting it float, and pin image digests wherever your registry makes that practical.

Larger platforms sometimes add a third step called rendered manifests, where CI does the templating and commits the finished plain YAML, so a reviewer sees exactly what will change rather than the values that will change it.

Whichever format you land on, keep the repository self-contained, because an agent that cannot resolve part of your tree has no way to tell you which part it missed.

A Kubernetes Secret is only base64 encoded, as chapter nine of the Kubernetes primer covered, which is an encoding rather than any form of protection. Committing one to a repository publishes it permanently, to everybody who can read that repository now and to everybody who reads its history later.

Encrypting in the repository is one of the two workable patterns, and SOPS is the tool most teams use for it. SOPS is a command-line editor for encrypted files, and it encrypts only the values while leaving the field names in plain text, so a reviewer can see that a password changed without seeing the password itself. The key that protects those values is either an offline one, usually an age keypair because that is two short strings rather than a whole keyring, or a cloud key management service. Sealed Secrets inverts the trust, because a controller in the cluster holds the private key and only that controller can decrypt the ciphertext you committed. Both keep the repository complete, which means a cluster can be rebuilt from the repository alone. Both cost you key management and rotation, and both leave ciphertext in history permanently, so a compromised key becomes a retrospective problem rather than a forward-only one.

Keeping only references in the repository is the other pattern, where the External Secrets Operator or the secrets store CSI driver reads an object naming a secret held in Vault, AWS Secrets Manager, Azure Key Vault, or something similar, then materializes the value at runtime. The repository never contains the value in any form, rotation happens centrally in the platform built for it, and the repository is no longer self-contained, which changes your recovery procedure.

Referencing is the better choice for organizations already running a secrets platform, because rotation then happens in the system built to do it. Encrypting in the repository is the pragmatic choice where no such platform exists, and it improves substantially on whatever it usually replaces.

Reference types

Every implementation maps a deployment to a specific reference in the source, and there are three kinds of reference to choose from. A branch reference tracks the head of that branch and moves with every merge. A tag reference names a specific point and stays there until somebody deliberately moves it. A commit hash is immutable and gives you the strongest guarantee available.

Branch tracking suits development and staging, where rapid iteration is worth more than control over timing. Production should track a tag or a commit hash instead. Under branch tracking, production changes whenever somebody merges, which puts the timing of every production change in the hands of whoever happened to merge rather than whoever is accountable for that change landing.

A representative promotion

A feature merges into the development branch, the development environment reconciles, and the change gets validated there. A reviewed pull request promotes it to staging, which reconciles and gets validated in turn. A release tag is then cut from the validated staging revision. An authorized operator updates the production configuration to reference that new tag, which is itself a controlled change subject to its own access control, and production reconciles against an explicit, immutable release.

Organizations with formal change approval turn automatic synchronization off on production, so a person has to trigger the reconciliation once the change has been signed off.

Repository access is production access

Once the repository is the only path into the cluster, repository access has become production access, and it needs governing to the same standard. That means branch protection on production references with required review and required status checks, restrictions on who can push directly, protected tags, and repository permissions reviewed as seriously as the cluster RBAC from chapter fourteen. A team that spent months tightening cluster access and left the deployment repository writable by everybody has not made production any safer.

Agent credentials should be read-only, because a GitOps agent reads desired state and applies it, and the one exception is image automation writing tags back. Whatever kind of credential it is, it belongs to a service identity rather than to a named person whose departure would quietly break delivery for everybody.

Audit

Repository history tells you what changed, why, and who approved it, since the pull request carries the review. Your agent's log tells you when that change was applied and to which environment, which the repository has no way of knowing. Neither half is sufficient alone, so send both to your SIEM, because together they let you answer, months afterward, who approved a change, when it merged, and when it actually reached production.

When the repository is unreachable, your running workloads are unaffected and keep serving on their last applied state. What you lose is the ability to change anything and, depending on the implementation, the ability to correct drift. Agents that maintain a local cache of the fetched artifact keep reconciling against that cache, which is a meaningful difference during a long outage of a hosted repository service, so find out which behavior yours has before you need to know.

When the agent or control plane goes down, the shape is the same as chapter thirteen of the Kubernetes primer described, because the data plane keeps flowing while the management plane goes dark. Per-cluster agents localize that to one cluster, while hub and centralized architectures make it fleet-wide, and that is the concrete cost of the topology choice from chapter four.

When a bad commit lands, the agent applies it faithfully, because applying what the repository says is the whole job. Reverting the commit is your rollback path, and rehearsing that path before you need it under pressure is time well spent. This is also where health assessment and progressive delivery earn their keep, because a sync reporting success while the application degrades is the failure a minimal setup will never catch.

Point a fresh cluster at the repository and it converges to the declared state, so rebuilding after a disaster becomes an ordinary job rather than a procedure nobody has tested. It works only when the repository is genuinely complete, which is why the secrets pattern and the coverage of cluster configuration matter more than they first appear to. A repository that rebuilds most of a cluster will not get you back into production.

The principles at opengitops.dev are the authoritative statement, and they are deliberately implementation-independent, which makes them the right thing to read before any vendor documentation. Read the documentation of whichever engine you actually run after that, because the differences that matter in practice live in reconciliation behavior, drift handling, and pruning semantics, and those are the details a primer cannot settle on your behalf.

The fastest way to internalize this is to break things deliberately somewhere that breaking things is free. Edit a live resource by hand and watch whether it gets reverted, and how long that takes to happen. Delete something from the repository and see whether it disappears from the cluster. Revert a commit and time how long the rollback actually takes. Suspend reconciliation, apply a fix by hand, commit it, and then resume. Those experiments teach you what your configuration actually does, which is a different thing from what it is supposed to do.

Understanding Kubernetes explains the reconciliation model that GitOps extends, and how Helm works explains the packaging format most GitOps repositories are built around. When you want to know what any of it is actually doing once it is running, understanding observability is the next lesson. Outside the sequence, the aside on designing systems that solve the right problem covers how the decision to build any of this should have been reached in the first place.