Comparison: Flux vs Argo CD

We’re Earthly. We streamline building software using containers - perfect for using alongside GitOps tools like Flux and Argo CD. It’s all about simplifying your CI/CD process. Give it a look.

Since February we have been working on adopting Kubernetes and cloud-native technologies for our cell simulation platform at Turbine.ai. Part of my job entailed figuring out how to onboard developers who didn’t practice DevOps before.

Companies I’ve worked at during the past 7 years have all used Kubernetes in some way; and the last one, Turbine.ai, adopted it with my lead. It’s been quite a journey for me since I first encountered the technology in 2015, not long after securing my first full time role as a software engineer at a SaaS startup. Back in those days, the only cloud vendor that had a managed public [K8s](/blog/k8s-autoscaling offering was Google Cloud Platform (GKE). The tech was fresh and all backend engineers at our company were pretty hyped about migrating to GKE from Heroku.

Later that year, I moved to a role more aligned with my aspirations of working on distributed data processing pipelines with Apache Spark and had little exposure to K8s for over a year and a half. My path eventually lead back to the container orchestrator when I started working with the machine learning team, running ML workflows in our cloud environment. I recall manually installing and upgrading Apache Airflow (which was the only service I operated) with Helm, all from my development laptop. If the templates rendered the release was good to go.

Fast-forward to my next role where we operated an internal data platform with a more mature development lifecycle. I encountered GitOps for the first time here, as backend teams were using Argo CD to deploy their applications. It was a radical quality of life improvement over what I’d been practicing previously. Automatic change detection, a nice GUI, alerts on failures and a unified delivery approach for all applications instead of pile of deployment scripts; what’s not to love?

Then, I was hired by Turbine.ai, which is a cool biotech startup. In the simulation team, we decided to move our workflow orchestrator to K8s, with expectations that other services will follow suit gradually. However, there was a problem. Backend developers weren’t generally practicing DevOps in the company, and even a simple configuration change in a web app deployment often involved a sysadmin in the loop. Moving to K8s can be daunting for such newcomers, as they have to learn how to rebuild their existing applications according to cloud native application development principles such as 12 factor app; learn the fundamentals of K8s alongside with its limitations and idiosynchronicities, pick up new tools and infrastructure components, etc. Moreover, the cloud native landscape is vast and rapidly changing, so the best way to do X might be completely different than it was two years ago.

So, long story short, I was afraid that if we didn’t offer a smooth developer experience, DevOps would be too much pain to do, which would lead to backlash. GitOps is a method that can largely simplify infrastructure operations for developers, and I managed to convince our team that we should deliver it as part of the milestone marking k8s general availability for the rest of the teams. But what is GitOps and how can it help?

What Is GitOps?

The term was coined by Weaveworks with the following definition found on gitops.tech:

GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools. The core idea of GitOps is having a Git repository that always contains declarative descriptions of the infrastructure currently desired in the production environment and an automated process to make the production environment match the described state in the repository. If you want to deploy a new application or update an existing one, you only need to update the repository - the automated process handles everything else. It’s like having cruise control for managing your applications in production.

Making git the single source of truth for cluster state has many benefits. Without completeness:

1. Offers observability and time-travel with the full change history recorded. This simplifies rollbacks and helps developers move with confidence.
2. Enables modifying the application’s configuration and source code with a unified approach (even in a single changeset)
3. Simplifies the sharing and reuse of common configuration patterns (eg. with ordinary file editing / templating tools)
4. Enables the adoption of already existing DevOps/CI practices to infrastructure, such as static validation, tests, manual approvals, automated vulnerability scans, etc.
5. Git is the industry standard for source control, everyone should use it already

After this short introduction, now it’s time to get on with our topic: comparing Argo CD and Flux, two popular GitOps tools. If you are completely new to GitOps, you’ll certainly want to learn more before going ahead. If this is the case, gitops.tech is a good place to continue. You can also find plenty of videos on YouTube.

Introducing the Two Contenders

initial release	Flux2: Jun 25, 2020 Flux (succeeded): Jun 27, 2017	Mar 18, 2018
license
maturity	CNCF Incubating Project LF Project CNCF End User Tech Radar Continuous Delivery, June 2020: Adopt	CNCF Incubating Project LF Project CNCF End User Tech Radar DevSecOps, September 2021: Adopt
enterprise offering	Weave GitOps Enterprise	Akuity

Both Flux and Argo CD are very popular with an active community. Flux defines itself as “a set of continuous and progressive delivery solutions for Kubernetes that are open and extensible”, whereas Argo CD is “a declarative, GitOps continuous delivery tool for Kubernetes”. Based exclusively on this, one might conclude that there’s no clear distinction in their mission statement, however as we dive deeper, we will see that they take a different approach and offer a slightly different feature set.

Argo CD is part of Argo, an umbrella project comprising of multiple productivity focused tools, and is currently incubating under the CNCF. Jesse Suen, creator of the Argo project, told in Kubernetes Podcast #172 about the origins of Argo CD: “we needed to build a delivery tool for developer teams (at Intuit - ed) and we heavily focused on things like the user experience and the UI, and GitOps happened to be the mechanism we chose to do the delivery aspect of it”. He claims that Argo CD is more developer-experience-centric, whereas Flux is more operator centric. There has been an attempt to merge the two projects, but in the end the Flux team went with a different approach which became the GitOps Toolkit (Flux2).

Flux predates Argo CD and has been around since 2017. I explore the second major version of Flux, which resolves many shortcomings, offers better observability, ease of integrating, composability, and extensibility over the first, which is now in maintenance mode. Flux 2 is comprised of GitOps Toolkit components, k8s operators that reconcile GitOps resources of different kinds. For example, the source controller is responsible for source repositories, the helm controller - Helm releases, etc.

This article follows with the comparison of the two frameworks organized by core aspects, such as how they carry out reconciliaton, what tools they support, etc. Bear in mind that I do not attempt a full comparison, for the sake of conciseness and because of my limited research, covering the core functionalities and our use cases. I still believe that it could prove useful for many.

Reconciliation

Reconciliation or synchronization (sync) is the act of modifying the cluster state to match the description stored in git.

Both platforms support automated sync, i.e they can reconcile the cluster state automatically after a change in GitOps; and manual sync where the action is triggered directly by a human or some external service agent.

Manual Sync

Argo CD

With Argo CD, you declaratively specify manual sync by setting syncPolicy: {} on the Application GitOps resource. This way, Argo CD will detect changes, show them on the UI, etc, but will not take action to reconcile them. Instead, synchronization can be manually triggered on the web UI (which is very straightforward for beginners) or the CLI with argocd app sync.

Flux

Using Flux, automatic reconciliation is the norm, but you can opt-out of it by ‘suspending’ the GitOps resource. This can be done declaratively by setting suspend: true.

To do manual sync on-demand on a suspended GitOps resource, set the reconcile.fluxcd.io/requestedAt annotation to the current time:

kubectl annotate --field-manager=flux-client-side-apply --overwrite \
kustomization/podinfo reconcile.fluxcd.io/requestedAt="$(date +%s)"

It’s worth noting that running flux reconcile against a suspended resource will not trigger the reconciliation. Requiring a manual edit to the cluster state for this override was an intentional design choice. Essentially, on-demand, manual synchronization doesn’t have a declarative setting, so Flux doesn’t wish to support it via its CLI either.

Source Tracking

Source tracking controls how changes are detected in the GitOps resource. Both Argo CD and Flux can be configured to track a branch, a tag pattern, or a fixed commit hash in git.

Cluster Drift Reconciliation (Self Heal)

With source tracking the cluster will follow the desired state in git, but what if someone carries out a manual edit to the cluster? Cluster drift reconciliation (self heal in Argo lingo) entails resyncing the cluster state after a change outside GitOps control, e.g a manual edit with kubectl. It can ensure that the cluster adheres to the declared state (eventually). Both Argo CD and Flux support this feature with caveats.

Argo CD provides this as an optional feature, which requires automatic sync to be enabled, and it disables rollbacks.

In Flux, support varies by GitOps resource kind. For Kustomizations, cluster drift is reconciled by default, and the only way to opt-out is to annotate individual resources. On the other hand, Flux does not support this feature for Helm releases at all. (We’ll see more on these in the Helm section.) Flux does not distinguish by trigger cause, consequently ‘self-heal’ won’t be carried out if the resource is ignored or the owning GitOps resource is suspended.

Garbage Collection (Pruning)

Garbage collection controls what happens to resources getting untracked in source control. Both tools take a similar approach, exposing a setting whether they should be deleted or kept. You can also prevent garbage collection of specific resources with an annotation (Argo, Flux).

Sync Windows

There are cases when you don’t want to allow resource updates, only during a specific maintenance window.

Using Argo CD, this can be achieved with sync windows. Using sync windows, automatic or all syncs can be denied except for a certain time frame.

Flux doesn’t offer this feature, although a design has already been proposed. Currently, it can be achieved with a CronJob that resumes the resource for the duration of the maintenance window.

Selective Sync

Argo CD supports selective (or partial) syncs, i.e only selected resources get synced. Similarly to an ordinary manual sync, this can be done from the web UI or the CLI. However, selected syncs are not recorded in history and hooks are not run.

In Flux there’s no mechanism for this, however if your only use case is to ignore certain resources during reconciliation you can label them (Flux).

Hooks

Argo CD sync behavior can be customized with hooks. If you are familiar with Helm hooks, this is the same thing in essence, e.g allows you to deploy resources in a specific order, run a job (such as a database migration) or trigger a notification after the deployment. Argo CD also understands Helm hooks.

Flux doesn’t provide hooks in general, but an individual tool (such as Helm) might provide their own.

Summary

Kustomize

Kustomize is a utility for customizing application configuration in a template-free way, and is a core K8s tool shipping with kubectl. Both Argo CD and Flux supports Kustomize.

Argo CD relies on a tool detection mechanism, which checks the directory contents and uses kustomize if it finds a kustomization.yaml, kustomization.yml, or Kustomization.

Bear in mind that tool-specific settings will override the implicit behavior, which can be surprising at first.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  ...
spec:
  ...
  source:
    ...

    # Tool -> plain directory
    directory:
      recurse: false
...

The above snippet will make Argo CD detect a plain directory even in the presence of a kustomization.yaml. The same applies to Helm charts.

With Flux, the kustomize-controller operator and its kustomize.toolkit.fluxcd.io/Kustomization CRD is used to manage applications configured with Kustomize.

Configuration

Flux supports defining strategic merge and JSON patches, overriding images and the [namespaces](/blog/k8s-namespaces in the kustomize.toolkit.fluxcd.io/Kustomization resource. Argo CD is less flexible, you have to place your edits in the overlays of your kustomization (with a few exceptions). This might be a problem for certain repo layouts, e.g where kustomizations of an app live in a separate repo which is owned by a different team, and adding a new kustomization there is not preferable / feasible. If you are familiar with Helm, it’s not hard to see how this will cause a bigger problem there, but about that later. As an additional customization, Flux supports variable templating and substitution.

Summary

Helm

Helm is a popular package manager for Kubernetes applications.

Configuration

Helm values

Helm charts can be configured with values. With Flux it is possible to provide a values block with the desired configuration in the HelmRelease resource. (Similarly to Kustomization, where you provide patches). Additionally, the contents can come from ConfigMap or Secret resources deployed in the cluster, which will be merged in the same manner the Helm CLI does it for values files.

Unfortunately, specifying values inline or inside the cluster is not supported by Argo CD. Instead, the files have to be placed alongside the chart in its repo (in case the source is git) or packaged with it (in case a Helm repository is used). This method works for bespoke applications, however it is quite problematic for those off the shelf.

Helm is essentially a package manager for charts. Many off the shelf (OTS) charts are available for open-source projects and can be downloaded from the internet. So there’s a misalignment between Helm and Argo CD as an OTS chart cannot possibly contain the configuration for its users, which makes the above values file resolution mechanism useless for anything beyond reading default values. To work around this issue, one can create a wrapper, placed in git, that refers to the original as a chart dependency, and include the custom value files there. Although this is much better than copy-pasting the entire chart, it still results in boilerplate and complicates versioning.

Kustomize Helm Releases

There are cases when you would like to run Kustomize on Helm’s rendered output. For example, when the chart isn’t flexible enough and you have to override some configuration not supported by the chart, you could apply it as a patch with Kustomize. Flux supports Kustomize as a postRenderer, and can be used for this purpose. Argo CD doesn’t have this feature, however you can create a custom rendering plugin for it.

Source Tracking

Tracking changes in Helm releases with GitOps is more complicated than Kustomize. This is because there’s an additional notion of charts, handled differently by Argo CD and Flux.

Helm Charts in Helm Registries

Helm uses SemVer for versioning. Helm charts are expected to be immutable, similarly to other software packages. This means, whenever the template is changed, the chart version must be bumped.

For charts in Helm registries, both Flux and Argo CD support specifying SemVer ranges, so you may receive updates on new package versions. For example using a range of >=4.0.0 <5.0.0, your cluster will automatically receive updates for major version 4.

Helm Charts in Git

Source tracking of Helm charts works similarly to Kustomizations using both platforms, i.e they can be configured so that reconciliation tracks commits on a branch, a tag pattern, or is fixed to a commit hash.

However, there is a very important property in Flux that you should be aware of. Under the hood, Flux packages the Helm chart contained in the git repository and caches it for internal consumption by HelmReleases. By default (i.e with the ChartVersion reconcile strategy), it assumes that the chart is unchanged unless the version is different in Chart.yaml, no matter the git revision. In other words, it assumes immutable packages, even for git sources. This means that if you don’t want surprises, you should bump the Chart version on each revision that changes a template.

This behavior can be changed however by setting the reconcile strategy to Revision. This will configure Flux to append build metadata containing the git commit SHA to the version, thus reflecting every commit in a new package version.

Chart Dependencies

Both tools support chart dependencies, which are charts too and may come from different repositories altogether, so care must be taken to allow only trusted sources. Both platforms provide a way for limiting trust.

You shouldn’t use chart dependencies to define runtime ordering between applications. As detailed in the referenced article, when Helm installs the charts it renders all the chart objects, sorts all the Kubernetes objects by Kind, and then installs each Kind. This can prevent collections of charts from installing cleanly, as some charts might depend on previously installed charts with all their Kinds running. Ideally, chart dependencies should be used for libraries, as a way to extract common patterns to keep your application charts DRY.

Summary

Scaling Out

GitOps frameworks should provide appropriate abstractions to support adoption in large organizations.

Recursion

“What is the tortoise standing on?” “You’re very clever, young man, very clever,” said the old lady. “But it’s turtles all the way down!” – conversation between scientist and old lady in Stephen Hawking’s Brief History of Time

In this context, recursion means applying GitOps techniques to manage GitOps resources. For example, think of a team managing an application having multiple deployments (e.g in different environments). To keep their GitOps resources free of duplication they decide to extract common configuration with the use of kustomize patches. A straightforward way to do this is to create a parent GitOps resource that uses Kustomize to generate the inferior GitOps configurations.

In Argo CD, this can be achieved with the App of Apps pattern, which lets us define an Application resource that contains child Applications (and AppProjects). Argo CD watches the root application as well as synchronizes any application it generates. (By the way, the referenced article also points out how to do Kustomized Helm.) The child apps need not reside in the same cluster as their parent. By using the Apps of Apps pattern we can use the same techniques for generating GitOps resources as app resources, which makes it feasible to template or kustomize Applications, consequently, avoid duplication.

Similarly in Flux, we can define GitOps resources recursively. Having dedicated CRDs for sources, sync-able resources, notifications, makes it even more flexible than Argo CD.

Dependency Ordering

Applications depend on each other, and we should be able to express that to some degree. As initial approach, one can distinguish between infrastructure applications providing core functionalities such as ingress, secret management, RBAC, cert management, service mesh, etc; and product applications. For example if you use the popular service mesh linkerd, it must be in place before any applications come up, because it injects sidecars into application pods starting up. Having a way to stall the installation of product applications until the infra is ready spares us the chore of dealing with such race-conditions and other problems.

With Flux’s dependsOn, we can prevent an application to be synced unless its dependencies are in the Ready state, in other words, to guarantee installation ordering. Unfortunately, this feature is missing from Argo CD but it is proposed.

Permissions and Access Control

It makes sense to organize applications based on ownership to simplify permission setup. VCS based permission management can be put into place, e.g. CODEOWNERS can be used to control write permissions based on globs within a monorepo. A higher degree of confidentiality can be achieved if each team gets their own repo, as even read can be forbidden for outsiders. The GitOps platform can offer additional features for access control.

With Argo CD, the AppProject is used to specify that an application belongs to a project, which makes use of Argo CD’s own user management and permission system; enabling us to allow/deny

• Access to sources.
• Deploying resource kinds.
• Access for users
• Deploying to target clusters.

Flux doesn’t offer its own user management like Argo CD does. Instead, platform administrators should use Kubernetes RBAC and policy driven validation to establish security. Flux has a multi-component design, and integrates with many other systems. Having different components and CRDs such as git repositories, charts, notifications, etc, helps separate concerns, thus facilitates setting up fine-grained policies. Platform admins can enforce service account impersonation to minimize privileges.

Everything Is A CRD

Everything is on cob! The whole planet is on a cob! – Rick Sanchez, Rick, and Morty - S02E10 The Wedding Squanchers

While Argo CD offers only two major CRDs, Application, and AppProject, Flux has separate CRDs for each concept such as Kustomization (kustomize-controller), HelmRelease, HelmChart (helm-controller), HelmRepository, GitRepository (source-controller), Alert, Event (notification-controller), etc. This allows for a cleaner design, which precipitates in details such as:

• Using Flux, notification configuration is placed in CRDs, whereas for Argo CD, it is placed in ConfigMaps in the Argo CD deployment’s namespace. Access to that namespace is often restricted to the platform administrator.
• Using Flux, private repository credentials stored in a Secret should be referenced in Flux’s GitRepository, whereas for Argo CD, they should be placed in Secrets in the Argo CD deployment’s namespace. Reusing the same credential for multiple repos requires a rather strange technique. Not only Flux’s method is much more conventional and flexible, access to Argo CD’s namespace is often restricted to the platform administrator.
• Analogously to repository credentials, cluster credentials (in a multi-cluster scenario) are set up directly in the CRDs in Flux, whereas for Argo CD, its a Secret in its own namespace

Polyrepo Support

Both platforms support multiple git repos.

Multi-Cluster Deployment

Both platforms support syncing remote cluster targets.

Argo CD offers a dedicated ApplicationSet resource for templating Applications targeting multiple clusters. With Flux, you can use conventional tooling (such as kustomize overlays) to generate GitOps manifests for the separate targets.

Summary

Conclusion

In this article, we explored GitOps frameworks, focusing mainly on core capabilities, leaving out features like multi-tenancy, RBAC, notifications, image automation, etc. We found that Argo CD and Flux are comparably efficient, with each having its pros and cons. After careful deliberation, at Turbine.ai we chose Flux due to its superior support for OTS Helm charts and operational simplicity.

Also as you prioritize your organization’s specific needs when choosing between these platforms, you might also want to consider supercharging your build automation. If that’s the case, give Earthly a try!

It could be a valuable addition to your GitOps practices.