helmfile: difference between sync and apply (Helm 3)
Tags: helm, kubernetes, devops
Published: 2021-10-11 16:58:04

[Helm](https://helm.sh/) and [helmfile](https://github.com/roboll/helmfile) are great tools to automate kubernetes deployments. However, they have some subtleties that are sometimes hard to understand and may lead to catastrophic problems. One of them is the difference between `helmfile sync` and `helmfile apply`, a question raised many times, for example in [StackOverflow](https://stackoverflow.com/questions/59703760/helmfile-sync-vs-helmfile-apply).

Running `helmfile -h`, the explanation of those two commands is:
> `sync` → sync all resources from state file (repos, releases and chart deps)
> `apply` → apply all resources from state file only when there are changes

But what does it *mean* exactly ? What are the differences and pitfalls ? Let's dive in together, starting at the basics of Helm 3 up to helmfile.

*Note*: if you are familiar with how Helm 3 upgrades work, you can skip directly to the last section.

## Helm states

The first thing to understand is how Helm stores *states*.

Helm generates the Kubernetes manifests to apply to a Kubernetes cluster by "compiling" a Chart's templates against some values, that can come from the chart's `values.yaml` or the values override (defined in helmfile, passed using the `--set` option of the cli, etc.). All those information together (chart, values, options) are what we will call a *state*.

Whenever you install a release, Helm stores this state in a secret called (the `v1` suffix being the *revision*):
```ruby
sh.helm.release.v1.{RELEASE_NAME}.v1
```
This secret is simply a compressed, base64-encoded JSON stored in a single key - `release` -, which contains *everything* needed to reconstruct exactly the helm chart, and to re-apply it with *exactly* the same values to reconstruct the release.

It can be inspected using the following command (see [this gist](https://gist.github.com/DzeryCZ/c4adf39d4a1a99ae6e594a183628eaee)):
```bash
kubectl get secret sh.helm.release.v1.<RELEASE_NAME>.v<REV> \
  -o jsonpath='{.data.release}' | base64 -d | base64 -d | gzip -d
```

{% collapsible Content of the helm-release secret %}
If you decode this secret, you'll see a JSON that contains:
* name, namespace, and version of the release
* list of all chart files (name + base64 content of all files, excluding `templates/*`, `Chart.yaml` and `values.yaml`)
* metadata (content of `Chart.yaml`)
* values (content of `values.yaml`)
* templates files (name + base64 content)
* config (value overrides via cmd or helmfile)
* values schema (content of `values.schema.json`)
* hooks
* info → current state of the release in Kubernetes (e.g. "*install complete*"), dates of first/last deployment, etc.
* actual Kubernetes manifest (output of all rendered templates, this time in plain text)
{% endcollapsible %}

When you upgrade a release, the new state is stored in a new secret, with the version incremented to the new revision: 
```ruby
sh.helm.release.v1.{RELEASE_NAME}.v{REVISION}
```

## How helm 3 upgrade/rollback works

Now, let's understand how Helm decides what to do during an upgrade.

> ⚠️ *Every time you run `helm upgrade` or `helm rollback`, a new revision (and secret) is always created, whether or not there are changes*. 

#### Helm 2 and two-way merge

Back in **Helm 2**, the upgrade process was plain and simple: helm reconstructed the *old state* by decoding the helm-release secret of the current revision, and compared it with the *desired state* to create the different patches to apply. This is known as **two-way merge**:
```
old state → desired state
```
The *desired state* can be reconstructed from a helm-release secret (rollback), or from a new version of the chart + values (upgrade).

The important point is that **Helm 2 didn't take the *live state* into account**, that is, what is effectively present in the cluster. In other words, if you modified anything manually (add a value to a ConfigMap, or a sidecar container in a deployment), this change was not seen at all by Helm, and could either be left as-is, disappear, or be overwritten depending on Helm old/desired states.

#### Helm 3 and three-way strategic merge

**Helm 3** introduced a brand new way of computing patches required for upgrades and rollbacks, known as **three-way strategic merge**.
The article [Three-way merging: A look under the hood](https://blog.plasticscm.com/2016/02/three-way-merging-look-under-hood.html), gives a good explanation of what three-way merging means (heavily used in git), while Helm doc's section [Improved Upgrade Strategy: 3-way Strategic Merge Patches](https://helm.sh/docs/faq/changes_since_helm2/#improved-upgrade-strategy-3-way-strategic-merge-patches) focuses more on what it means in Helm.

But simply put, **Helm 3 now takes the *live state* into account**: 
```
(old state → desired state) → (live state → desired state)
```

The rules are (fields = key+value):
1. `+` (add) → new fields in the *desired state* not present in the *old state* are added (overwriting any *live state*)
2. `⌫` (remove) → fields existing in the *old state* that are not present in the *desired state* are removed (even if their value changed in the *live state*)
3. `±` (overwrite) → fields in the *live state* that are also present in the *desired state* but have a different value are updated (whatever the *old state*)
4. `∅` (ignore) → the rest is left unchanged (e.g. new fields in the *live state* stay)

Moreover, the patches do *merge* operations, meaning maps are deep-merged (vs completely replaced). This is a huge improvement from Helm 2. Among others, it means that in Helm 3:

* it is possible to add a sidecar container to a deployment manually, or a new data entry in a ConfigMap. If they are not managed by Helm (no fields in the generated manifests about it), they will stay unchanged after upgrade/rollback;
* if you modify fields managed by Helm manually, doing a rollback will effectively reset the fields to the Helm values.

{% collapsible Simple example %}
Let's say you install a deployment with Helm with the following labels (manifest stripped for readability):
```yaml
apiVersion: apps/v1
kind: Deployment
# ...
spec:
  # ...
  template:
    # ...
    metadata:
      labels:
        label-1: install
        label-2: install
        label-3: install
    # ...
```
Now, you change the labels manually to:
```yaml
labels:
  label-1: manual
  label-2: manual
  label-3: manual
  new-one: added   # also add one
```
And finally do a Helm upgrade, with the new values being:
```yaml
labels:
  label-1: install  # no change
  label-2: upgrade  # change
                    # delete
  label-4: upgrade  # add
```

The result will be:
```yaml
labels:
  label-1: install  # overwritten
  label-2: upgrade  # overwritten
                    # deleted
  label-4: upgrade  # added
  new-one: added    # ignored/kept
```
{% endcollapsible %}

## Helmfile: sync vs apply

Now that we understand how helm upgrades work, let's dive into helmfile sync vs apply.

The `helmfile sync` command will **run `helm upgrade` on all releases**. This means all releases will have their revision incremented by one. However, as Helm does three-way strategic merges, if there is no change between the live and desired state, no patch will actually be applied: there is just a new helm-release secret created.

The `helmfile apply` command will **run `helm upgrade` only when there are changes**. However, **to detect changes**, helmfile uses the [helm-diff](https://github.com/databus23/helm-diff) plugin, which only computes the difference between the *old* vs *desired state*; it **doesn't look at the _live state_** (similar to Helm 2). If something changed outside of Helm, helm-diff will return *"no change"*, and the release won't be upgraded.

In other words, `apply` has the advantage of not creating useless new revisions, but doesn't guarantee the coherency of the *live state*, as manual changes may go undetected. `sync` is exactly the opposite: it always creates new revisions for *all* releases, but will detect and undo any manual change that happened on Helm-managed fields.

---------
**UPDATE (May 2022)**: helm-diff added support for three way merge diffs on [v3.3.0](https://github.com/databus23/helm-diff/tree/v3.3.0) (January 10, 2022). As the helm-diff process inherits environment from the helmfile process, one can do:
```bash
# use three-way merge strategy for diffing
HELM_DIFF_THREE_WAY_MERGE=true helmfile apply
``` 
This way, `helmfile apply` can now detect and change manual changes as well.
---------
 
`sync` or `apply` is thus down to a trade-off, which is alleviated if you decide to never change anything manually (or always use three-way merge diffs). If you stick with this best practice, `apply` is always the way to go.

-----------

Written with ❤ by [derlin](https://github.com/derlin)