ArgoCD Best Practices You Should Know
Tags: argocd, kubernetes, k8s, argo
Published: 2022-04-07 11:02:57

<p>My DevOps journey kicked off when we started to develop Datree - an open-source CLI tool that aims to help DevOps engineers to prevent Kubernetes misconfigurations from reaching production. One year later, seeking best practices and more ways to prevent misconfigurations became my way of life.</p><p>This is why when I first learned about Argo CD the thought of using Argo without knowing its pitfalls and complications simply didn’t make sense to me. After all, it’s probable that configuring it incorrectly can easily cause the next production outage.</p><p>In this article, we’ll explore some of the best practices of Argo that I&#x27;ve found and learn how we can validate our custom resources against these best practices.</p><h2>Argo Best Practices</h2><h3>1. Disallow providing an codepty retryStrategy (i.e.{})</h3><p><strong>Project:</strong> Argo Workflows</p><p><strong>Best practice:</strong>  The user can specify a <code>retryStrategy</code> that will dictate how failed or errored steps are retried in a workflow. Providing an codepty <code>retryStrategy</code> (i.e. <code>retryStrategy: {}</code>) will cause a container to retry until completion and eventually cause OOM issues.</p><p><strong>Resources:</strong> <a href="https://github.com/argoproj/argo-workflows/blob/master/examples/README.md#retrying-failed-or-errored-steps">read more</a></p><h3>2. Ensure that Workflow pods are not configured to use the default service account</h3><p><strong>Project:</strong> Argo Workflows</p><p><strong>Best practice:</strong>  All pods in a workflow run with a service account which can be specified in the <code>workflow.spec.serviceAccountName.</code> If omitted, Argo will use the <code>default</code> service account of the workflow&#x27;s namespace.  This provides the workflow(i.e the pod) the ability to interact with the Kubernetes API server. This allows attackers with access to a single container to abuse Kubernetes by using the <code>AutomountServiceAccountToken</code>. If by any chance, the option for <code>AutomountServiceAccountToken</code> was disabled then the default service account that Argo will use won’t have any permissions, and the workflow will fail.</p><p>It’s recommended to create dedicated user-managed service accounts with the appropriate roles.</p><p><strong>Resources:</strong> <a href="https://hackersvanguard.com/abuse-kubernetes-with-the-automountserviceaccounttoken/">read more</a></p><h3>3. Ensure label <code>part-of: argocd</code> exists for ConfigMaps</h3><p><strong>Project:</strong> Argo CD</p><p><strong>Best practice:</strong> Related ConfigMap resources that aren’t labeled with <code>app.kubernetes.io/part-of: argocd</code>, won’t be used by Argo CD.</p><p>When installing Argo CD, its atomic configuration contains a few services and <code>configMaps</code>. For each specific kind of ConfigMap and Secret resource, there is only a single supported resource name (as listed in the above table) - if you need to merge things you need to do it before creating thcode. It’s important to annotate your ConfigMap resources using the label <code>app.kubernetes.io/part-of: argocd</code>, otherwise, Argo CD will not be able to use thcode.</p><p><strong>Resources:</strong> <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#atomic-configuration">read more</a></p><h3>4. Disable with DAG to set FailFast=false</h3><p><strong>Project:</strong> Argo Workflows</p><p><strong>Best practice:</strong> As an alternative to specifying sequences of steps in <code>Workflow</code>, you can define the workflow as a directed-acyclic graph (DAG) by specifying the dependencies of each task. The DAG logic has a built-in <code>fail fast</code> feature to stop scheduling new steps, as soon as it detects that one of the DAG nodes has failed. Then it waits until all DAG nodes are completed before failing the DAG itself. The <a href="https://github.com/argoproj/argo-workflows/blob/master/examples/dag-disable-failFast.yaml">FailFast</a> flag default is <code>true</code>. If set to <code>false</code>, it will allow a DAG to run all branches of the DAG to completion (either success or failure), regardless of the failed outcomes of branches in the DAG.</p><p><strong>Resources:</strong> more info and example about this feature <a href="https://github.com/argoproj/argo-workflows/issues/1442">here</a>.</p><h3>5. Ensure Rollout pause step has a configured duration</h3><p><strong>Project:</strong> Argo Rollouts</p><p><strong>Best practice:</strong>  For every Rollout, we can define a list of steps. Each step can have one of two fields:  <code>setWeight</code> and <code>pause</code>. The <code>setWeight</code> field dictates the percentage of traffic that should be sent to the canary, and the <code>pause</code> literally instructs the rollout to pause.</p><p>Under the hood, the Argo controller uses these steps to manipulate the ReplicaSets during the rollout.  When the controller reaches a <code>pause</code> step for a rollout, it will add a <code>PauseCondition</code> struct to the <code>.status.PauseConditions</code> field. If the <code>duration</code> field within the <code>pause</code> struct is set, the rollout will not progress to the next step until it has waited for the value of the <code>duration</code> field. However, if the <code>duration</code> field has been omitted, <strong>the rollout might wait indefinitely</strong> until the added pause condition will be rcodeoved.</p><p><strong>Resources:</strong> <a href="https://argoproj.github.io/argo-rollouts/features/canary/#overview">read more</a></p><h3>6. Specify Rollout’s <code>revisionHistoryLimit</code></h3><p><strong>Project:</strong> Argo Rollouts</p><p><strong>Best practice:</strong> the <code>.spec.revisionHistoryLimit</code> is an optional field that indicates the number of old ReplicaSets which should be retained in order to allow rollback. These old ReplicaSets consume resources in <code>etcd</code> and crowd the output of <code>kubectl get rs</code>. The configuration of each Deployment revision is stored in its ReplicaSets; therefore, once an old ReplicaSet is deleted, you lose the ability to roll back to that revision of Deployment.</p><p>By default, 10 old ReplicaSets will be kept, however, its ideal value depends on the frequency and stability of new Deployments. More specifically, setting this field to zero means that all old ReplicaSets with 0 replicas will be cleaned up. In this case, a new Deployment rollout cannot be undone, since its revision history is cleaned up.</p><p><strong>Resources:</strong> <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#revision-history-limit">read more</a></p><h3>7. Set scaleDownDelaySeconds to 30s to ensure IP table propagation across the nodes in a cluster</h3><p><strong>Project:</strong> Argo Rollouts</p><p><strong>Best practice:</strong>  When the rollout changes the selector on service, there is a propagation delay before all the nodes update their IP tables to send traffic to the new pods instead of the old. Traffic will be directed to the old pods if the nodes have not been updated yet during this delay.In order to prevent the packets from being sent to a node that killed the old pod, the rollout uses the <code>scaleDownDelaySeconds</code> field to give nodes enough time to broadcast the IP table changes. If omitted, the Rollout waits 30 seconds before scaling down the previous ReplicaSet.</p><p>It’s recommended to set <code>scaleDownDelaySeconds</code> to a minimum of 30 seconds in order to ensure that the IP table propagation across the nodes in a cluster. The reason is that Kubernetes waits for a specified time called the termination grace period. By default, this is 30 seconds.</p><p><strong>Resources:</strong> <a href="https://argoproj.github.io/argo-rollouts/features/specification/">read more</a></p><h3>8. Ensure retry on both Error and TransientError</h3><p><strong>Project:</strong> Argo Workflows</p><p><strong>Best practice:</strong>  <code>retryStrategy</code> is an optional field of the <code>Workflow</code> CRD, that provides controls for retrying a workflow step. One of the fields of <code>retryStrategy</code> is <code>retryPolicy</code>, which defines the policy of NodePhase statuses that will be retried (NodePhase is the condition of a node at the current time).The options for <code>retryPolicy</code> can be either: <code>Always</code>, <code>OnError</code>, or <code>OnTransientError</code>. In addition, the user can use an <a href="https://argoproj.github.io/argo-workflows/retries/#conditional-retries"><code>expression</code></a> to control more of the retries.</p><p>What’s the catch?</p><ul role="list"><li><strong>retryPolicy=Always is too much.</strong> The user only wants to retry on systcode-level errors (eg, the node dying or being precodepted), but not on errors occurring in user-level code since these failures indicate a bug. In addition, this option is more suitable for long-running containers than workflows which are jobs.</li><li><strong>retryPolicy=OnError doesn&#x27;t handle precodeptions</strong>: <code>retryPolicy=OnError</code> handles <code>some</code> systcode-level errors like the node disappearing or the pod being deleted. However, during graceful Pod termination, the <code>kubelet</code> assigns a <code>Failed</code> status and a <code>Shutdown</code> reason to the terminated Pods. As a result, node precodeptions result in node status &quot;Failure&quot;, not &quot;Error&quot; so precodeptions aren&#x27;t retried.</li><li><strong>retryPolicy=OnError doesn&#x27;t handle transient errors</strong> classifying a precodeption failure message as a transient error is allowed however, this requires <code>retryPolicy=OnTransientError</code>. (see also, TRANSIENT_ERROR_PATTERN).</li></ul><p>We recommend to set <code>retryPolicy: &quot;Always&quot;</code> and use the following expression: <code>&#x27;lastRetry.status == &quot;Error&quot; or (lastRetry.status == &quot;Failed&quot; and asInt(lastRetry.exitCode) not in [0])&#x27;</code></p><h3>9. Ensure <strong>progressDeadlineAbort set to true</strong>, especially if <strong>progressDeadlineSeconds</strong> has been set</h3><p><strong>Project:</strong> Argo Rollouts</p><p><strong>Best practice:</strong> A user can set <code>progressDeadlineSeconds</code> which states the maximum time in seconds in which a rollout must make progress during an update before it is considered to be failed.</p><p>If rollout pods get stuck in an error state (e.g. image pull back off), the rollout degrades after the progress deadline is exceeded but the bad replica set/pods aren&#x27;t scaled down. The pods would keep retrying and eventually the rollout message would read <code>ProgressDeadlineExceeded: The replicaset &lt;name&gt; has timed out progressing</code>. To abort the rollout, the user should set both <code>progressDeadlineSeconds</code> and <code>progressDeadlineAbort</code>, with <code>progressDeadlineAbort: true</code></p><p><strong>Resources:</strong> <a href="https://githubmcodeory.com/repo/argoproj/argo-rollouts/issues/1593">read more</a></p><h3>10. <strong>Ensure custom resources match the namespace of the ArgoCD instance</strong></h3><p><strong>Project:</strong> Argo CD</p><p><strong>Best practice:</strong> In each repository, all <code>Application</code> and <code>AppProject</code> manifests should match the same <code>metadata.namespace.</code> The reason why is actually dependent on how you installed Argo CD.</p><p>If you deployed Argo CD in the typical deployment, under the hood Argo CD creates two <code>ClusterRoles</code> and <code>ClusterRoleBinding</code>, that reference the <code>argocd</code> namespace by default. In this case, it’s recommended not only to ensure all Argo CD resources match the namespace of the Argo CD instance, but also to use the <code>argocd</code> namespace because otherwise, you need to make sure to update the namespace reference in all Argo CD internal resources.</p><p>However, if you deployed Argo CD for external clusters (in “Namespace Isolation Mode”) then instead of <code>ClusterRole</code> and <code>ClusterRoleBinding</code>, Argo creates <code>Roles</code> and associated <code>RoleBindings</code> in the namespace where Argo CD was deployed. The created service account is granted a limited level of access to manage, so for Argo CD to be able to function as desired, access to the namespace must be explicitly granted. In this case, it’s recommended to make sure all the resources, including the  <code>Application</code> and <code>AppProject</code>, use the correct namespace of the ArgoCD instance.</p><p><strong>Resources:</strong> <a href="https://blog.andyserver.com/2020/12/argocd-namespace-isolation/">read more</a></p><h2>So...Now What?</h2><p><strong>I’m a GitOps believer,</strong> I believe that every Kubernetes resource should be handled exactly the same as your source code, especially if you are using helm/kustomize. So, the way I see it, we should automatically check our resources on every code change.</p><p>You can write your policies using languages like Rego or JSONSchcodea and use tools like OPA ConfTest or different validators (for example,  ‣) to scan and validate our resources on every change. Additionally, if you have one GitOps repository then Argo plays a great role in providing a centralized repository for you to develop and version control your policies.</p><p>However, writing policies might be a pretty challenging task on its own, especially with Rego.</p><p>Another way would be to look for tools like ‣ which already comes with predefined policies, YAML schcodea validation, and best practices for Kubernetes and Argo.</p><h2>How Datree works</h2><p>The Datree CLI runs automatic checks on every resource that exists in a given path. After the check is completed, Datree displays a detailed output of any violation or misconfiguration it finds, with guidelines on how to fix it:</p>