How to apply security at the source using GitOps
Tags: gitops, kubernetes, devops, security
Published: 2022-07-27 09:53:19

<p>
If your <strong>GitOps deployment model has security issues</strong> (for example, a misconfigured permission because of a typo), <strong>this will be propagated until it is hopefully discovered at runtime</strong>, where most of the security events are scanned or found.
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-00.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-00.png" alt="" width="1200" height="661" class="aligncenter size-full wp-image-52514" /></a>
</p>
<p>
<em>What if you can fix potential security issues in your infrastructure at the source?</em>
</p>
<p>
Let's start with the basics.
</p>


<h2 id="git">What is Git?</h2>
<p>
<strong><a rel="noopener nofollow noreferrer" target="_blank" href="https://git-scm.com/">Git</a> is an open source distributed version control system</strong>. It tracks changes made in files (usually text files such as source code) <strong>allowing and fostering a collaborative work model</strong>. It is the<em> de facto </em>standard in version control systems nowadays.
</p>
<p>
You can have your own git repo locally on your laptop, host it on your own server, or use some provider such as GitLab or GitHub.
</p>
<p>
There are different “flows” on how to manage a repository (git-flow, github-flow, etc.), but a basic example on how git is used is something like this: Changes in the files are “committed” by users by “forking” the repository and doing the proper changes in a “branch”.
</p>
<p>
Then, the user creates a request (either "pull request", "merge request", or just "send a patch") to include those changes in the repository.
</p>
<p>
After that, usually a discussion happens between the “owner” and the user creating the request, and if everything goes fine the change is accepted and added to the repository.
</p>
<p>
NOTE: If you want to know more, <a rel="noopener nofollow noreferrer" target="_blank" href="https://martinfowler.com/bliki/PullRequest.html">here</a> is much more detailed information about the git pull request mechanism.
</p>
<p>
To see a real world example, just browse your favorite open source GitLab or GitHub repository and browse the Pull Request (or Merge Request) tab (or see <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/danielmiessler/SecLists/pull/155">this</a> for a fun one). You can see the proposed changes, comments, labels, who proposed the changes, tools running validations against the proposed changes, notifications sent to people watching the repository, etc.
</p>


<h2 id="gitops">What is GitOps?</h2>
<p>
To put it simply, <strong>GitOps is just a methodology that uses a git repository as the single source of truth for your software assets</strong> so you can leverage the git deployment model (pull requests, rollbacks, approvals, etc.) to your software.
</p>
<p>
There are books (<a rel="noopener nofollow noreferrer" target="_blank" href="https://developers.redhat.com/e-books/path-gitops">The Path to GitOps</a>, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.manning.com/books/gitops-and-kubernetes">GitOps and Kubernetes</a> or <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.gitops.tech/">GitOps Cloud-native Continuous Deployment</a>), <a rel="noopener nofollow noreferrer" target="_blank" href="https://go.weave.works/automating-kubernetes-with-gitops-wp.html">whitepapers</a>, and <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/weaveworks/awesome-gitops">more</a> <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.weave.works/technologies/gitops/">blog posts</a> <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.weave.works/blog/gitops-operations-by-pull-request">than</a> we <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.redhat.com/en/topics/devops/what-is-gitops">can</a> <a target="_blank" href="https://sysdig.com/blog/gitops-k8s-security-configwatch/" rel="noopener">manage</a> <a rel="noopener nofollow noreferrer" target="_blank" href="https://about.gitlab.com/blog/2022/04/07/the-ultimate-guide-to-gitops-with-gitlab/">to</a> <a rel="noopener nofollow noreferrer" target="_blank" href="https://cloud.google.com/kubernetes-engine/docs/tutorials/gitops-cloud-build">count</a> but let us elaborate on the GitOps purpose by taking a quick look on how things evolved in the last few years.
</p>
<p>
Before the cloud, adding a new server to host your application took weeks. You had to ask for permissions, purchase it, and perform a lot of manual tasks. Then, virtualization made things much easier. You request a virtual machine with some specs and after a few minutes, you have access to it.
</p>
<p>
Then, the cloud. Requesting servers, network, storage, and even databases, messaging queues, containers, machine learning stuff, serverless… is just an API call away! You request it and a few seconds later, you get it, just like that. You just need to pay for what you use. This also means the infrastructure can be managed as code performing API calls… and where do you store your code? In a git repository (or any other content version system).
</p>
<p>
The GitOps term was coined back in <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.weave.works/blog/the-history-of-gitops">2017 by Weaveworks</a>, and paraphrasing <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/open-gitops/documents/blob/main/PRINCIPLES.md">OpenGitOps</a>, a GitOps system is based on the following principles:
</p>
<ul>
<li><strong>Declarative</strong>: it defines “what.”
</li><li><strong>Versioned and immutable</strong>: hence “git.”
</li><li><strong>Pulled automatically</strong>: an agent observes the desired state and the changes happening in the code.
</li><li><strong>Continuously reconciled</strong>: did someone mention Kubernetes?
</li>
</ul>
<p>
The <strong>essence of the GitOps methodology is basically a Kubernetes controller or controllers</strong> (or agents) running on your cluster that observes the Kubernetes objects running on top of it (defined by a <a rel="noopener nofollow noreferrer" target="_blank" href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CustomResource</a>) <strong>comparing the current state against the state specified in the Git repo</strong>. If it doesn't match, it <em>remediates</em> the application by applying the manifests found in the repository.
</p>
<p>
NOTE: There are slightly different approaches to GitOps, for example, push vs. pull, how to handle the configuration management, etc. Those are advanced topics, but for now, let's stick to the basics.
</p>
<p>
The following diagram shows a simplified GitOps system:
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-01.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-01.png" alt="GitOps diagram showing a developer sending changes, the GitOps process and the agent deployed on Kubernetes observing the changes" width="1999" height="1132" class="aligncenter size-full wp-image-52515 u-no-shadow" /></a>
</p>
<ul>
<li>A code change is submitted to the Git repository by the user.
</li><li>Then, a process is triggered on the repository to incorporate those changes into the application itself, including running automation tools against that new code to validate it.
</li><li>Once everything is in place, the GitOps agent running in the Kubernetes cluster, which is observing the repository, performs the reconciliation between the desired state (the code in the repository) and the current state (the objects running on the Kubernetes cluster itself).
</li>
</ul>
<p>
Being based on Git means frictionless for developers. They don't need to worry about a new tool to interact with, but rather apply the same practices used to manage the code in the Git repository.
</p>
<p>
Speaking about GitOps tools, there are a few already available, including open source tools such as <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.cncf.io/projects/flux/">Flux</a> or <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.cncf.io/projects/argo/">ArgoCD</a>, both <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.cncf.io/projects/">CNCF incubating projects</a>.
</p>
<p>
To get a feeling on what an application definition looks like via GitOps, this is an example of a simple application (stored in a GitHub repository) managed by Flux or ArgoCD.
</p>
<p>
With Flux:
</p>
<pre>---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: my-example-app
  namespace: hello-world
spec:
  interval: 30s
  ref:
    branch: master
  url: https://github.com/xxx/my-example-apps.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-example-app
  namespace: hello-world
spec:
  interval: 5m0s
  path: ./myapp
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-example-app
  targetNamespace: hello-world
</pre>
<p>
With ArgoCD:
</p>
<pre>apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-example-app
  namespace: hello-world
spec:
  destination:
    namespace: my-example-app
    server: https://kubernetes.default.svc
  project: default
  source:
    path: myapp/
    repoURL: https://github.com/xxx/my-example-apps.git
    targetRevision: HEAD
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
</pre>
<p>
Both reference the Git repository where the application manifests are stored (Deployments), the NameSpaces, and a few more details.
</p>


<h2 id="gitops-vs-iac">GitOps vs. IaC</h2>
<p>
<strong>Infrastructure as Code</strong> is a methodology of treating the building blocks of your infrastructure as code using different techniques and tools. This means that instead of manually creating your infrastructure such as VMs, containers, networks, or storage via your favorite infrastructure provider web interface manually, you define them as code, and then those are created/updated/managed by the tools you choose, such as <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.terraform.io/">terraform</a>, <a rel="noopener nofollow noreferrer" target="_blank" href="https://crossplane.io/">crossplane</a>, or <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.pulumi.com/">pulumi</a>, among others.
</p>
<p>
The benefits are huge. You can manage your infrastructure as if it was code (it <em>is</em> code now) and leverage your development best practices (automation, testing, traceability, versioning control, etc.) to your infrastructure assets. In fact, there is a trend of using "Infrastructure as Software" as a term instead because it is much more than just code.
</p>
<p>
There is tons of information out there on this topic, but the <a target="_blank" href="https://sysdig.com/learn-cloud-native/cloud-security/what-is-infrastructure-as-code-iac-security/" rel="noopener">following resource</a> is a good starting point.
</p>
<p>
As you have probably figured, GitOps leverages Infrastructure as Code as the declarative model to define the infrastructure. In fact, IaC is one of the GitOps cornerstones! But it is much more as IaC doesn't mandate the rest of the GitOps principles.
</p>


<h2 id="gitops-vs-devops">GitOps vs. DevOps</h2>
<p>
There are lots of definitions of the "DevOps" term. It depends who you ask but to put it simply, "DevOps is the combination of practices and tools to build and deliver software reducing friction and to a high speed."
</p>
<p>
DevOps methodologies can leverage GitOps as GitOps provides a framework that matches DevOps practices but it is not strictly necessary.
</p>


<h2 id="noops">What about NoOps?</h2>
<p>
NoOps was coined by Forrester in <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.forrester.com/blogs/11-02-07-i_dont_want_devops_i_want_noops/">2011</a> and it is a radical approach to handling operations where the IT environment is abstracted and automated to the point there is no need to manage it manually.
</p>
<p>
GitOps helps to reduce the manual changes by remediating those with the desired state in the Git repository, but <strong>applying a real NoOps to the whole IT environment is an aspirational goal rather than a real goal as of today</strong>.
</p>


<h2 id="kubernetes">Is GitOps just for Kubernetes?</h2>
<p>
No. Kubernetes, the <a rel="noopener nofollow noreferrer" target="_blank" href="https://kubernetes.io/docs/concepts/architecture/controller/#controller-pattern">controller pattern</a>, and the <a rel="noopener nofollow noreferrer" target="_blank" href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/">declarative model</a> to define Kubernetes objects are a perfect match for a GitOps methodology, but it doesn't mean GitOps methodologies cannot be applied without Kubernetes. There are a few challenges if using GitOps outside of Kubernetes, such as handling the idempotency, the deletion/creation of the assets, secrets managements, etc. But <strong>the GitOps principles can be applied without Kubernetes</strong> (and applying a little bit of creativity).
</p>


<h2 id="security">GitOps &amp; Security</h2>
<p>
Let's talk about the security aspects now. Most security tools detect potential vulnerabilities and issues at runtime (too late). In order to fix them, either a reactive manual process needs to be performed (e.g., modifying directly a parameter in your k8s object with kubectl edit) or ideally, the fix will happen at source and will be propagated all along your supply chain. This is what is called “<a target="_blank" href="https://sysdig.com/blog/shift-left-secure-software-delivery/" rel="noopener">Shift Security Left</a>”. From fixing the problem when it is too late to fixing it before it happens.
</p>
<p>
This doesn't mean every security issue can be fixed at the source, but adding a security layer directly at the source can prevent some issues.
</p>
<p>
First of all, the general security recommendations apply.
</p>
<ul>
<li>Reduce the attack surface
</li><li>Encrypt the secrets (using External Secrets or Sealed Secrets)
</li><li>Network segmentation
</li><li>RBAC
</li><li>Keep software up to date
</li><li>Enforce least privilege
</li><li>Monitor &amp; measure
</li><li>…
</li>
</ul>
<p>
Let's see a few scenarios where the GitOps methodology can improve your security in general:
</p>
<ul>
<li><strong>Avoid/Refuse manual changes (avoiding Drift)</strong>. The Git repository is the source of truth. If you try to modify the application definition, the GitOps tool will revert those changes by applying the version stored in the Git repository.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-02.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-02.png" alt="GitOps UI showing the differences between the object running on the cluster and the definition stored in the Git repository" width="1714" height="892" class="aligncenter size-full wp-image-52516" /></a>
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-03.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-03.png" alt="GitOps UI showing the remediation performed because of the differences" width="1714" height="294" class="aligncenter size-full wp-image-52517" /></a>
</p>
<ul>
<li><strong>Rollback changes</strong>. Imagine you introduced a potential security issue in a specific commit by modifying some parameter in your application deployment. Leveraging the Git capabilities, you can rollback the change if needed directly at the source and the GitOps tool will redeploy your application without user interaction.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-04.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-04.png" alt="GitOps UI showing the reverted changes" width="1719" height="593" class="aligncenter size-full wp-image-52518" /></a>
</p>
<ul>
<li><strong>Fast response.</strong> If you find you are using a vulnerable container image in your application (e.g., MariaDB), you just need to create a PR to update the tag into the deployment file and the GitOps tool will use the new tag in a new deployment.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-05.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-05.png" alt="Detail of the deployment performed by the GitOps tool shown in the GitOps UI" width="1719" height="593" class="aligncenter size-full wp-image-52519" /></a>
</p>
<ul>
<li><strong>Traceability.</strong> Using Git capabilities, you can easily check when a file was changed, the changes themselves and the user that promoted the changes. You've got an audit log <em>for free</em>.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-06.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-06.png" alt="GitHub UI showing a log of the committed changes" width="1484" height="435" class="aligncenter size-full wp-image-52520" /></a>
</p>
<ul>
<li><strong>Disaster recovery.</strong> Again, the Git repository is the source of truth. If you need to redeploy your application because something happened, the definition is there (of course you need to have a disaster recovery plan for other stuff such as the data itself).
</li><li><strong>Access Control.</strong> You can apply different permissions to different users on your Git repositories and even policies such as “only merge a change after two positive reviews”.
</li>
</ul>
<p>
Those benefits are good enough to justify using GitOps methodologies to improve your security posture and they came out of the box, but GitOps is a combination of a few more things. We can do much more. GitHub, GitLab, and other Git repositories providers allow you to run <em>actions</em> or <em>pipelines</em> based on the changes you perform in your Git repository, including by a pull request, so the possibilities are endless. A few examples:
</p>
<ul>
<li><strong>Linting.</strong> The definition of the application <em>is code</em>, what if the definition is checked for wrong syntax, missing parameters, and more? There are tools (such as the <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/oxsecurity/megalinter">megalinter</a>) that can be executed against the changes you've performed so you avoid surprises later on.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-07.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-07.png" alt="Megalinter output showing the tests executed, alerts and details" width="1561" height="1208" class="aligncenter size-full wp-image-52521" /></a>
</p>
<ul>
<li><strong>Testing.</strong> You can even automate the <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/marketplace/actions/kind-kubernetes-in-docker-action">creation of a temporary Kubernetes cluster</a> to run your application with the changes and run some tests against it.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-08.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-08.png" alt="GitHub UI showing a GitHub action log deploying a Kind Kubernetes cluster" width="1719" height="1304" class="aligncenter size-full wp-image-52522" /></a>
</p>
<ul>
<li><strong>Vulnerability scanning</strong>. By checking the <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/sysdiglabs/secure-inline-scan-examples/tree/main/github">container images</a> you are using for vulnerabilities before they are deployed in your environment.
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-09.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-09.png" alt="GitHub UI showing a detailed view of policies checking the code directly in the pull-request" width="1713" height="1185" class="aligncenter size-full wp-image-52523" /></a>
</p>
<ul>
<li><strong>Policy-as-code.</strong> Leveraging <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/open-policy-agent/setup-opa">OPA</a>, you can even apply policies to your manifests to check for potential issues or custom policies
</li>
</ul>
<p>
<a href="https://sysdig.com/wp-content/uploads/Security-source-GitOps-10.png"><img src="https://sysdig.com/wp-content/uploads/Security-source-GitOps-10.png" alt="GitHub UI showing the output of a GitHub action running OPA policies against the change performed" width="1719" height="1304" class="aligncenter size-full wp-image-52524" /></a>
</p>


<h2 id="conclusion">Final thoughts</h2>
<p>
The <strong>GitOps methodology brings a few improvements to the deployment model</strong> and <strong>security benefits to the table without having to add another tool</strong>.
</p>
<p>
<strong>It improves the security posture by adding a “shift left” layer directly to the source code</strong> and thanks to the flexibility of the pull-request model, you can easily add extra security checks without affecting or modifying the runtime.
</p>