Protect secrets in Git with the clean/smudge filter
Tags: git, security, productivity
Published: 2024-01-26 02:39:18

<p>When working on public <a href="/topics/gitops">Git</a> repositories, you need to pay close attention so that you don't accidentally push secret information such as tokens, private server addresses, personal email addresses, and the like. One of the tools that can help you is Git's clean/smudge filter.</p>

<h2>Clean and smudge your Git repository&nbsp;</h2>

<p>The clean/smudge filter is quite simple to use. Create a filter driver with two commands—<code>clean</code> and <code>smudge</code>—and then apply the filter per record in the <code>.gitattributes</code> file.</p>

<p>This makes the filter stand between the working directory and the staging area for the specific <code>.gitattributes</code> record. When adding content from the working directory to the staging area with <code>git add</code>, files that match the <code>.gitattributes</code> record will go through the <code>clean</code> filter. When pulling content back into the working directory with <code>git pull</code>, those same files will go through the <code>smudge</code> filter.</p>

<h2>Create the filter driver</h2>

<p>Let's create a filter driver named <code>cleanPass</code> that uses <code>sed</code> expressions to replace the value <code>secretpassword</code> with the value <code>hiddenpassword</code> and vice versa:</p>

```bash
# mac users should use gsed instead of sed
git config filter.cleanPass.clean "sed -e 's/secretpassword/hiddenpassword/g'"
git config filter.cleanPass.smudge "sed -e 's/hiddenpassword/secretpassword/g'"
```

<p class="Indent1"><strong>Tip</strong>: Add <code>--global</code> to the <code>config</code> command to make the filter driver available globally to all repositories at <code>~/.gitconfig</code> instead of locally at <code>.git/config</code>.</p>

<p>As an example, let's apply the <code>cleanPass</code> filter driver to the JSON file type. In the repository's <code>.gitattributes</code>, look for the JSON file type record and modify it. The result should look something like this, depending on the original configuration:</p>

<pre>
<code>*.json text eol=lf filter=cleanPass
</code></pre>

<p class="Indent1"><strong>Tip</strong>: To avoid pushing, this can be done in <code>.git/info/attributes</code>, which takes precedence over <code>.gitattributes</code>. Just be sure not to mess up any of the original record configurations like <code>eol</code>.</p>

<p>From now on, every new or modified JSON file in the repository will go through the <code>cleanPass</code> filter. Figure 1 provides an animated look at what happens next.</p>

<drupal-media data-caption="Figure 1. The clean filter in action." data-entity-type="media" data-entity-uuid="81d56a84-e2be-4fc5-b6c9-4fd6495e3eba"></drupal-media>

<h2>One filter to rule them all</h2>

<p>That was cool, but using this filter can get quite cumbersome at times. For instance, on a recent task, I worked with multiple repositories using multiple (but identical) services from my team's <a href="/products/openshift/overview">Red Hat OpenShift</a> lab. To add to the complexity, I usually work from two different computers—my desktop at my office and my laptop at home.</p>

<p>That all means that I needed to create multiple filters (or one complex one) and apply them to multiple file types, for multiple repositories, on multiple computers. That gave me a headache, especially when the fact hit me that this workflow will probably reoccur in future tasks.</p>

<p>Eventually, I decided to create a very simple script that handles multiple value replacements in both directions, save it on each of my computers, and configure it as a global filter.</p>

<h3>Configure a global filter</h3>

<p>Adding values to be replaced in this script is super easy, and attaching it to <code>.gitattributes</code> records, as seen above, is quite simple as well.</p>

<p>First, create a script in a path available to all repositories. For instance, you could put it in your home directory at <code>~/scripts/git-smudge-clean-filter.sh</code>:</p>

```bash
#!/bin/bash

declare -A mapArr

mapArr["my-work-private-server.mywork.com"]="&lt;reducted-work-server&gt;"
mapArr["my-personal-private-server.myowndomain.org"]="&lt;reducted-personal-server&gt;"
mapArr["A*&amp;#QAADDA(77##F"]="super-secret-token"
mapArr["oops@mypersonal.email"]="support@correct.email"

# mac users should use gsed instead of sed
sedcmd="sed"
if [[ "$1" == "clean" ]]; then
  for key in ${!mapArr[@]}; do
    sedcmd+=" -e \"s/${key}/${mapArr[${key}]}/g\""
  done  
elif [[ "$1" == "smudge" ]]; then
  for key in ${!mapArr[@]}; do
    sedcmd+=" -e \"s/${mapArr[${key}]}/${key}/g\""
  done  
else  
  echo "use smudge/clean as the first argument"
  exit 1
fi

eval $sedcmd
```

<p>Add as many pairs as you like to <code>mapArr</code>.</p>

<p class="Indent1"><strong>Note:</strong> Per the nature of Git's filter driver, you can't use the same key or value more than once.</p>

<p>Next, create the filter driver globally (note the script argument):</p>

```bash
git config --global filter.reductScript.smudge "~/scripts/git-smudge-clean-filter.sh smudge"
git config --global filter.reductScript.clean "~/scripts/git-smudge-clean-filter.sh clean"
```

<p>Now, for every <code>.gitattributes</code> record to which you apply the <code>filter=reductScript</code> configuration in any repository, every file matching this record will go through the script, and every value specified in <code>mapArr</code> will be reducted (or returned).</p>

<h2>Conclusion</h2>

<p>That's all there is to it. I hope you find this useful. We'll cover other Git tips in future articles.</p>

<p>In the meantime, check out the following Git resources:</p>

<ul>
	<li><a href="https://gist.github.com/TomerFi/0911f573ea0474b9ab74bcfcef0f2a49">My GitHub gist</a></li>
	<li><a href="https://git-scm.com/docs/gitattributes">gitattributes documentation</a></li>
	<li><a href="https://git-scm.com/docs/git-config">git-config documentation</a></li>
	<li><a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-filterltdrivergtclean">Git filter.&lt;driver&gt;.clean command</a></li>
	<li><a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-filterltdrivergtsmudge">Git filter.&lt;driver&gt;.smudge command</a></li>
	<li><a href="https://developers.redhat.com/cheat-sheets/git/old">Git Cheat Sheet</a></li>
</ul>