Git: scanning repositories for secrets using Gitleaks
Tags: security, git, devops, tutorial
Published: 2021-08-17 05:11:39

---
title: Git: scanning repositories for secrets using Gitleaks
published: true
date: 2021-08-16 07:17:40 UTC
tags: security,git,devops,tutorial
canonical_url: https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/
---

### Git: scan repositories for secrets using Gitleaks

![](https://cdn-images-1.medium.com/max/244/1*HAnGUQ9Zf3UKjgKfrTP6xQ.png)

A confidential data leak such as RDS keys or passwords to a Git repository, even if it is a private Github repository, is a very bad thing and it’s good to check your repositories to know if any developer pushed a commit with such data.

### **Contents**

- [Scanning utilities](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Scanning_utilities)
- [Planning](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Planning)
- [Gitleaks — manual run](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Gitleaks_-_manual_run)
- [Github token](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Github_token)
- [Jenkins job](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Jenkins_job)
- [Pipeline script](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Pipeline_script)
- [Loops in Groovy](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Loops_in_Groovy)
- [Jenkins Docker plugin](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Jenkins_Docker_plugin)
- [Ignoring errors in a Jenkins stage{}](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Ignoring_errors_in_a_Jenkins_stage)
- [Slack notifications from Jenkins](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Slack_notifications_from_Jenkins)
- [Create a Slack Bot](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Create_a_Slack_Bot)
- [Jenkins credentials](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Jenkins_credentials)
- [jenkins.plugins.slack.StandardSlackService postToSlack Response Code: 404](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#jenkinspluginsslackStandardSlackService_postToSlack_Response_Code_404)
- [File upload to Slack](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#File_upload_to_Slack)
- [Gitleaks configuration](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Gitleaks_configuration)
- [Commits to check](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Commits_to_check)
- [Gitleaks configuration file](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/#Gitleaks_configuration_file)

### Scanning utilities

To check Git repositories for a leak, at first glance there are a lot of utilities:

- [Gittyleaks](https://github.com/kootenpv/gittyleaks) — looks interesting, but the last update 2 years ago
- [Repo Supervisor](https://github.com/auth0/repo-supervisor) — has a WebUI, uses AWS Lambda, fully integrated with Github, can be checked later
- [Truffle Hog](https://github.com/trufflesecurity/truffleHog) — CLI only, looks not bad
- [Git Hound](https://github.com/ezekg/git-hound) — a plugin for git, can perform scan before commits only, but not remote repositories
- [Gitrob](https://github.com/michenriksen/gitrob) — the last update was three years ago
- [Watchtower](https://try.nightfall.ai/radar) — looks interesting, even has a WebUI, but they didn’t even post about their pricing on the website, so out of the race
- [GitGuardian](https://www.gitguardian.com/) — a really good solution, but overpriced
- [gitleaks](https://github.com/zricethezav/gitleaks) — CLI only, the one, we will use in this post

So, from the list above it’s worth trying [Truffle Hog](https://github.com/trufflesecurity/truffleHog) and [gitleaks](https://github.com/zricethezav/gitleaks), but I didn’t like the [Truffle Hog](https://github.com/trufflesecurity/truffleHog) documentation.

[Repo Supervisor](https://github.com/auth0/repo-supervisor) looks promising too, will check it in the following post.

From those two:

- Gitleaks: just a scanner — gave an URL of a repository, and it will generate a JSON report with findings
- Repo Supervisor: can be used in two ways:
- just to scan a local directory
- scan a remote repository on PullRequest/push/etc

So, for the Gitleaks we can create a cronjob in Jenkins or Kubernetes that will take a list of repositories to be checked, and then will send a report a Slack channel.

Also, Gitleaks can be used with [Github Actions](https://rtfm.co.ua/github-obzor-github-actions-i-deploj-s-argocd/), see more [here>>>](https://github.com/marketplace/actions/gitleaks), but not all our developer teams use Actions. Another way can be [pre-commit hooks](https://pre-commit.com/).

### Planning

So, for now, let’s try a solution with Jenkins, although there are various ways to run it:

- trigger a job with the [GitHub Pull Request Builder](https://rtfm.co.ua/jenkins-github-pull-request-builder-plagin/)
- trigger a job with the через _GitHub hook trigger for GITScm polling_ или _Poll SCM_
- run just as a crontask

At first, we will create a simple job running by schedule, and then will check for other solutions.

What do we have in our project:

- around 200 Github repositories
- around 10 developers teams — backend, frontend, analytics, iOS, and Android mobile applications, gaming, devops.

What we can do with Gitleaks:

- create a Jenkins job for every team
- the job will accept a parameter with a list of repositories of the team
- will create a dedicated Slack channel for every team
- once a day will run scanning and will send reports to a corresponding Slack channel

At first, let’s run Gitleaks manually to see how it’s working, and then will do an automation job.

### Gitleaks — manual run

Install it. On Arch Linux, can be installed from AUR:

```
$ yay -S gitleaks
```

#### Github token

Next, need to create a token to access a Github organization’s repositories.

Go to your Github user’s setting, create a token:

![](https://cdn-images-1.medium.com/max/1007/0*z5HqQKYM3PUAmkOt.png)

Give it repo permissions:

![](https://cdn-images-1.medium.com/max/757/0*U30huQCfTWl2qIXu.png)

And run Gitleaks with the token, a repository’s URL, add `--verbose`, save results to a file:

```
$ gitleaks — access-token=ghp_C6h***3z5 — repo-url=https://github.com/example/BetterBI — verbose — report=analytics-repo.json
…\
INFO[0036] scan time: 32 seconds 756 milliseconds 672 microseconds\
INFO[0036] commits scanned: 1893
WARN[0036] leaks found: 111
```

Check the report:

```
$ less analytics-repo.json
```

And an example from the findings:

```
...
 {
  "line": " "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADA***CCaM=\n-----END PRIVATE KEY-----\n\",",
  "lineNumber": 5,
  "offender": "-----BEGIN PRIVATE KEY-----",
  "offenderEntropy": -1,
  "commit": "0f047f0cca3994b3465821ef133dbd3c8b55ee7a",
  "repo": "BetterBI",
  "repoURL": "https://github.com/example/BetterBI",
  "leakURL": "https://github.com/example/BetterBI/blob/0f047f0cca3994b3465821ef133dbd3c8b55ee7a/adslib/roas_automation/example-service-account.json#L5",
  "rule": "Asymmetric Private Key",
  "commitMessage": "DT-657 update script for subs and add test\n\nDT-657 create test for check new json (add new subs)",
  "author": "username",
  "email": "example@users.noreply.github.com",
  "file": "adslib/roas_automation/example-service-account.json",
  "date": "2021-05-11T19:46:46+03:00",
  "tags": "key, AsymmetricPrivateKey"
 },
...
```

Here:

- `line`: what exactly was found
- `offender`: a rule, triggered for this finding
- `commit`: a commit ID with the secret

Finding are performed with regex expressions, described in the [`default.go`](https://github.com/zricethezav/gitleaks/blob/master/config/default.go).

Also, you can create your own configuration file and pass it to Gitleaks.

For example, the private RSA key above was found by the [Asymmetric Private Key](https://github.com/zricethezav/gitleaks/blob/master/config/default.go#L76:) rule:

```
[[rules]]
    description = "Asymmetric Private Key"
    regex = '''-----BEGIN ((EC|PGP|DSA|RSA|OPENSSH) )?PRIVATE KEY( BLOCK)?-----'''
    tags = ["key", "AsymmetricPrivateKey"]
```

So, we can create a dedicated config file for each team or repository and pass them via Kubernetes ConfigMap or as a file in a Jenkins job.

### Jenkins job

Now, when we’ve seen how Gitleaks can be started, let’s add a Jenkins job to run it periodically.

#### Pipeline script

So, for each team, we will create a dedicated Jenkins job that will have a parameter with a repositories list of the team.

#### Loops in Groovy

Some time ago I did a similar solution using Golang, check the [Go: checking public repositories list in Github. Go slices comparison](https://rtfm.co.ua/en/go-checking-public-repositories-list-in-github-go-slices-comparison-the-first-golang-experience/) post for details, and there it was a bit simpler to run a loop over the list. With Groovy, had to google a bit.

Create a new Jenkins job, set its type to the Pipeline:

![](https://cdn-images-1.medium.com/max/510/0*TRwXCexnLwbqUaCa.png)

In the job’s setting, create a string parameter with a list of the team’s repositories, here is only two used:

![](https://cdn-images-1.medium.com/max/662/0*eTpbukwOv5iN-csY.png)

Next, go to the Jenkins script.

Set a variable named `$repos_list`, that will accept an environment variable `$TEAM_REPOS`, and then by using the `split()` method divide the lists' objects.

Then, by using the for loop integrate over them:

![](https://cdn-images-1.medium.com/max/417/0*JeI3FAhIUJP8X0aH.png)

```
node('master') {

  def repos_list = "${env.TEAM_REPOS}".split(',')

  for (repo in repos_list) {
    println repo
  }

}
```

Run the job:

![](https://cdn-images-1.medium.com/max/603/0*xMZoYTsrcHszPcG7.png)

#### Jenkins Docker plugin

Our default approach to run Jenkins builds is by using a Docker container to keep the hosts’ system clean.

Add another parameter with the _Password_ type, save Github token here:

![](https://cdn-images-1.medium.com/max/1024/0*-WkWkCFpOXEF5Tln.png)

By using the [Jenkins Docker Plugin](https://www.jenkins.io/doc/book/pipeline/docker/) create a Docker container with Gitleaks, pass the token, URL, and report’s file. Pay attention, that the report’s file will contain a repository’s name:

```
node('master') {
      
  def repos_list = "${env.TEAM_REPOS}".split(',')

  for (repo in repos_list) {
    stage("Repository ${repo}") {
      docker.image('zricethezav/gitleaks').inside('--entrypoint=""') {
        sh "gitleaks --access-token=${GITHUB_TOKEN} --repo-url=https://github.com/example/${repo} --verbose --report=analytics-${repo}-repo.json"
      }
    }
  }
}
```

Here, for every repository name from the `repos_list` list, we will create a dedicated Jenkins Pipeline Stage that also will use a repository's name.

Run and check:

![](https://cdn-images-1.medium.com/max/728/0*Iq7bk1hwj-HkFMe5.png)

Um… And here is an issue: scan will be stopped right after the first findings in the first scanned repo, as Geatleaks found a leak, returned the exit 1 code, and the job was immediately stopped:

![](https://cdn-images-1.medium.com/max/596/0*HOgijuZyQccU__hW.png)

#### Ignoring errors in a Jenkins stage{}

To solve it, we can use the `try/catch` solution: each stage will be running in its `try`, in case of errors, we will catch them with `catch`, and will proceed with the build:

```
node('master') {
  
  def repos_list = "${env.TEAM_REPOS}".split(',')
  def build_ok = true
  
  for (repo in repos_list) {
    try {
      stage("Repository ${repo}") {
        docker.image('zricethezav/gitleaks').inside('--entrypoint=""') {
          sh "gitleaks --access-token=${GITHUB_TOKEN} --repo-url=https://github.com/example/${repo} --verbose --report=analytics-${repo}-repo.json"
        }
      }
    } catch(e) {
        currentBuild.result = 'FAILURE'
    }
  }
}
```

Run it:

![](https://cdn-images-1.medium.com/max/596/0*WSWdiUY6wtZKKHGU.png)

Good — now all stages are running despite a previous stage result.

#### Slack notifications from Jenkins

The next step for us is to configure sending alarms to a Slack workspace.

Let’s use the [Slack Notification](https://plugins.jenkins.io/slack) plugin for this. See its documentation [here>>>](https://github.com/jenkinsci/slack-plugin/blob/master/README.md#pipeline-job).

#### Create a Slack Bot

Go to the [Slack Apps](https://api.slack.com/apps), create a new application:

![](https://cdn-images-1.medium.com/max/974/0*bmBHa4LnE9ZFDK1w.png)

![](https://cdn-images-1.medium.com/max/703/0*Y6kXVoXdJXcW4Mo3.png)

Go to the _Permissions_:

![](https://cdn-images-1.medium.com/max/699/0*xfKNX3hvaL8hg0_p.png)

Add the following:

- `files:write`
- `chat:write`

![](https://cdn-images-1.medium.com/max/699/0*k18TwkF3aB5jaelG.png)

Go to the _OAuth & Permissions_, install the bot to the Slack workspace:

![](https://cdn-images-1.medium.com/max/699/0*xudz38w7t5sc7XjX.png)

Save the token:

![](https://cdn-images-1.medium.com/max/694/0*Lf3zwxHUtSkjM8T5.png)

#### Jenkins credentials

Add the token to the Jenkins — go to the _Manage Jenkins > Manage Credentials_:

![](https://cdn-images-1.medium.com/max/978/0*fIMIvOjaoXCwz9vO.png)

Add a new one:

![](https://cdn-images-1.medium.com/max/649/0*2myxnmt5b1T4tHZm.png)

Set its type to the _Secret file_:

![](https://cdn-images-1.medium.com/max/781/0*wMEpXnY9VEC52xQc.png)

In the Slack workspace, create a new channel:

![](https://cdn-images-1.medium.com/max/545/0*MNUBm62-getKzLKB.png)

Invite the bot to the channel:

![](https://cdn-images-1.medium.com/max/543/0*3NIwbtz6ojgccV9o.png)

![](https://cdn-images-1.medium.com/max/510/0*7pb09_eW8u5boLeV.png)

Add a new function to the Jenkins script  — ` notifySlack()`, and run it from the `catch{}` to send alarms if any secrets were found during scan:

```
def notifySlack(String buildStatus = 'STARTED') {

    // Build status of null means success.
    buildStatus = buildStatus ?: 'SUCCESS'

    def color
    //change for another slack chanel
    def token = 'gitleaks-slack-bot'

    if (buildStatus == 'STARTED') {
        color = '#D4DADF'
    } else if (buildStatus == 'SUCCESS') {
        color = '#BDFFC3'
    } else if (buildStatus == 'UNSTABLE') {
        color = '#FFFE89'
    } else {
        color = '#FF9FA1'
    }

    def msg = "${buildStatus}: `${env.JOB_NAME}` #${env.BUILD_NUMBER}:\n${env.BUILD_URL}"
    slackSend(color: color, message: msg, tokenCredentialId: token, channel: "#devops-alarms-gitleaks-analytics")
}

node('master') {
  
  def repos_list = "${env.TEAM_REPOS}".split(',')
  
  for (repo in repos_list) {
    try {
      stage("Repository ${repo}") {
        docker.image('zricethezav/gitleaks').inside('--entrypoint=""') {
          sh "gitleaks --access-token=${GITHUB_TOKEN} --repo-url=https://github.com/example/${repo} --verbose --report=analytics-${repo}-repo.json"
        }
      }   
    } catch(e) {
        currentBuild.result = 'FAILURE'
        notifySlack(currentBuild.result)
    } 
  }     
}
```

#### jenkins.plugins.slack.StandardSlackService postToSlack Response Code: 404

Run the build, and get the following error:

> 12:42:39 ERROR: Slack notification failed. See Jenkins logs for details.

Check Jenkin’s logs on the _https://<JENKINS_URL>/log/all_:

![](https://cdn-images-1.medium.com/max/535/0*Btpx4crhL8K9pUyh.png)

Go to the _Manage Jenkins > Configure System_, find Slack plugin’s options, set the _Custom slack app bot user_:

![](https://cdn-images-1.medium.com/max/350/0*0Ne4gw8GqtvrTRfp.png)

The credentials here are default, we are overriding them from the pipeline.

In the _Advanced_ remove the _Override URL,_ if it was set:

![](https://cdn-images-1.medium.com/max/1024/0*ooonPJHTWtrPpkwF.png)

Run again and now everything is working:

![](https://cdn-images-1.medium.com/max/598/0*htXa6-fm87t8LoDL.png)

#### File upload to Slack

Now, let’s add a report file upload with findings to the message in the Slack channel by using the `slackUploadFile()` function:

```
def notifySlack(String buildStatus = 'STARTED', reportFile) {

    // Build status of null means success.
    buildStatus = buildStatus ?: 'SUCCESS'

    def color
    //change for another slack chanel
    def token = 'gitleaks-slack-bot'

    if (buildStatus == 'STARTED') {
        color = '#D4DADF'
    } else if (buildStatus == 'SUCCESS') {
        color = '#BDFFC3'
    } else if (buildStatus == 'UNSTABLE') {
        color = '#FFFE89'
    } else {
        color = '#FF9FA1'
    }

    def msg = "${buildStatus}: `${env.JOB_NAME}` #${env.BUILD_NUMBER}:\n${env.BUILD_URL}"
    slackSend(color: color, message: msg, tokenCredentialId: token, channel: "#devops-alarms-gitleaks-analytics")
    slackUploadFile(credentialId: token, channel: "#devops-alarms-gitleaks-analytics", filePath: "${reportFile}")
}

node('master') {
  
  def repos_list = "${env.TEAM_REPOS}".split(',')
  
  for (repo in repos_list) {
    try {
      stage("Repository ${repo}") {
        docker.image('zricethezav/gitleaks').inside('--entrypoint=""') {
          sh "gitleaks --access-token=${GITHUB_TOKEN} --repo-url=https://github.com/example/${repo} --verbose --report=analytics-${repo}-repo.json"
        }
      }   
    } catch(e) {
        currentBuild.result = 'FAILURE'
        notifySlack(currentBuild.result, "analytics-${repo}-repo.json")
    } 
  }     
}
```

Channel here can be moved to the job’s parameter later.

Here, in the `notifySlack()` we've added another parameter - the `reportFile`, and then during the `notifySlack()` function call, we are passing the report's file as a second argument to the function.

Run the job, check the Slack channel:

![](https://cdn-images-1.medium.com/max/1024/0*TLS3NXve1xRkI4K6.png)

And the final thing is to set a schedule to run the job:

![](https://cdn-images-1.medium.com/max/369/0*QcwvaPYPM1S7adbT.png)

### Gitleaks configuration

#### Commits to check

At this moment, Gitleasks will perform a full scan of the repository — all commits, all history.

If we will run it every day, then each day we will get messages about some old problematic commits.

As a way to mitigate it, we can create two jobs: in the first job, we will do a full scan, and in the second one will perform a kind of incremental scan for changes, made during the last 24 hours.

I.e. the “incremental” job will be run daily at 12:00 pm when all developers are in the office, and the job will check commits for the last day only.

To do so, Gitleaks has the `--commit-since` option. Let's add a new variable called `yesterday` with yesterday's date taken by the `previous()` method of the `Date()` class, and then this date will be passed to the `--commit-since`:

```
...

node('master') { 
  
  def repos_list = "${env.TEAM_REPOS}".split(',')
  def yesterday = new Date().format( 'yyyy-MM-dd' ).previous()

  println yesterday
  
  for (repo in repos_list) {
    try {
      stage("Repository ${repo}") {
        docker.image('zricethezav/gitleaks').inside('--entrypoint=""') {
          sh "gitleaks --access-token=${GITHUB_TOKEN} --repo-url=https://github.com/example/${repo} --verbose --report=analytics-${repo}-repo.json --commit-since=${yesterday}"
        }
      }
    } catch(e) {
        currentBuild.result = 'FAILURE'
        notifySlack(currentBuild.result, "analytics-${repo}-repo.json")
    }
  }
}
```

#### Gitleaks configuration file

Another thing is to create a dedicated rules file for the Gitleaks.

This can be done with the `--repo-config-path`, and in each repository, we can add its own configuration file.

Add some default rules there, plus I’d like to check for passwords passed as plaintext to commits:

```
...

[[rules]]
    description = "Plaintext password"
    regex = '''(?i)pass*[a-z]{5}[:|=]? +["|'](.*)["|']'''
    tags = ["password", "PlainTextPassword"]

[allowlist]
    description = "Allowlisted files"
    files = ['''^\.?gitleaks.config$''']
```

With the `(?i)pass*[a-z]{5}[:|=]? +["|'](.*)["|']` regular expression, we are looking for a string started with _pass_, then for a ":" or "=" symbol, then it can contain or not a space, then a quote mark, then any text, and a quote mark again.

Seems must be working:

![](https://cdn-images-1.medium.com/max/572/0*QtapRptZYdxZh8Jk.png)

Save it to the repository as `.github/gitleaks.config`, and in the job add another parameter by using this file:

```
...
        docker.image('zricethezav/gitleaks').inside('--entrypoint=""') {
          sh "gitleaks --access-token=${GITHUB_TOKEN} --repo-url=https://github.com/example/${repo} --verbose --report=analytics-${repo}-repo.json --commit-since=${yesterday} --repo-config-path=.github/gitleaks.config"
        }
...
```

That’s all for now.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/git-scanning-repositories-for-secrets-using-gitleaks/)_._

* * *