Implement DevSecOps to Secure your CI/CD pipeline
Tags: security, devops, devsecops, cicd
Published: 2022-09-28 06:16:37

Before understanding DevSecOps, let’s understand what is DevOps. DevOps is the combination of cultural philosophies, practices, and tools that increase an organization’s ability to deliver applications and services at high velocity.

In fast-moving projects, security often lags behind and given low priority which may lead to buggy code and hacks. Let’s see how we can reduce the risk of attack by integrating security in our DevOps pipeline.

## What is to DevSecOps (DevOps + Security)
[DevSecOps](/devsecops-consulting-services/) is a cultural approach where every team and person working on an application considers security throughout it's lifecycle. It ensures that security is implemented at every stage of the application software development lifecycle (SDLC) by incorporating required security checks embedded into [CI/CD automation](/ci-cd-consulting/) using appropriate tools.

For example, let's see how the DevSecOps process can detect and prevent zero-day vulnerabilities like [log4j](https://www.wsj.com/articles/what-is-the-log4j-vulnerability-11639446180). Using [Syft](https://github.com/anchore/syft) tool, we can generate SBOM for our application code and pass this SBOM report to [Grype](https://github.com/anchore/grype) which can detect these new vulnerabilities and report to us if there is any fix or patch available. As these steps are part of our CI/CD, we can alert our developers and security team to remediate this issue as soon as it is identified.

### What are the benefits of using DevSecOps?

-   Finds vulnerability and bugs at an earlier stage of development
-   Streamlined compliance
-   Speedy recovery
-   Secure supply chain
-   Cost saving
-   Can include AI-based monitoring for detecting anomalies
-   Reduces the risk of surface attack and increases confidence
-   Full visibility of potential threats and possible ways to remediate it


## How to make security culture your default state?

> Unless you’ve included security in every employee’s onboarding, creating a widespread security culture mindset will be challenging. Employees will need to think differently, behave differently, and eventually turn those changes into habits so that security becomes a natural part of their day-to-day work.

Read more about [Gitlab's DevSeOps security culture](https://about.gitlab.com/blog/2020/07/15/security-culture-devsecops).

### What does DevSecOps CI/CD pipeline look like?

![DevSecOps CI/CD Pipeline](https://d33wubrfki0l68.cloudfront.net/81fbed5f6b2223cbb02b799bbd6216860c2d0c32/e0ecd/assets/img/blog/devsecops-pipeline/devsecops-pipeline-diagram.png)

### Jenkins DevSecOps pipeline

![Jenkins DevSecOps Pipeline](https://d33wubrfki0l68.cloudfront.net/b5b451ecc262c62223800bfc611831e9f79dea74/de2dd/assets/img/blog/devsecops-pipeline/jenkins-devsecops-pipeline.png)

In this post, we will cover the following standard CI/CD stages and how to secure them:

1. **[Plan/Design](#1-plandesign)**
2. **[Develop](#2-develop)**
3. **[Build and Code analysis](#3-build-and-code-analysis)**
4. **[Test](#4-test)**
5. **[Deploy](#5-deploy)**
6. **[Monitor and Alert](#6-monitoring-and-alerting)**

Alright, let's dive into how to implement DevSecOps! 

## 1. Plan/Design
In this stage, we outline where, how, and when integration, deployment, and security testing will be done.

### 1.1 Threat modeling:

It effectively puts you in the mindset of an attacker and allows us to see the application through the attacker's eyes and block their attack before they get a chance to do anything about it. We can use [OWASP threat modeling](https://owasp.org/www-community/Threat_Modeling) or [Simple questions method from Microsoft](https://docs.microsoft.com/en-us/security/compass/applications-services#simple-questions-method) to design our threat modeling. We can also use [OWASP Threat Dragon](https://owasp.org/www-project-threat-dragon/) and [Cairis](https://github.com/cairis-platform/cairis) open source threat modeling tools to create threat model diagrams for our secure development lifecycle.

### 1.2 Secure SDLC
> A Secure SDLC requires adding security testing at each software development stage, from design to development, to deployment, and beyond. Examples include designing applications to ensure that your architecture will be secure, as well as including security risk factors as part of the initial planning phase.  
> [- Snyk Secure SDLC](https://snyk.io/learn/secure-sdlc)

In a secure software development cycle, we should separate our development, testing, and production environments and also have authorization processes that control the deployment promotion from one environment to another. This reduces the risk of a developer making unauthorized changes. This ensures that any modifications pass through a standard approval process.

## 2. Develop
The Development stage starts with writing code and we can use shift-left security [best practice](https://spectralops.io/resources/how-to-choose-a-secret-scanning-solution-to-protect-credentials-in-your-code/) which incorporates security thinking in the earliest stages of development. For example:

-   Install linting tools inside the code editor like [Visual Studio Code](https://code.visualstudio.com/). One of the most popular linting tools is [SonarLint](https://marketplace.visualstudio.com/items?itemName=SonarSource.sonarlint-vscode). Which highlights bugs and security vulnerabilities as you write code.
-   Use [Pre-commit hooks](https://github.com/pre-commit/pre-commit) to prevent adding any secrets to code.
-   Setup [Protected branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/about-protected-branches) and code reviews process.
-   Sign [git commit with GPG key](https://docs.gitlab.com/ee/user/project/repository/gpg_signed_commits/).
-   Always verify the downloaded binary/file hash.
-   Enable 2-factor authentication.

## 3. Build and code analysis

Before building, we need to scan our code for vulnerabilities and secrets. By doing static code analysis, it may detect a bug or a possible overflow in code and these overflows lead to a memory leak, which degrades the system performance by reducing the amount of memory available for each program. Sometimes it can be used as an attack surface by hackers to exploit the data.

### 3.1 Scan for secrets and credentials

[detect-secret](https://github.com/Yelp/detect-secrets) is an enterprise-friendly tool for detecting and preventing secrets in the code base. We can also scan the non-git tracked files. There are other tools as well like [Gitleaks](https://github.com/zricethezav/gitleaks) which also provide similar functionality.

```bash
detect-secrets scan test_data/ --all-files
```

### 3.2 Software Bill of Materials (SBOM)

[SBOM](https://www.linuxfoundation.org/tools/the-state-of-software-bill-of-materials-sbom-and-cybersecurity-readiness/) lets us Identify all software components, libraries, and modules that are running in our environment, even their dependencies. It speeds up response time for new vulnerabilities - including zero-day vulnerabilities like Log4j.

We can use below tools for SBOM report.

#### 3.2.1 Syft with Grype and Trivy
[Syft](https://github.com/anchore/syft) tool gives container image and filesystem SBOM result in [CycloneDX](https://github.com/CycloneDX) open source format which can be shared easily. Syft also supports cosign attestations for verifying legit images.

```bash
syft nginx:latest -o cyclonedx-json=nginx.sbom.cdx.json
```

So we have generated an SBOM report that shows what libs and modules are running in our software. Now, let's scan for vulnerabilities in SBOM reports using [Grype](https://github.com/anchore/grype).

```bash
[root@laptop ~]# grype sbom:./nginx.sbom.cdx.json | head
 ✔ Vulnerability DB    	[no update available]
 ✔ Scanned image       	[157 vulnerabilities]
NAME          	INSTALLED          	FIXED-IN      	TYPE  VULNERABILITY 	SEVERITY   
apt           	2.2.4                                		deb   CVE-2011-3374 	Negligible  
bsdutils      	1:2.36.1-8+deb11u1                   	deb   CVE-2022-0563 	Negligible  
coreutils     	8.32-4+b1                            		deb   CVE-2017-18018	Negligible  
coreutils     	8.32-4+b1          	(won't fix)   	deb   CVE-2016-2781 	Low    	 
curl          	7.74.0-1.3+deb11u1                   	deb   CVE-2022-32208	Unknown	 
curl          	7.74.0-1.3+deb11u1                   	deb   CVE-2022-27776	Medium 	 
curl          	7.74.0-1.3+deb11u1 	(won't fix)   	deb   CVE-2021-22947	Medium 	 
curl          	7.74.0-1.3+deb11u1 	(won't fix)   	deb   CVE-2021-22946	High   	 
curl          	7.74.0-1.3+deb11u1 	(won't fix)   	deb   CVE-2021-22945	Critical  

# Or we can directly use Grype for SBOM scanning
grype nginx:latest
```

**Note:** - Many of the vulnerabilities that are scanned by SCA tools are neither exploitable nor fixable via regular updates. curl and glibc are some examples. So these tools show them as not fixable or won't fix.

The latest version of [Trivy](https://github.com/aquasecurity/trivy#highlights) can also generate SBOM reports, but it's mostly used for finding vulnerabilities in containers and filesystems.

#### 3.2.2 OWASP Dependency-Check
[OWASP Dependency-Check](https://owasp.org/www-project-dependency-check/) a Software Composition Analysis (SCA) tool that attempts to detect publicly disclosed vulnerabilities contained within a project's dependencies. It does this by determining if there is a Common Platform Enumeration (CPE) identifier for a given dependency. If found, it will generate a report linking to the associated CVE entries. We can also publish our SBOM report to [Dependency-Track](https://www.infracloud.io/blogs/manage-vulnerabilities-dependency-track/) and visualize our software components and their vulnerabilities.

```bash
dependency-check.sh --scan /project_path
```

Once we know what type of vulnerabilities are present in our software, we can patch them and make our application safe and secure.

### 3.3 Static Application Security Testing (SAST)

It's a method of debugging code without running the program. It analyzes the code based on predefined rule sets. 

[SonarQube](https://github.com/SonarSource/sonarqube) allows all developers to write cleaner and safer code. It supports lots of programming languages for scanning (Java, Kotlin, Go, JavaScript). It also supports running unit testing for code coverage. It can be easily integrated with Jenkins and Azure DevOps. [Checkmarx](https://checkmarx.com/product/cxsast-source-code-scanning/), [Veracode](https://www.veracode.com/products/binary-static-analysis-sast), and [Klocwork](https://www.perforce.com/products/klocwork) also provide similar functionality but these are paid tools.

```bash
docker run \
    --rm \
    -e SONAR_HOST_URL="http://${SONARQUBE_URL}" \
    -e SONAR_LOGIN="AuthenticationToken" \
    -v "${YOUR_REPO}:/usr/src" \
    sonarsource/sonar-scanner-cli
```
Source: [Running SonarScanner from the Docker image](https://docs.sonarqube.org/latest/analysis/scan/sonarscanner).

### 3.4 Unit test

In Unit tests, individual software code components are checked if it is working as expected or not. Unit tests isolate a function or module of code and verify its correctness. We can use tools like [JaCoCo](https://github.com/jacoco/jacoco) for Java and [Mocha](https://mochajs.org/), and [Jasmine](https://jasmine.github.io/) for NodeJS to generate unit test reports. We can also send these reports to SonarQube which shows us code coverage and the percentage of your code covered by your test cases.

Once SAST is done, we can scan our Dockerfile as well.

### 3.5 Dockerfile static scanning

Always scan the Dockerfile for vulnerabilities as while writing Dockerfile we may miss some of the [best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) which may lead to vulnerable containers. To name a few common mistakes that we can avoid.
  1.  Do not use the `latest` docker image tag
  2.  Ensure that a user for the container has been created

[Checkov](https://github.com/bridgecrewio/checkov) or [docker scan](https://docs.docker.com/engine/scan/#get-a-detailed-scan-report) can be used to scan Dockerfile which follows best practice rules to write Dockerfile.

```bash
docker run -i -v $(pwd):/output bridgecrew/checkov -f /output/Dockerfile -o json
```

After building a container image, we scan it for vulnerabilities and sign our container image.

### 3.6 Container image scan

Scanning images gives the security state of the container images and let us take actions that result in a more secure container image. We should avoid installing unnecessary packages and use a multistage method. This keeps the image clean and safe. Scanning of images should be done in both development and production environments. 

Below are a few well-known open-source and paid tools that we can use for container scanning:

-   **Open source:** [Trivy](https://github.com/aquasecurity/trivy), [Gryp](https://github.com/anchore/grype) and [Clair](https://github.com/quay/clair) are widely used open source tools for container scanning.
-   **[Docker scan](https://docs.docker.com/engine/scan/#how-to-scan-images):** It uses [Snyk](https://snyk.io/product/container-vulnerability-management/) as the backend engine for scanning. It can also scan Dockerfile.
-   **[Aqua scan](https://www.aquasec.com/products/container-security/):** Provides container image scanning but it has one unique feature [Aqua DTA (Dynamic Threat Analysis)](https://blog.aquasec.com/dynamic-container-analysis) for containers which monitors behavioral patterns and Indicators of Compromise (IoCs) such as malicious behavior and network activity, to detect container escapes, malware, cryptocurrency miners, code injection backdoors and additional threats.

```bash
trivy image nginx:latest
# OR
docker scan nginx:latest
```

### 3.7 Container image signing and verifying 

If the container build process is compromised, it makes users vulnerable to accidentally using the malicious image instead of the actual container image. Signing and verifying the container always ensure we are running the actual container image.

Using [distroless](https://github.com/GoogleContainerTools/distroless) images not only reduces the size of the container image it also reduces the surface attack. The need for container image signing is because even with the distroless images there is a chance of facing some security threats such as receiving a malicious image. We can use [cosign](https://github.com/sigstore/cosign) or [skopeo](https://github.com/containers/skopeo) for container signing and verifying. You can read more about [securing containers with Cosign and Distroless Images in this blog](/blogs/secure-containers-cosign-distroless-images/).
 

```bash
cosign sign --key cosign.key custom-nginx:latest
cosign verify -key cosign.pub custom-nginx:latest
```

### 3.8 Container image validation test

Adding an extra layer of security on the container image to verify if it is working as expected and has all required files with correct permissions. We can use [dgoss](https://github.com/aelsabbahy/goss/tree/master/extras/dgoss) to do validation tests of container images.

For example, let's do a validation test for the nginx image that is running on port 80, has internet access, and verifies the correct file permission of `/etc/nginx/nginx.conf`, and the nginx user shell in the container.

```bash
dgoss edit nginx
goss add port 80
goss add http https://google.com
goss add file /etc/nginx/nginx.conf
goss add user nginx

# Once we exit it will copy the goss.yaml from the container to the current directory and we can modify it as per our validation.
# Validate
[root@home ~]# dgoss run -p 8000:80 nginx
INFO: Starting docker container
INFO: Container ID: 5f8d9e20
INFO: Sleeping for 0.2
INFO: Container health
INFO: Running Tests
Port: tcp:80: listening: matches expectation: [true]
Port: tcp:80: ip: matches expectation: [["0.0.0.0"]]
HTTP: https://google.com: status: matches expectation: [200]
File: /etc/nginx/nginx.conf: exists: matches expectation: [true]
File: /etc/nginx/nginx.conf: mode: matches expectation: ["0644"]
File: /etc/nginx/nginx.conf: owner: matches expectation: ["root"]
File: /etc/nginx/nginx.conf: group: matches expectation: ["root"]
User: nginx: uid: matches expectation: [101]
User: nginx: gid: matches expectation: [101]
User: nginx: home: matches expectation: ["/nonexistent"]
User: nginx: groups: matches expectation: [["nginx"]]
User: nginx: shell: matches expectation: ["/bin/false"]
Total Duration: 0.409s
Count: 13, Failed: 0, Skipped: 0
INFO: Deleting container
```

We can also use [kgoss](https://github.com/aelsabbahy/goss/tree/master/extras/kgoss) to do validation tests on pods.

Till now we have built and scanned the container image but before deploying let's test and scan the deployment or Helm chart.

## 4. Test

Testing ensure that the application is working as expected and has no bug or vulnerabilities.

### 4.1 Smoke test

[Smoke tests](https://www.geeksforgeeks.org/smoke-testing-software-testing/) are small but check critical components and functionality of the application. When implemented, It runs on every application build to verify critical functionality passes before integration and end-to-end testing can take place which can be time-consuming. Smoke tests help create fast feedback loops that are vital to the software development life cycle.

For example, in a smoke test, we can run the curl command on API to get the HTTP response code and latency.

### 4.2 API testing

Today's applications might expose hundreds of highly valuable endpoints that are very appealing to hackers. Ensuring your APIs are secure before, during, and after production is crucial. Hence we need to test our APIs.

API Testing reports what type of authentication is required and whether sensitive data is encrypted over HTTP and SQL injections allowing you to bypass the login phase.

We can use [Jmeter](https://jmeter.apache.org/usermanual/get-started.html#non_gui), [Taurus](https://github.com/Blazemeter/taurus), [Postman](https://learning.postman.com/docs/integrations/ci-integrations/#configuring-ci-integration), and [SoapUI](https://www.soapui.org/tools/soapui/) tools for API testing. Below is a small example using Jmeter where `test.jmx` contains the API test cases.

```bash
jmeter -n --t test.jmx -l result.jtl
```

### 4.3 Dynamic application security testing (DAST)

DAST is a web application security test that finds security issues in the running application. DAST tools are also known as web application vulnerability scanners which can detect common vulnerabilities like SQL injection, cross-site scripting, security misconfigurations, and other common issues detailed in [OWASP Top 10](https://owasp.org/www-project-top-ten/). We can use [HCL Appscan](https://www.hcltechsw.com/appscan), [ZAP](https://owasp.org/www-project-zap/), [Burp Suite](https://portswigger.net/burp), and [Invicti](https://www.invicti.com/) which finds vulnerabilities in the running web application. Here is a [list of DAST scanning tools](https://owasp.org/www-community/Vulnerability_Scanning_Tools) provided by OWASP. We can easily integrate these tools with our CI/CD pipeline.

```bash
zap.sh -cmd -quickurl http://example.com/ -quickprogress -quickout example.report.html
```

## 5. Deploy

Deployment can be of infrastructure or application; however, we should scan our deployment files. We can also add a manual trigger where the pipeline waits for external user validation before proceeding to the next stage, or it can be an automated trigger.

### 5.1 Static scan of Kubernete manifest file or Helm chart

It is always a good practice to scan your Kubernetes deployment or Helm chart before deploying. We can use [Checkov](https://github.com/bridgecrewio/checkov) to scans Kubernetes manifests and identifies security and configuration issues. It also supports Helm chart scanning. We can also use [terrascan](https://github.com/tenable/terrascan) and [kubeLinter](https://github.com/stackrox/kube-linter) to scan the Kubernetes manifest.

```bash
docker run -t -v $(pwd):/output bridgecrew/checkov -f /output/keycloak-deploy.yml -o json
# For Helm
docker run -t -v $(pwd):/output bridgecrew/checkov -d /output/ --framework helm -o json
```
### 5.2 Pre-deploy policy check Kubernete manifest YAML file

[Kyverno](https://github.com/kyverno/kyverno/) adds an extra layer of security where only the allowed type of manifest is deployed onto kubernetes, otherwise, it will reject or we can set `validationFailureAction` to audit which only logs the policy violation message for reporting. [Kubewarden](https://www.kubewarden.io/) and [Gatekeeper](https://github.com/open-policy-agent/gatekeeper) are alternative tools available to enforce policies on Kubernetes CRD.

Here is a simple [Kyverno policy](https://kyverno.io/policies/) to [disallow the image's latest tag](https://kyverno.io/policies/best-practices/disallow_latest_tag/disallow_latest_tag/).

### 5.3 kube-bench for CIS scan

[kube-bench](https://github.com/aquasecurity/kube-bench) checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark. We can [deploy kube-bench](/blogs/securing-kubernetes-cluster-kubescape-kubebench/) as a Job that runs daily and consume its report in CI/CD to pass or fail the pipeline based on the level of severity.

```bash
kubectl apply -f eks-job.yaml
kubectl logs kube-bench-pod-name
```

### 5.4 IaC scanning:

-   [Checkov](https://github.com/bridgecrewio/checkov), [Terrascan,](https://github.com/tenable/terrascan) and [Kics](https://github.com/Checkmarx/kics) can be used to scan our Infrastructure code. It supports Terraform, Cloudformation, and Azure ARM resources.
-   [Terratest](https://github.com/gruntwork-io/terratest) can be used to test infrastructure in real-time.

```bash
terraform init
terraform plan -out tf.plan
terraform show -json tf.plan | jq '.' > tf.json
checkov -f tf.json
```

After scanning for Kubernetes deployment and kube-bench we can deploy our application and start with the testing stage.

## 6. Monitoring and Alerting

Monitoring and alerting is the process of collecting logs and metrics about everything happening in our infrastructure and sending notifications based on the metrics threshold value.

#### 6.1 Metrics monitoring

-   [Prometheus](https://github.com/prometheus/prometheus): It's a widely used open source tool for metrics monitoring. It provides [various exporters](https://prometheus.io/docs/instrumenting/exporters/) that can be used for monitoring systems or application metrics. We can also use [Grafana](https://github.com/grafana/grafana) to visualize prometheus metrics.
-   [Nagios](https://www.nagios.org/projects/nagios-core/) and [Zabbix](https://github.com/zabbix/zabbix): These are open source software tools to monitor IT infrastructures such as networks, servers, virtual machines, and cloud services.
-   [Sensu Go](https://docs.sensu.io/sensu-go/latest/): It is a complete solution for monitoring and observability at scale.

#### 6.2 Log monitoring

-   [OpenSearch](https://opensearch.org/)/[Elasticsearch](https://github.com/elastic/elasticsearch): It is a real-time distributed and analytic engine that helps in performing various kinds of search operations.
-   [Graylog](https://www.graylog.org/products/open-source): It provides centralized log management functionality for collecting, storing, and analyzing data.
-   [Grafana Loki](https://grafana.com/oss/loki/): It's a lightweight log aggregation system designed to store and query logs from all your applications and infrastructure.

#### 6.3 Alerting

-   [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/): The Alertmanager handles alerts sent by client applications such as the Prometheus server.
-   [Grafana OnCall](https://github.com/grafana/oncall): Developer-friendly incident response with phone calls, SMS, slack, and telegram notifications.

Security-focused logging and monitoring policy is used to prevent sensitive information from being logged in plain text. We can write a test case in our logging system to look for certain patterns of data. For example, a regex to find out sensitive information so that we can detect the logs in a lower environment.

Application performance Monitoring ([APM](https://www.elastic.co/observability/application-performance-monitoring)) improves the visibility into a distributed microservices architecture. The APM data can help enhance software security by allowing a full view of an application. [Distributed tracing](https://www.dynatrace.com/news/blog/what-is-distributed-tracing/) tools like [Zipkin](https://github.com/openzipkin/zipkin) and [Jaeger](https://github.com/jaegertracing/jaeger) kind of stitch all logs together and bring full visibility of requests from start to end. It speeds up response time for new bugs or attacks.

Although all cloud providers have their own monitoring toolsets and some tools are accessible from the marketplace. Also, there are paid monitoring tool providers like [Newrelic](https://newrelic.com/), [Datadog](https://www.datadoghq.com/), [Appdynamics](https://www.appdynamics.com/), and [Splunk](https://www.splunk.com/) that provide all types of monitoring.

### 6.4 Security information and event management (SIEM)

Security information and event management ([SIEM](https://www.splunk.com/en_us/data-insider/what-is-siem.html)) offer real-time monitoring and analysis of events as well as tracking and logging of security data for compliance or auditing purposes. [Splunk](https://www.splunk.com/en_us/data-insider/what-is-siem.html), [Elastic SIEM](https://www.elastic.co/security/siem), and [Wazuh](https://github.com/wazuh/wazuh) which give automated detection of suspicious activity and tools with behavior-based rules also can detect anomalies using prebuilt ML jobs.

### 6.5 Auditing

After the deployment visibility comes from the level of auditing that has been put in place on application and infrastructure. The goal would be to have your auditing at a level that allows you to feed info into a security tool to give needed data. We can enable audits on the AWS cloud using [CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html) and on Azure with [platform logs](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/platform-logs-overview). For auditing applications, we can enable inbuilt audit logs and send the audit data to any logging tool like [Elasticseach](https://www.elastic.co/enterprise-search) using [auditbeat](https://www.elastic.co/beats/auditbeat) or [Splunk](https://www.splunk.com/) and create an auditing dashboard.

### 6.6 Kubernetes runtime security monitoring

[Falco](https://falco.org/) is a cloud native Kubernetes threat detection tool. It can detect unexpected behavior, intrusions, and data theft in real time. In the backend, it uses Linux eBPF technology to trace your system and applications at runtime. For example, it can detect if someone tries to read a secret file inside a container, access a pod as a root user, etc, and trigger a webhook or send logs to the monitoring system. There are similar tools like [Tetragon](https://github.com/cilium/tetragon), [KubeArmor](https://github.com/kubearmor/KubeArmor), and [Tracee](https://github.com/aquasecurity/tracee) which also provide Kubernetes runtime security.

Till now, we have seen how DevSecOps CI/CD pipeline looks like. Now, let's dive into adding more security layer on top.

## Best practices to secure your infrastructure for DevSecOps CI/CD

### Network Security

Networking is our first defense against any kind of attack and to prevent attacks on our application we should harden our network.

-   Create a separate private network for the workload (eg. App and DB) and only allow internet access from NAT.
-   Set fine-grained access on inbound and outbound network rules. Also, we can use [Cloud custodian to set the security compliance](https://cloudcustodian.io/docs/aws/examples/securitygroupsdetectremediate.html) which automatically removes any unwanted network traffic.
-   Always configure [Network ACL](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html) (NACL) for subnets in AWS. The best practice would be to block all outbound traffic and then allow the required rules.
-   Use Web Application Firewall (WAF).
-   Enable DDOS protection.
-   [Nmap](https://nmap.org/) and [Wireshark](https://www.wireshark.org/), [tcpdump](https://www.tcpdump.org/) tools can be used to scan networks and packets.
-   Use [VPN](https://www.cisco.com/c/en_in/products/security/vpn-endpoint-security-clients/what-is-vpn.html) or [Bastion host](https://goteleport.com/blog/ssh-bastion-host/) for connecting to infrastructure networks.


### Web Application Firewall (WAF)

[WAF](https://www.cloudflare.com/en-gb/learning/ddos/glossary/web-application-firewall-waf/) is a layer7 firewall that protects our web applications against common web exploits (like XSS and SQL injection) and bots that may affect availability, compromise security, or consume excessive resources. Most cloud service provider provides WAF and with a few clicks, we can easily integrate it with our application.

[Curiefense](https://github.com/curiefense/curiefense) is an open source cloud native self-managed WAF tool that can be used to protect all forms of web traffic, services, DDoS, and APIs. We can also use WAF as a service from [Cloudflare](https://www.cloudflare.com/waf/) and [Imperva](https://www.imperva.com/products/web-application-firewall-waf/).

### Identity Access Management (IAM)

IAM is a centrally defined policy to control access to data, applications, and other network assets. Below are a few methods that help prevent unauthorized access.

-   Have centralized user management using Active Directory or LDAP.
-   Use RBAC access management.
-   Have fine-grained access Policy for AWS IAM role.
-   Rotate user's access and secret keys periodically.
-   Use [Teleport](https://goteleport.com/) for centralized connectivity, authentication, authorization, and audit.
-   [Store secrets](https://spectralops.io/blog/top-9-secret-management-tools-for-2022/) into vaults and ensure that it is only accessible to authorized users.
-   [Implement Zero trust](https://www.microsoft.com/en-us/insidetrack/implementing-a-zero-trust-security-model-at-microsoft) within your services.

### Cloud, server, and application hardening

We can use [CIS benchmark](https://www.cisecurity.org/cis-benchmarks/) to harden cloud, operating system, and application. It is always a good practice to use a hardened OS as it reduces the attack surface of the server. Most of the cloud providers provide a hardened image or we can create our own custom hardened image.

Nowadays, most applications run inside containers. We need to harden our applications as well as containers by doing static analysis and container image scanning.

To protect against viruses, trojans, malware, and other malicious threats we can install Antivirus like [Falcon](https://www.crowdstrike.com/products/endpoint-security/falcon-prevent-antivirus/), [SentinelOne](https://www.sentinelone.com/), or [Clamav](http://www.clamav.net/).

### Server patching

The most common attack vector exploits vulnerabilities in the OS or applications running on servers. Running regular vulnerability scans against the environments and updating regular packages reduces the risk of vulnerability. 

We can create an automation pipeline to patch the server using [Foreman](https://www.theforeman.org/) or [Red Hat Satellite](https://www.redhat.com/en/technologies/management/satellite) and for scanning, we can use [OpenVAS](https://github.com/greenbone/openvas-scanner) or [Nessus](https://www.tenable.com/products/nessus) to get the list of vulnerabilities.

### Securing Kubernetes

Kubernetes has become the backbone of modern infrastructure and to make sure we are running it securely we can use the below tools:

-   Use correct [security-context](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#securitycontext-v1-core) in Kubernetes YAML file.
-   Use [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to block all the traffic by default and allow only required traffic.
-   Use Service Mesh ([Linkerd](https://github.com/linkerd/linkerd2), [Istio](https://github.com/istio/istio)) to have [mTLS](https://linkerd.io/2.11/features/automatic-mtls/#what-is-mtls) communication between microservices and implement Authorization to have fine-grained access.
-   Implement [kube-bench](https://github.com/aquasecurity/kube-bench) for a CIS benchmark report for the kubernetes cluster. We can run this scan daily in our Kubernete cluster and fix any reported vulnerabilities.
-   Use tool like [Kube-hunter](https://github.com/aquasecurity/kube-hunter), [Popeye](https://github.com/derailed/popeye) and [Kubescape](https://github.com/armosec/kubescape) for security weaknesses and misconfigurations in kubernetes clusters and visibility of security issues.
-   Use [Checkov](https://github.com/bridgecrewio/checkov), [KubeLinter](https://github.com/stackrox/kube-linter), and [Terrascan ](https://github.com/tenable/terrascan) for scanning Kubernete YAML, Helm chart with best practices and vulnerabilities.
-   Implement pre-deployment policy checks like [Kyverno](https://github.com/kyverno/kyverno/), [Kubewarden](https://www.kubewarden.io/), and [Gatekeeper](https://github.com/open-policy-agent/gatekeeper) can block non-standard Deployment.
-   Use a hardened image for the worker server. All cloud providers provide CIS benchmark harden images. We can also build our own custom hardened image using [amazon-eks-ami](https://github.com/awslabs/amazon-eks-ami).
-   Store Kubernetes secret in an encrypted format or [use an external secret manager like Vault](https://www.infracloud.io/blogs/kubernetes-secrets-hashicorp-vault/).
-   Use [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to assign AWS roles directly to Kubernete service accounts.
-   Implement [Chaos Mesh](https://github.com/chaos-mesh/chaos-mesh) and [Litmus](https://github.com/litmuschaos/litmus) chaos engineering framework to understand the behavior and stability of application in real-world use cases.
-   Follow [best practice to secure kubernetes](https://www.aquasec.com/cloud-native-academy/kubernetes-in-production/kubernetes-security-best-practices-10-steps-to-securing-Kubernetes/) 
-   Use tool like [Falco](https://falco.org/), [Tracee](https://github.com/aquasecurity/tracee) to monitor runtime privileged and unwanted system calls.

### Containers

Containers are the smallest level of abstraction for running any workload in modern infrastructure. Below are a few methods to secure our containers and we have also seen above how to integrate them into our CI/CD pipeline.

-   Scan the Container image and Dockerfile.
-   Reduce the Docker image size with a [multi-stage](https://docs.docker.com/develop/develop-images/multistage-build/) build and using a distroless image to reduce the attack surface.
-   Don't use the root user and privileged containers.
-   Have [Gvisor](https://gvisor.dev/) and [Kata containers](https://katacontainers.io/) for kernel isolation.
-   Use container image signing and verification.
-   Set up a list of known good containers registry.
-   Implement the [Container security best practices](https://blog.aquasec.com/docker-security-best-practices).

### Software supply chain security

> Supply chain security is crucial to the overall security of a software product. An attacker who can control a step in the supply chain can alter the product for malicious intents that range from introducing backdoors in the source code to including vulnerable libraries in the final product.  
> [- in-toto Specification](https://github.com/in-toto/docs/blob/master/in-toto-spec.md#12-motivation)

As per [Anchore 2022 Software Supply Chain Security Report](https://anchore.com/software-supply-chain-security-report-2022/) 62% of Organizations Surveyed have been Impacted by Software Supply Chain Attacks. [Implementing DevSecOps CI/CD](/devsecops-consulting-services/) significantly reduces the Supply chain attack as we scan all our software components like Code, SBOM, Containers, infrastructure, Sign and verify containers, etc.

### Center for Internet Security (CIS) standards 

CIS is a non-profit organization that provides security standards benchmark to secure our infrastructure. One of the benefits of following the [CIS benchmark](https://www.cisecurity.org/cis-benchmarks/) is that it directly maps to several established standards guidelines including the NIST Cybersecurity Framework (CSF), ISO 27000 series of standards, PCI DSS, HIPAA, and others. Also, [Level 2 CIS](https://www.cisecurity.org/cis-benchmarks/cis-benchmarks-faq#:~:text=The%20Level%202%20profile) benchmark gives more security controls.

### Vulnerability Assessment & Penetration Testing (VAPT)

[VAPT](https://www.valencynetworks.com/articles/how-to-decide-frequency-of-vapt-vulnerability-assessment-penetration-testing.html) is a security testing method used by organizations to test their applications, network, endpoint, and cloud. It can be performed by internal and external third-party vendors. Depending upon compliance and regulation also how risky the technology is, organizations do schedule VPAT scans quarterly, half-yearly, or annually.

#### Vulnerability Assessment

A Vulnerability Assessment (VA) is a security process to identify weaknesses or vulnerabilities in an application, system, and network. The goal of a vulnerability assessment is to determine all vulnerabilities and help the operator fix them. [DAST](#43-dynamic-application-security-testing-dast) scanning is also part of vulnerability assessment and it's often quick ranging from 10 minutes to 48 hours depending on the configuration. It is easier to integrate with our CI/CD pipeline whereas pen testing goes beyond VA and has aggressive scanning and exploitation after discovering any vulnerabilities.

#### Penetration Testing

Pen testing is a proactive cybersecurity practice where security experts target individual components or whole applications to find vulnerabilities that can be exploited to compromise the application and data. [ZAP](https://owasp.org/www-project-zap/), [Metasploit](https://www.metasploit.com/), and [Burp Suite](https://portswigger.net/burp) can be used for doing pen tests and it can discover vulnerabilities that might be missed by SAST and DAST tools. The downside of a pen test is that it takes more time depending on the coverage and configuration. The proper pen test might take up to several weeks, and with DevOps development speed, it becomes unsustainable. However, it's still worth adding Internal VAPT which can be done on every feature release to move fast and external VAPT can be done biannually or annually to keep overall security in check.

## Conclusion

To quickly recap what we have done to make the DevSecOps pipeline, we scanned for secrets, SAST and SBOM to find any vulnerabilities in our code. After that, we scanned our Dockerfile, container image, Kubernete manifest and did a container validation test, and signed our container image to ensure it's safe and secure. After deployment, we did Smoke, API test, and DAST scanning to ensure there is no bug in deployment. Always remember security requires constant attention and improvement. However, these can be the first few steps on the never-ending journey towards DevSecOps.

Implementing DevSecOps best practices reduces the risk of vulnerabilities and hacking. Scanning all parts of your infrastructure and application gives full visibility of potential threats and possible ways to remediate them. "The only way to do security right is to have multiple layers of security" hence we talked about multiple methods and tools that can be used for finding vulnerabilities.

Here is a list of a few tools that we have used to set up our DevSecOps pipeline.

| Category        | Tools           |
| ------------- |:-------------|
| Threat modeling      | [Threat dragon](https://owasp.org/www-project-threat-dragon/), [Cairis](https://cairis.org/cairis/tmdocsmore/)  | 
| Secret scan      | [detect-secret](https://github.com/Yelp/detect-secrets), [Gitleaks](https://github.com/zricethezav/gitleaks), [git-secrets](https://github.com/awslabs/git-secrets)  |
| SBOM scan      | [Syft](https://github.com/anchore/syft), [Grype](https://github.com/anchore/grype), [Trivy](https://github.com/aquasecurity/trivy), [Dependency-check](https://owasp.org/www-project-dependency-check/), [Dependency-track](https://github.com/DependencyTrack/dependency-track)  |
| SAST scan | [SonarQube](https://github.com/SonarSource/sonarqube), [Checkmarx](https://checkmarx.com/product/cxsast-source-code-scanning/), [Veracode](https://www.veracode.com/products/binary-static-analysis-sast), [Klocwork](https://www.perforce.com/products/klocwork) |
| Unit testing | [JaCoCo](https://github.com/jacoco/jacoco), [Mocha](https://mochajs.org/), [Jasmine](https://jasmine.github.io/)      |
| Dockerfile scan | [Checkov](https://github.com/bridgecrewio/checkov), [docker scan](https://docs.docker.com/engine/scan/)     |
| Container scan |  [Trivy](https://github.com/aquasecurity/trivy), [Grype](https://github.com/anchore/grype), [Clair](https://github.com/quay/clair), [docker scan](https://docs.docker.com/engine/scan/), [Aqua scan](https://www.aquasec.com/products/container-analysis/)   |
| Container signing |  [Cosign](https://github.com/sigstore/cosign), [Skopeo](https://github.com/containers/skopeo)    |
| Container validation |  [goss](https://github.com/aelsabbahy/goss), [kgoss](https://github.com/aelsabbahy/goss/tree/master/extras/kgoss)    |
| Kubernete manifest scan |  [Checkov](https://github.com/bridgecrewio/checkov), [Terrascan](https://github.com/tenable/terrascan), [KubeLinter](https://github.com/stackrox/kube-linter)    |
| Kubernetes manifest pre-check |  [Kyverno](https://github.com/kyverno/kyverno), [Kubewarden](https://www.kubewarden.io/), [Gatekeeper](https://github.com/open-policy-agent/gatekeeper)     |
| CIS scan | [kube-bench](https://github.com/aquasecurity/kube-bench), [CIS-CAT Pro](https://www.cisecurity.org/cybersecurity-tools/cis-cat-pro), [Prowler](https://github.com/prowler-cloud/prowler)     |
| IaC scan | [Checkov](https://github.com/bridgecrewio/checkov), [Terrascan](https://github.com/tenable/terrascan), [KICS](https://github.com/Checkmarx/kics), [Terratest](https://github.com/gruntwork-io/terratest)     |
| API testing |  [JMeter](https://github.com/apache/jmeter), [Taurus](https://github.com/Blazemeter/taurus), [Postman](https://www.postman.com/), [SoapUI](https://github.com/SmartBear/soapui)     |
| DAST scan |  [ZAP](https://owasp.org/www-project-zap/),[HCL Appscan](https://www.hcltechsw.com/appscan), [Burp Suite](https://portswigger.net/burp), [Invicti](https://www.invicti.com/learn/dynamic-application-security-testing-dast/), [Checkmarx](https://checkmarx.com/product/application-security-platform/), [InsightAppSec](https://www.rapid7.com/products/insightappsec/)    |
| Distributed tracing |  [Zipkin](https://github.com/openzipkin/zipkin), [Jaeger](https://github.com/jaegertracing/jaeger)     |
| Cloud native runtime security |  [Falco](https://github.com/falcosecurity/falco), [Tetragon](https://github.com/cilium/tetragon), [Kubearmor](https://github.com/kubearmor/KubeArmor), [Tracee](https://github.com/aquasecurity/tracee)    |
| Service mesh |  [Istio](https://github.com/istio/istio), [Linkerd](https://github.com/linkerd/linkerd2), [Cilium](https://github.com/cilium/cilium), [Traefik](https://github.com/traefik/traefik)    |
| Network security scan |  [Nmap](https://github.com/nmap/nmap), [Wireshark](https://github.com/wireshark/wireshark), [tcpdump](https://www.tcpdump.org/), [OpenVAS](https://github.com/greenbone/openvas-scanner), [Metasploit](https://docs.rapid7.com/metasploit/discovery-scan/)    |
| Antivirus scan | [Falcon](https://www.crowdstrike.com/products/endpoint-security/falcon-prevent-antivirus/), [SentinelOne](https://www.sentinelone.com/), [Clamav](http://www.clamav.net/)  |
| OS vulnerability scan| [OpenVAS](https://github.com/greenbone/openvas-scanner), [Nessus](https://www.tenable.com/products/nessus), [Nexpose](https://www.rapid7.com/products/nexpose/) |
| OS patching | [Foreman](https://www.theforeman.org/), [Red Hat Satellite](https://www.redhat.com/en/technologies/management/satellite), [Uyuni](https://www.uyuni-project.org/)  |
| Pen testing | [ZAP](https://owasp.org/www-project-zap/), [Metasploit](https://www.metasploit.com/), [Burp Suite](https://portswigger.net/burp)  |


That's a wrap folks :) Hope the article was informative and you enjoyed reading it. For more posts like this one, do subscribe to our weekly newsletter. I'd love to hear your thoughts on this post, so do start a conversation on [LinkedIn](https://www.linkedin.com/in/alok-maurya-091ba682) :)

Looking for help with DevSecOps? do check out our capabilities and how we’re helping startups & enterprises as an [DevSecOps consulting services provider](https://www.infracloud.io/devsecops-consulting-services/).


## References & further reading
- [AWS what is devops?](https://aws.amazon.com/devops/what-is-devops/)
- [Awesome DevSecOps](https://github.com/devsecops/awesome-devsecops)
- [Ultimate DevSecOps library](https://github.com/sottlmarek/DevSecOps)
- [DevSecOps collection](https://github.com/hahwul/DevSecOps)
- [Gitlab is setting standard for devsecops](https://about.gitlab.com/blog/2021/06/01/gitlab-is-setting-standard-for-devsecops/)
- [DevSecOps: Implement security on CI/CD Pipeline](https://medium.com/@anshuman2121/devsecops-implement-security-on-cicd-pipeline-19eb7aa22626)
- [DevSecOps controls](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/secure/devsecops-controls)
- [Guide-devsecops-pipeline](https://www.xenonstack.com/insights/a-quick-guide-devsecops-pipeline)
- [Aquasec devsecops](https://www.aquasec.com/cloud-native-academy/devsecops/devsecops/)