# Container Security

## Summary


With the wide adoption of container-based applications, systems became more complex and security risks increased. This laid the groundwork for container security. Vulnerabilities like dirty copy-on-write only furthered this thinking. This led to a shift left in security along the software development lifecycle, making it a key part of each stage in container app development, also known as **DevSecOps**. The goal is to build secure containers from the ground up without reducing time to market.

DevOps practice has enabled faster release cycles. At runtime, an application may dynamically scale to hundreds or even thousands of containers. It's impossible to have manual policies to secure these containers. Moreover, a production environment involves many layers such as images, registries, orchestrators and container runtimes. Hence, traditional security practices can't be directly applied to securing containers.

## Discussion

### Which are the basic security mechanisms that come with containers?

In terms of filesystem, networking and processes, containers are isolated from one another using **namespaces**. To control access to resources such as CPU and RAM, there's **cgroups**. 

To control kernel calls from a container, we use Linux kernel features called **capabilities** and **seccomp**. Capabilities provides fine-grained access control such as denying "mount" operations or access to raw sockets. Even if someone gains root access within a container, the damage they can do is limited. Secure computing mode (seccomp) disables access to system calls except those allowed. Both these adopt a whitelist approach. 

Red Hat distributions comes with SELinux while Debian distributions come with AppArmor. Both these are security modules that provide access control by implementing **Mandatory Access Control (MAC)**. Only users and processes with sufficient permissions can perform various actions. The access policies can't be changed by users. 

We can also run the kernel with GRSEC and PAX to provide extra compile-time and runtime safety checks. 


### Is it okay to run my process with root access within the container?

Some processes running on typical servers run with root privileges. Examples are SSH daemon, cron daemon, logging daemons, kernel modules, and network configuration tools. It may therefore appear that these processes need to run with root privileges inside a container. In reality, such tasks are handled by the infrastructure surrounding the container such as the host machine or the container platform. Even when a specific process needs root privilege, limit access using *capabilities*. 

In general, stick to the **principle of least privilege**. Don't give more permissions than needed. One way to implement this is to use known user ID and group ID that have the minimum required privileges. An alternative available in Docker is to use a command-line option; example, `docker run --user 1001 ...`, which avoids root access. 

Most processes inside containers are app services and should not require root access. Docker itself requires root access but not the containers. 


### What are the various security threats facing containers?

NIST has identified key threats to containers:  

    + **Image**: An image may lack security updates or become outdated, particularly when security bugs have been discovered after the image was created. Image may contain a misconfiguration, such as running a command with a greater privilege than needed. Image may include malware, particularly when a base layer comes from a third party or untrusted source. Secrets such as SSH private keys may be in the image as clear texts.
    + **Registry**: Images stored at registries may be stale. Registries themselves can be compromised. Pulling images over insecure connections is also an issue.
    + **Orchestrator**: If access privileges are not properly scoped, users can access and affect containers that don't belong to them. Wrong configurations may allow unauthorized nodes to join a cluster.
    + **Container**: At runtime, a malicious software can affect other containers or the host OS. App vulnerabilities can affect the container. Unplanned or unsanctioned containers, especially in a development environment, are a risk.
    + **Host**: The host itself has an attack surface. Because the kernel is shared, runtime isolation is not as high as that of hypervisors.

### What are some best practices to secure containers?

We list some industry best practices to secure containers:

    + Called image provenance, have a system such as *Docker Content Trust* to identify the source of an image that's running in production.
    + Containers must run with minimum resources and privileges. Use *AppArmor* or *SELinux* to enforce access control. Perhaps the most important advice is to not use root privilege to run the container's main process.
    + At runtime, detect threats and respond to the same automatically.
    + Vulnerability scanning, regular auditing, and software updates must be automated.
    + Since containers are not permanent, traditional operational models may not work. Organizations may have to formalize new models.
    + Use *Docker Bench for Security* to know if you're following best practices around container deployment.
    + Minimize the use of third-party images and use only signed images.
    + Mount the host filesystem as read-only or disable UNIX sockets, thus limiting access to an attacker.

### Isn't it better to use VMs instead of containers for better security?

Containers offer only software-based isolation using namespaces and cgroups. Multiple containers share the same kernel. If one container is compromised, the attacker can potentially gain access to other containers on the host. This is where the hypervisor layer of VMs provides hardware-enforced isolation. 

But it's possible to mix both VMs and containers. By running multiple containers inside a VM, we get the performance of containers without sacrificing the security of hypervisor. It's common among cloud providers to isolate tenants using VMs. For greater security, we can use VMs to separate containers that need to run at different security levels. For example, a billing service may run at a higher security level compared to a user-facing web service. 

Joyent and IBM isolate tenants by running containers on bare metal without using VMs. Hyper, Intel and VMware are building fast VM-based frameworks with the aim of getting the best of both worlds. The idea is to secure a container without the bloat of a VM. This is being called **Kata Containers**. It uses *runV* container runtime. 


### Which tools can help me secure my containers?

While container platforms and containers themselves offer a basic level of security, this isn't enough. There's a need to scan container images or monitor a running container. There are many vendors offering products that do these. Some of them are Alert Logic, Anchore, Aporeto, Aqua Security, Capsule8, NeuVector, Qualys, StackRox, Sysdig, and Twistlock. 

Any good tool should have visibility of containers, images and host processes. It should validate container signatures. It should alert when there are image drifts. It should monitor slaves and masters of Swarm or Kubernetes clusters. Tool should have an easy-to-use dashboard. It should make use of image metadata to enable filtering by labels and tags. It should classify threats by severity and priority. Select a tool that integrates well with your CI/CD pipeline. 

Google's Container Registry vulnerability scanning, Red Hat's Atomic Scan, IBM's Bluemix Vulnerability Advisor, CoreOS Clair, Docker Security Scanning, Aqua Security's Peekr, and Twistlock Trust can do security scanning. Runtime threat detection and response are offered by Aqua Security, Joyent Triton SmartOS, Twistlock and Red Hat OpenShift. 

## Milestones

Aug  
2015

**Docker Content Trust (DCT)** is released as a new feature in Docker Engine 1.8. This adds a digital signature to an image before it's pushed to the registry. When images are pulled, the signatures are verified. Through the implementation of The Update Framework (TUF), rollback attacks and freeze attacks are also prevented. 

Oct  
2016

By using a vulnerability in the Linux kernel, called Dirty Copy-on-Write or CVE-2016-5195, one researcher creates an exploit that achieves **container escape**, essentially allowing the attacker to do things outside the container. The exploit uses Linux's virtual Dynamic Shared Object (vDSO). 

Sep  
2018

Google launches beta version of **Container Registry vulnerability scanning**. This helps within a CI/CD pipeline and prevents deployment of images that have vulnerabilities. 

Jan  
2019

Two researchers discover a way to escape from a container into the host system and obtain root privileges. This is possible with a compromised container even if it's using the default or hardened configuration. The exploit overwrites `docker-runc` binary on the host with malicious software. This bug in runc, called CVE-2019-5736, affects Docker and other container platforms. A patch is released in February.