Using immutable infrastructure to achieve cloud security
Tags: aws, security, devops, tutorial
Published: 2022-11-19 12:20:12

Maintaining cloud infrastructure, especially compute components, requires a lot of effort – from patch management, secure configuration, and more.  
Other than the efforts it takes for the maintenance part, it simply will not scale.  
Will we be able to support our workloads when we need to scale to thousands of machines at peak?  
Immutable infrastructure is a deployment method where compute components (virtual machines, containers, etc.) are never updated – we simply replace a running component with a new one and decommission the old one.  
Immutable infrastructure has its advantages, such as:  
* No dependent on previous VM/container state  
* No configuration drifts  
* The fast configuration management process  
* Easy horizontal scaling  
* Simple rollback/recovery process  

## The Twelve-Factor App
Designing modern or cloud-native applications requires us to follow 12 principles, documents in https://12factor.net  
Looking at this guide, we see that factor number 3 (config) guides us to store configuration in environment variables, outside our code (or VMs/containers).  
For further reading, see:  
* The Twelve-Factor App - Config  
https://12factor.net/config
* AWS - Applying the Twelve-Factor App Methodology to Serverless Applications  
https://aws.amazon.com/blogs/compute/applying-the-twelve-factor-app-methodology-to-serverless-applications/#config  
* Azure - The Twelve-Factor Application  
	https://learn.microsoft.com/en-us/dotnet/architecture/cloud-native/definition#the-twelve-factor-application  
* GCP - Twelve-factor app development on Google Cloud  
	https://cloud.google.com/architecture/twelve-factor-app-development-on-gcp#3_configuration  

If we continue to follow the guidelines, factor number 6 (processes) guides us to create stateless processes, meaning, separating the execution environment and the data, and keeping all stateful or permanent data in an external service such as a database or object storage.  
For further reading, see:  
* The Twelve-Factor App – Processes  
	https://12factor.net/processes  

## How do we migrate to immutable infrastructure?
### Build a golden image
Follow the cloud vendor's documentation about how to download the latest VM image or container image (from a container registry), update security patches, binaries, and libraries to the latest version, customize the image to suit the application's needs, and store the image in a central image repository.  
It is essential to copy/install only necessary components inside the image and remove any unnecessary components – it will allow you to keep a minimal image size and decrease the attack surface.  
It is recommended to sign your image during the storage process in your private registry, to make sure it was not changed and that it was created by a known source.  

For further reading, see:  
* Automate OS Image Build Pipelines with EC2 Image Builder  
	https://aws.amazon.com/blogs/aws/automate-os-image-build-pipelines-with-ec2-image-builder/  
* Creating a container image for use on Amazon ECS  
	https://docs.aws.amazon.com/AmazonECS/latest/userguide/create-container-image.html  
* Azure VM Image Builder overview  
	https://learn.microsoft.com/en-us/azure/virtual-machines/image-builder-overview  
* Build and deploy container images in the cloud with Azure Container Registry Tasks  
	https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-quick-task  
* Create custom images  
	https://cloud.google.com/compute/docs/images/create-custom  
* Building container images  
	https://cloud.google.com/build/docs/building/build-containers  

### Create deployment pipeline
Create a CI/CD pipeline to automate the following process:  
* Check for new software/binaries/library versions against well-known and signed repositories  
* Pull the latest image from your private image repository  
* Update the image with the latest software and configuration changes in your image registry  
* Run automated tests (unit tests, functional tests, acceptance tests, integration tests) to make sure the new build does not break the application  
* Gradually deploy a new version of your VMs / containers and decommission old versions  

For further reading, see:  
* Create an image pipeline using the EC2 Image Builder console wizard  
	https://docs.aws.amazon.com/imagebuilder/latest/userguide/start-build-image-pipeline.html  
* Create a container image pipeline using the EC2 Image Builder console wizard  
	https://docs.aws.amazon.com/imagebuilder/latest/userguide/start-build-container-pipeline.html		
* Streamline your custom image-building process with the Azure VM Image Builder service  
	https://azure.microsoft.com/de-de/blog/streamline-your-custom-image-building-process-with-azure-vm-image-builder-service/  
* Build a container image to deploy apps using Azure Pipelines  
	https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/build-image  
* Creating the secure image pipeline  
	https://cloud.google.com/software-supply-chain-security/docs/create-secure-image-pipeline  
* Using the secure image pipeline  
	https://cloud.google.com/software-supply-chain-security/docs/use-image-pipeline  
	
### Continues monitoring
Continuously monitor for compliance against your desired configuration settings, security best practices (such as CIS benchmark hardening settings), and well-known software vulnerabilities.  
In case any of the above is met, create an automated process, and use your previously created pipeline to replace the currently running images with the latest image version from your registry.  

For further reading, see:  
* How to Set Up Continuous Golden AMI Vulnerability Assessments with Amazon Inspector  
	https://aws.amazon.com/blogs/security/how-to-set-up-continuous-golden-ami-vulnerability-assessments-with-amazon-inspector/  
* Scanning Amazon ECR container images with Amazon Inspector  
	https://docs.aws.amazon.com/inspector/latest/user/enable-disable-scanning-ecr.html  
* Manage virtual machine compliance  
	https://learn.microsoft.com/en-us/azure/architecture/example-scenario/security/virtual-machine-compliance  
* Use Defender for Containers to scan your Azure Container Registry images for vulnerabilities  
	https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-containers-vulnerability-assessment-azure  
* Automatically scan container images for known vulnerabilities  
	https://cloud.google.com/kubernetes-engine/docs/how-to/security-posture-vulnerability-scanning  

## Summary
In this article, we have reviewed the concept of immutable infrastructure, its benefits, and the process for creating a secure, automated, and scalable solution for building immutable infrastructure in the cloud.  

### References
* The History of Pets vs Cattle and How to Use the Analogy Properly  
	https://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/  
* Deploy using immutable infrastructure  
	https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_tracking_change_management_immutable_infrastructure.html  
* Immutable infrastructure CI/CD using Jenkins and Terraform on Azure  
	https://learn.microsoft.com/en-us/azure/architecture/solution-ideas/articles/immutable-infrastructure-cicd-using-jenkins-and-terraform-on-azure-virtual-architecture-overview  
* Automate your deployments  
	https://cloud.google.com/architecture/framework/operational-excellence/automate-your-deployments  

### About the Author
Eyal Estrin is a cloud and information security architect, the owner of the blog [Security & Cloud 24/7](https://security-24-7.com/) and the author of the book [Cloud Security Handbook](https://amzn.to/3xMI4Ak), with more than 20 years in the IT industry.
You can connect with him on [Twitter](https://twitter.com/eyalestrin) and [LinkedIn](https://www.linkedin.com/in/eyalestrin/).