Well-Architected Review - Part II - Operational Excellence
Tags: aws, cloud, engineering, devops
Published: 2022-10-15 01:14:23

_Written by John Nguyen and Manuel Vogel_

## Introduction

This blog post is part of a series about the [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected); what it is, why it makes sense to do it, and how we at kreuzwerker do it. In this entry, we will focus on the __Operational Excellence Pillar__.

## What it is - a quick recap

Using the collective knowledge and experience of their architects and clients, AWS is continuously working on a __Well-Architected Framework__, which consists of key concepts, design principles, and best practices on how to architect and run workloads in the AWS Cloud. AWS developed a Well-Architected Framework in order to understand what makes some customers succeed in the cloud while others fail. They also wanted to identify common problems, decisional and architectural patterns as well as anti-patterns. In other words, what is well-architected and what is not, and to make this knowledge available to all, regardless of whether someone is only just considering migrating to the cloud, or is already running thousands of workloads there. 

The Well-Architected Framework is built on six pillars

1. operational excellence 👨🏽‍💻
1. security 🔒
2. reliability 💪🏾
3. performance efficiency 🚀
4. cost optimization 💵
5. sustainability 🌳

The __AWS Well-Architected Review__ process provides a consistent approach for customers and partners to evaluate architectures and implement scalable designs.

It's important to note that the Well-Architected Review is not an audit. It's nothing to be afraid of; there are no penalty points for not getting things right the first time. A Well-Architected Review is a way of working together on how to improve your architecture. The process leads through several foundational questions and checks. It has been derived from years of experience that have been gained from working with the AWS cloud regarding security, cost efficiency, and performance. Hence, it provides good advice on improvements. It helps you to build secure, high-performing, resilient, and efficient infrastructure for your applications and workloads.

The hard facts about AWS Well-Architected reviews in 2022 are:

- it consists of 58 questions in total across all pillars
- it takes around 4-6 hours for one workload (without tool support)
- the goal is to remediate 45% of the high-risk findings with a minimum of 20 questions answered.

We describe the process from our perspective in more detail [here](https://kreuzwerker.de/en/solution/well-architected-framework). 

## How we do it at kreuzwerker

> Why should you do it with us?

As a [Well-Architected partner](https://partners.amazonaws.com/search/partners?facets=Program%20%3A%20AWS%20Well-Architected%20Partner%20Program&pName=kreuzwerker%20GmbH), we do at least 20 well-architected reviews per year and have built overall deep architectural expertise for every pillar and [hands-on](https://www.wellarchitectedlabs.com/) experience.

> How do we perform such a review?

For us, it's an interactive process: we inspect and adapt every time we do it by requesting feedback from our clients and doing a short internal retrospective. As of now, we perform it as follows:

We do it in 2 blocks from 09:00-12:00 and 13:00-15:00 with a lunch break. However, we can continually adapt if we are faster, e.g., we shift the gap, and we are also flexible whether doing it remotely or at your office.
We do it in an interactive, story-telling mode. This means: you talk, we listen, and then dig deeper into specific areas while being able to cover multiple questions.
Our process is supported by tools (more in the other part of the blog post series 🥳)
We do not just handle the questioning but give guidance to answering them. We can tell you how and why there could be improvements to be made.  Alright, enough for the introduction. Let's jump right into it.

## Operational Excellence Pillar

### In a nutshell

This pillar outlines best practices to build architecture, which includes the ability to support development and run workloads effectively, to gain insights in their operations, and to improve continuously processes and procedures in order to deliver business value.

### Design principles

- __Perform operations as code__
Of course, the best way to learn AWS is by using the AWS console and click through the input. But can you remember all the clicks and replicate these in another region? That would be manual work and you should always automate repetitive tasks. This design principle says that you should define your infrastructure to code. There are plenty of ways to write infrastructure as code (IaC). By IaC, you limit human error and enable consistent responses to events.
- __Make frequent, small, reversible changes__
Imagine, your designer added a small animation which allows a better user experience but you don't want to wait to the next release (which happens in 20 days) to deploy that. Or in a bigger scope, you promised your stakeholders that with next big release everything better and then the deployment went wrong. By making frequent, small and reversible changes, you would deploy faster and wouldn't defend your team by the stakeholders for the failure.
- __Refine operations procedures frequently__
Now, your app is running but you cannot rely on that everything works smoothly. Make sure to set up regular game days to review and validate that all procedures are effective and that teams are familiar with them. Furthermore, a bug found by a customer is a bit embarrassing and can cost some reputation. Therefore, it is always important as you evolve your workload to test it frequently.
- __Anticipate failure__
Failures are good and you can only learn from them. Perform pre-mortem exercises to identify potential sources of failures. Combine that with a game day and let the team be familiar with all the procedures in case of a failure.
- __Learn from all operational failures__
It's also good to share and write post-mortem. Amazon, Netflix and many other big companies do that. These companies are not only sharing their success but also their failures and thus lessons learned to the public. Why? Because even big companies have outages and instead of fixing behind closed walls, they want to show that it happens to any company.

### Improvement process

There are four best practice areas for operational excellence in the cloud.

__Organization__

Always keep in mind that there are many people involved and a shared understanding of your entire workload helps to set __organization priorities__ that will enable business success.
Review your priorities regularly with external and internal stakeholders. Also evaluate governance and external compliance requirements. Use the [AWS Cloud Compliance](http://aws.amazon.com/compliance/) and [AWS Trusted Advisor](http://aws.amazon.com/premiumsupport/technology/trusted-advisor/) to educate your teams so that they can determine the impact on the priorities.

Understand your __operating model__ because each team need to understand their roles in the success of other teams.
Below, you find a table of different operating model. 

But before, here a list of abbreviations:

- Application Engineering and Operations (AEO)
- Infrastructure Engineering and Operations (IEO)
- Centralized Governance (CG)

<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Raleway, sans-serif;font-size:14px;
  overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Raleway, sans-serif;font-size:14px;
  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-94ls{background-color:#C80531;border-color:inherit;color:#ffffff;font-weight:bold;text-align:left;vertical-align:top}
.tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}
</style>

<table class="tg">

  <tbody>
    <tr>
      <th class="tg-94ls"></th>
      <th class="tg-94ls" style="width:200px">Model</th>
      <th class="tg-94ls">Description</th>
    </tr>
    <tr>
      <td align="left" valign="top"> 
      Fully Seperated</td>
      <td align="left" valign="top">
      <img src="//images.ctfassets.net/hqu2g0tau160/2OcrZeNXMFZw7xffjp7I5y/d7813d13b87b80519b5ecb52366f50d7/Nr.1.png" alt="image" width="50" height="100"></td>
      <td align="left" valign="top">
        In the fully separated model, each task in each quadrant is performed independently.<br>
        <br>PRO</br>
        <ul>
          <li>Clear separation of each task</li>  
          </ul>
    <br>CON</br>
    <ul>
          <li>Each team creates a bottleneck and delays</li>
          <li>Transition of tasks to or between teams increases complexity</li>
          <li>Defects identified late and requests are delayed until they become priority</li>
        </ul>
      </td>
    </tr>
     <tr>
      <td align="left" valign="top">
        Separated AEO and IEO with Centralized Governance</td>
      <td align="left" valign="top">
      <img src="//images.ctfassets.net/hqu2g0tau160/3f8bhG2R89sf0LAPTJbDTt/3a1b41791d42d95f8a47f2b2c17bfabc/Nr.2.png" alt="image" width=50% height="100"></td>
      <td align="left" valign="top">Here we follow the "you build it, you run it" methodology. The engineers are responsible for the engineering and operation of their workload. To organize the teams, you should use AWS Organizations and AWS Control Tower. The platform engineering team provides a standardized set of services (e.g. development or monitoring tools) and access to cloud services to the application team. The AWS Service Catalog can be used to govern the tooling.<br>
        <br>PRO</br>
        <ul>
          <li>Standards are distributed, provided, or shared</li>
          <li>Strong feedback loop</li>
          <li>Platform team supports Application team</li>
          <li>Adopting standards may reduce reviews to enter production</li>
          </ul>
    <br>CON</br>
    <ul>
  <li>When changes or additions, Application Team always needs to discuss with Platform Team</li>
        </ul>
      </td>
    </tr>
     <tr>
      <td align="left" valign="top">
        Separated AEO and IEO with CG and a Service Provider</td>
      <td align="left" valign="top">
        <img src="//images.ctfassets.net/hqu2g0tau160/4hla6kvkvSlpTfm16b3Sty/6c6d16e7805da8a527f7a7483d510fdd/nr3.png" alt="image" width="5" height="150">
      </td>
      <td align="left" valign="top">
        Similar to the centralized governance, but you offload some operations tasks such a patching and updating to Managed Services. These service is handled by AWS and they take care of these tasks<br>
        <br>PRO</br>
        <ul>
          <li>Offload "boring" operational tasks</li>
          <li>Gain advantage of your providers' standards, best practices, processes, and expertise</li>
          <li>Latest service offerings</li>
  </ul>
    <br>CON</br>
    <ul>
          <li>Does not address the bottlenecks and delays created by transition of tasks between teams</li>
        </ul>
      </td>
    </tr>
     <tr>
      <td align="left" valign="top">
        Separated AEO and IEO with CG and an Internal Service Provider Consulting Partner</td>
      <td align="left" valign="top">
        <img src="//images.ctfassets.net/hqu2g0tau160/6goRazuPzEerRDbIgMNtN9/55994f62a97d336cb285a175b6170abd/nr5.png" alt="image" width="5" height="150">
      </td>
      <td align="left" valign="top">This model also establishes the "you build it, you run it" methodology. But the difference to the previous model, this enables a Cloud Operations and Platform Enablement (COPE) team which supports when there are no cloud related topics. It provides a forum to ask questions, discuss needs, and identify solutions. The platform engineering team builds the core shared platform capabilities governance via the AWS Service Catalog.<br>
        <br>PRO</br>
        <ul>
          <li>Adopting more DevOps culture</li>
          <li>Enabling cloud transformation for teams, establishes centralized cloud governance, and defines account and organization management standards</li>
          <li>Application Team get CI/CD-pipeline from COPE</li>
          <li>Remove Barriers that slow application team adoption of beneficial cloud capabilities</li>
          </ul>
    <br>CON</br>
    <ul>
          <li>involves huge effort to facilitate cloud adoption and organization standards</li>
        </ul>
      </td>
    </tr>
    <tr>
      <td align="left" valign="top">
        Separated AEO and IEO with Decentralized Governance</td>
       <td align="left" valign="top">
        <img src="//images.ctfassets.net/hqu2g0tau160/6goRazuPzEerRDbIgMNtN9/55994f62a97d336cb285a175b6170abd/nr5.png" alt="image" width="5" height="150">
      </td>
      <td align="left" valign="top">In this model the application engineers and developers perform both platform and application for engineering and operational workloads. Standards are still distributed by the platform team but the application teams are more free to engineer and operate their own capabilities in support of their workload.<br>
        <br>PRO</br>
        <ul>
          <li>Fewer constraints</li>
          <li>More free in choosing own tooling</li>
  </ul>
    <br>CON</br>
    <ul>
          <li>Higher responsibilities of Application Engineer</li>
          <li>Risk of rework is higher</li>
          <li>Enforce policies (Governance via AWS Organizations and AWS Control Tower)</li>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

Furthermore, it is worth noting that __Resources, Processes and procedures, and operations activities have identified owners__. Each __team member should also know what they are responsible for. The responsibility between teams are predefined and negotiated__.

__Prepare__

Attaining operational excellence operational excellence requires your understanding of your workloads and their expected functionality.

- __Design Telemetry__
You should implement monitoring of your workload in order to see the health and internal state of your workload, identify when outcomes are at risk, and enable effective responses.
AWS provides their service [AWS Cloudwatch](http://aws.amazon.com/cloudwatch/) to centralize and structure logs of the application.
AWS Cloudwatch also allows you to create alarms and metrics.
Use AWS X-Ray for collecting and recording traces as transactions travel through your workload. AWS X-Ray generates maps to see how the transactions flow across your workload and services in real time. But there are many third-party tools such as Datadog, New Relic, and Splunk which do these jobs very well. Talk to us; we also have expertise in these tools 😉

- __Design for Operations__
You should implement your entire workload as code. The benefit is that you can apply the same engineering discipline that you use for application code to your infrastructure. Use version control system like [AWS Codecommit](http://aws.amazon.com/codecommit/) to enable tracking of changes and releases, and use [AWS Cloudformation](http://aws.amazon.com/cloudformation/) for your infrastructure templates.
It is recommendable to __test and validate changes__ to help limit and detect error quicker. We at kreuzwerker often use [Terraform](https://www.terraform.io/) and [CDK](https://aws.amazon.com/cdk/) 🤘
This pillar emphasizes automation to its max and you can put these processes in a CI/CD- pipeline. AWS comes with its own [developer tool suits](http://aws.amazon.com/products/developer-tools/) consisting of [AWS CodeBuild](https://aws.amazon.com/codebuild/) for building and testing your code (Continuous Integration), AWS CodeDeploy for automating your code deployments (Continuous Deployment), [AWS CodePipeline](https://aws.amazon.com/codepipeline/) for automating and orchestrating the releases of the previous services. Are you using a different CI/CD- tool? Cool, we are too 😁 We have expertise in [Gitlab](https://about.gitlab.com/), [Jenkins](https://www.jenkins.io/), and [Github Actions](https://github.com/features/actions).

Try to make __frequent, small, reversible changes__ to reduce the scope and impact of changes. Furthermore, you get feedback faster. Also, create multiple environments to experiment, develop, test before you deploy to your production environment. Below you will find a sample CI/CD pipeline, which we internally discussed during one of our [CoP](https://kreuzwerker.de/post/community-of-practice).

![wellarch](//images.ctfassets.net/hqu2g0tau160/01x6yuobPk5VKk2pMfAeuW/8c0afd48e3d5403e3db398a1e1a2660d/wellarch.png)

__Perform patch management__ regularly to stay compliant. [AWS Systems Manager Patch Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-patch.html) and [AWS Systems Manager Maintenance Windows](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-maintenance.html) are the services to automate your patch management workflow. When updating your images, you should manage those with [EC2 Image Builder](http://aws.amazon.com/image-builder/) for you machine images ([AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)) or push container images in [Amazon Elastic Container Registry](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html). It is important to __share these design standard and best practices__ with your team. Create awareness amongst them via push notifications.
Speaking of best practices, AWS offers its program analysis based on Machine Learning [Amazon CodeGuru](https://docs.aws.amazon.com/codeguru/latest/reviewer-ug/welcome.html) for your pipeline to automatically identify potential code and security issues. It performs quality checks of your code and gives recommendations.

- __Mitigate Deployment Risks__
You need to plan to revert to a known good state when __unsuccessful changes__ happen. In a CI/CD pipeline, it stops when test and validation of changes fails. Also, __use configuration management systems__ to track and implement changes. This reduces errors caused by manual processes and reduces the effort to deploy changes. [AWS Config](https://docs.aws.amazon.com/config/latest/developerguide/WhatIsConfig.html) and its [Rules](https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config.html) helps to automatically monitor your AWS resource configurations across accounts and regions.

- __Operational Readiness and Change Management__
Before putting your workload into production, you should have a checklist. This checklist should include your __personnel capabilities__. Here it's helpful to stay up-to-date via blogs, training and certifications and/or internal CoP.
Furthermore, review your AWS Config rules __regularly and ensure consistency__ of them. Also test your CI/CD pipeline and add (manual) approval steps. __Use metadata to identify your resources__ with tags and enable a tagging strategy for easy identification and findings with [Resource Tags](https://d1.awsstatic.com/whitepapers/aws-tagging-best-practices.pdf) or [Resource Groups](https://docs.aws.amazon.com/ARG/latest/APIReference/Welcome.html). Also, prepare game days or "pre-mortems" to anticipate failure and create procedures where appropriate. This makes sure that your team gains confidence in case of outages or defects.

### Operate

__Understanding Workload Health__

You and your team __define, capture, and analyze workload metrics__ to measure the health of the workload such as error rate from the logs. Amazon Cloudwatch gives you workload insights. Your key performance indicators (KPIs) should be based on desired business outcomes (e.g. profit vs. operating expenses) and customer outcomes (e.g. customer satisfaction). Evaluate them regularly. Build dashboards with [AWS Personal Health Dashboard](http://aws.amazon.com/premiumsupport/technology/personal-health-dashboard/) or setup [Amazon OpenSearch Service](http://aws.amazon.com/elasticsearch-service/) to monitor your workload. Let AWS alert you proactively when __workload outcomes are at risk__ or __anomalies are detected__ with [Cloudwatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Create_Anomaly_Detection_Alarm.html) and [Cloudwatch Anomaly Detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html).

__Understanding Operational Health__

You will want to use metrics based on operations outcomes to gain useful insights. Now you want to do analytics on your logs and use [Cloudwatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) or store the logs in Amazon S3, which then triggers an [AWS Glue](http://aws.amazon.com/glue/) crawler to create an [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html) that then can be queried using [Amazon Athena](http://aws.amazon.com/athena/) using standard SQL. The results can be visualized in [Amazon Quicksight](http://aws.amazon.com/quicksight/).

__Responding to Events__

Dr. Werner Vogels, CTO of AWS, said "Everything fails all the time." You should be prepared and deliver consistent results when responding to alerts. Always perform a root cause analysis (RCA) after outage, and prevent recurrence of failures or document workarounds. __Have a process per alert__, ideally a runbook or playbook, with identified owner for any event for which you raise an alert. __Prioritize operational events based on business impact__ and __communicate directly with your users__ when the services they use are impacted. To respond to an event from a state change in your AWS resources, you should create a [CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html) to send notifications to __automate responses to events__ when for example a certain threshold is reached. Keep in mind that it is done better by third-party tools such as Datadog, New Relic or Splunk and investing in such a tool could prevent you from even more financial loss or even reputation.

### Evolve

The last best practice area, Evolution, is the continuous cycle of improvement over time. In order to do that, you need to __Learn, Share, and Improve.__

__Learn, Share, and Improve__

This is quite simple, you want you and your team to learn from failures. Those lessons learned should be shared. Share them via __performing post-incident analysis__ and ideally __document them__. Also, share these learnings across your organization or do it like Netflix or AWS and share them publicly. This will allow you to focus on delivering desired features.

## Conclusion

Operational Excellence is crucial for business success and it is an iterative, as well as ongoing effort. Furthermore, it is a team effort and involves many team members from different units. Here are some key takeaways:

- Automate your workload
  -  Create a CI/CD- Pipeline which tests your code and deploys to a staging environment
  - Perform operation as code and treat it as you would treat it in any software development project
- Monitor and Observe your workload
  - Use logs in order to understand the health of your application
  - Implement alerting on your defined metrics
  - Let AWS Config monitor and govern configuration changes of your AWS resources
- Embrace the whole organization
  - Train your team members to understand metrics and their roles
  - Share lessons learned across the whole organization
  - Perform post-incident analysis and document failures and outages

Take care and the final words are: we're happy to perform an AWS Well-Architected Review with you.