How I build resiliency on the financial service application
Tags: aws, architecture, resilience, resiliency
Published: 2024-01-20 14:52:08

[What is resiliency?](#what-is-resiliency)
[Basic Architecture for resilient](#basic-architecture-for-resilient)
[Terminology](#terminology)
[What is an Operational Readiness Score (ORS)?](#what-is-an-operational-readiness-score-ors)
[What is Chaos Engineering?](#what-is-chaos-engineering)
[AWS Fault Injection Simulator (FIS) targets and actions](#aws-fault-injection-simulator-fis-targets-and-actions)
[Action risk level](#action-risk-level)
[What is AWS Resilience Hub?](#what-is-aws-resilience-hub)
[How to plan resiliency?](#how-to-plan-resiliency)
[Best practices for resiliency](#best-practices-for-resiliency)
[What are Static stability patterns?](#what-are-static-stability-patterns)
[Advanced Architecture for resilient](#advanced-architecture-for-resilient)


<br/>&nbsp;<br/>&nbsp;

## What is resiliency?
Resiliency refers to the ability to recover from disruptions and continue operating even when impaired. It also involves dynamically scaling to meet demand.

<br/><u>Example:</u>
In the case of a disruption, RDS (Amazon Relational Database Service) automatically attempts to launch a new instance in the same Availability Zone and attach the EBS (Elastic Block Store) volume, ensuring continuity of operations.

<br/><u>AWS Tools for Resiliency:</u>

Relational Database (RDS): 

- AWS provides tools like the AWS Advanced JDBC Wrapper and Amazon RDS Proxy to enhance resiliency and improve database performance and availability.

Elastic Block Store (EBS): 

- The Fast Snapshot Restore (FSR) feature offered by AWS helps in quickly restoring EBS snapshots, contributing to faster recovery times and improved resiliency.




<br/>&nbsp;<br/>&nbsp;

## Basic Architecture for resilient

<u>Auto-scaling group</u>

Utilize an auto-scaling group to dynamically adjust the number of instances in response to workload changes, ensuring optimal resource utilization.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n44l12nxh4aamq2kxk1e.PNG)

<br/><u>Multiple Availability Zones (AZs) instances</u>
Deploy instances across multiple AZs to enhance fault tolerance and high availability by distributing the workload across different physical locations.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pk6d3e7wu8c37r8jh4wi.PNG)

<br/><u>Multi-regional database services</u>
Implement database services across multiple regions to achieve data redundancy, disaster recovery, and improved performance.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ar8xxamr04derc6pozui.PNG)

<br/><u>DNS (Route53) routing rules for multi-region traffic</u>
Configure DNS (Route53) routing rules to direct customers' traffic to multiple regions, enabling load balancing and improved responsiveness.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5xjnqefvvq6gqrg421rr.PNG)

<br/><u>AWS Global Accelerator (global load-balancer)</u>
Utilize AWS Global Accelerator as a global load-balancer to optimize network traffic distribution and improve the performance of applications across regions.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rz3en3qk8dr3t4rhlk11.PNG)



<br/>&nbsp;<br/>&nbsp;

## Terminology

<br/><u>EventBridge</u>

- Generates an event that triggers actions or workflows based on predefined rules.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q1nbzz1lws2ggegpydm1.PNG)

<br/><u>Amazon Detective</u>

- Investigates the source of a potential threat or security incident, providing insights and analysis.

<br/><u>Route 53</u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pv9hjbwskt0tz7icmmly.PNG)

- Domain Name System (DNS) servers
- Facilitates region failover by routing traffic to another region in the event of an outage or disruption.
- Provides failover validation to ensure the failover process is functioning correctly.

<br/><u>Resilience posture</u>
- Overall state or condition of an entity's resilience
- Readiness of the system to recover from disruptions
- preparedness: proactively identifying risks
- redundancy: backup systems
- flexibility: response to changing circumstances
- robustness: building systems that can withstand stress or shocks
- rapid recovery: quick restoration of operations after a disruption
- scalability: scaling resources based on demand
- continuity: uninterrupted functioning of critical operations


<br/><u>AWS Resilience Hub</u>

- Manage and monitor resilience posture and disaster recovery (DR) capabilities across multiple AWS accounts and Regions
- View the resilience status, health, and compliance on the dashboard
- Define resilience readiness goals and track progress
- Assign resiliency posture to applications based on their resilience tier
- Assesses applications against the targets set by their respective tiers, either through the console, scheduled evaluations, or CI/CD pipelines

<br/><u>Resiliency Policies</u>
- the set of guidelines that guide an organization's approach to building resilience
- outline the desired outcomes for managing disruptions
- including technology, operations, security, disaster recovery, business continuity
- consistency, standardization of resilience efforts across different departments
- guidelines for monitoring, testing
- clear communication, awareness, and training throughout the organization


<br/><u>Resiliency Tier</u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jeto1gccymiv99gj38sk.PNG)
- classification system based on their level of resilience and availability
- defined based on factors such as downtime tolerance, recovery time objectives (RTO), recovery point objectives (RPO)
- higher resiliency tiers indicate greater levels of redundancy, fault tolerance, and recovery capabilities
- setting expectations of service level agreements (SLAs) and recovery capabilities of systems and services


<br/><u>AWS Fault Injection Simulator (FIS)</u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8z2vvumsetiijba0miu5.PNG)

- Perform fault injection experiments on your AWS workloads
- Based on the principles of chaos engineering
- Creating disruptive events to observe system responds
- Improve the performance and resiliency

<br/><u>FIS experiments</u>

- Conducts fault injection experiments on AWS workloads to test resilience and identify potential weaknesses.

<br/><u>Standard Operating Procedure (SOP)</u>

- Refers to a set of step-by-step instructions compiled by an organization to guide workers in carrying out routine operations.

<br/><u>Data sovereignty regulations</u>

- Country-specific requirements that dictate data must remain within the borders of the jurisdiction where it originated, ensuring compliance with data protection and privacy laws.

<br/><u>Security posture: </u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z9lifwb960kypplt18v3.PNG)

- Refers to the overall level of cyber security readiness within an organization.
- Prevention, Detection, Response, Recovery, Adaptability

<br/><u>Recovery Point Objective - RPO:</u>

- the maximum amount of data that can be lost after a recovery
 
<br/><u>Recovery Time Objective  - RTO</u>

- the maximum acceptable amount of time for restoring an application



<br/>&nbsp;<br/>&nbsp;

## What is an Operational Readiness Score (ORS)?

Ensure compliance with Recovery Point Objective (RPO) and Recovery Time Objective (RTO) Service Level Agreement (SLA) objectives.

<u>Redundancy:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kaek3k7kchgek5xj6fg3.PNG)
Create copies of volumes within or across Availability Zones to establish redundancy and enhance system resilience.

<u>Disaster recovery:</u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xlp5kbmlx7waj5t2xspr.PNG)

Copy volumes to another region to establish a disaster recovery solution, ensuring data protection and the ability to recover from major disruptions.

<u>Re-deploy production data for test/dev:</u>
Utilize production data for testing and development purposes by redeploying it, allowing for realistic simulations and accurate testing environments.


<br/>&nbsp;<br/>&nbsp;

## What is Chaos Engineering?

<u>Goal:</u>

- The goal is to build a more resilient infrastructure.
- The aim is to enable the system to automatically recover from various types of failures.

<br/><u>How:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/29wvqf798v75xyc2zmzr.PNG)

- An environment is established to facilitate Chaos Engineering.
- Production is deliberately disrupted in a controlled manner for learning.
- The intention is to shift the culture and priorities towards resilience.
- The system's ability to automatically recover from failures, such as the loss of an entire region or a spike in traffic, is examined.
- Artificial failures or disruptions, including taking services offline, deleting data centers, or overwhelming the system with fake traffic, are simulated.

<br/><u>What does it do?</u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2iwxjrtpxi5oscbi90fk.PNG)

- Chaos Engineering involves conducting purposeful fault injection experiments.
- It helps assess how the system will react to unknowns.
- Real-world conditions are necessary to uncover hidden bugs.
- It aids in managing bottlenecks in distributed systems.
- The system's behavior is continuously tested under unpredictable conditions.
- The goal is to identify weaknesses and single points of failure.



<br/>&nbsp;<br/>&nbsp;

## AWS Fault Injection Simulator (FIS) targets and actions

<u>Amazon EC2 actions</u>

- Stop, reboot, and terminate instance
- Spot instance interruptions
- Inject API internal, throttle, unavailable error
- Network Latency

<br/><u>Amazon EKS actions</u>

- Pod CPU stress
- Pod delete
- Pod I/O stress
- Pod memory stress
- Pod network latency
- Terminate Nodegroup instances

<br/><u>Amazon ECS actions</u>

- Task kill process
- Task Stop
- Drain container instances

<br/><u>Amazon S3 actions</u>

- Denies traffic to and from the Regional endpoint 
- Add latency to and from S3 from EC2

<br/><u>Networking actions</u>

- Denies the specified traffic to the target subnets

<br/><u>RDS actions</u>

- Failover Cluster
- Reboot Instance


<br/>&nbsp;<br/>&nbsp;

## Action risk level

<u>Low risk</u>

- CPU/memory street
- Instance reboots
- Container restarts
- Pause I/O

<br/><u>Medium risk</u>

- Downstream latency increase

<br/><u>High risk</u>

- Availability Zone/Region impaired
- AWS service impaired


<br/>&nbsp;<br/>&nbsp;

## What is AWS Resilience Hub?

<u>Central security tool: </u>
AWS Resilience Hub serves as a centralized security tool.

<br/>Displays the security and compliance status:

- It provides a visual representation of the security and compliance status.


<br/><u>What it can do?</u>

Analyze applications against targets: 

- The Resilience Hub can analyze applications to ensure they meet Resiliency Policies such as Recovery Point Objective (RPO) and Recovery Time Objective (RTO) for app, infrastructure, availability zone, and region.

<br/>Aggregate security alerts: 

- It aggregates security alerts from various AWS services including AWS Config, AWS GuardDuty, AWS Macie, IAM Access Analyzer, AWS Systems Manager, AWS Firewall, and AWS Health.

<br/>Proactively prepare and protect applications: 

- The Hub helps to proactively prepare and protect AWS applications from disruptions.

<br/>Proactively identify resiliency gaps:

-  It identifies resiliency gaps in the infrastructure to take proactive measures for improvement.

<br/>Metric-driven approach: 

- The Resilience Hub adopts a metric-driven approach, providing scores to assess the resiliency posture of applications

<br/><u>Benefits</u>

Covers multiple accounts:

-  It can be used to cover multiple AWS accounts, providing a comprehensive view of the security and compliance status across all accounts.

<br/>Finds potential issues: 

- The Hub helps in identifying potential issues or vulnerabilities in the infrastructure.

<br/>Visualizing dashboard: 

- It offers a visual dashboard that displays the security and compliance status of applications.




<br/>&nbsp;<br/>&nbsp;

## How to plan resiliency?

<u>Failover plan for resilience:</u>
outlines procedures for replacing failed components, restarting services, or switching to an alternative Availability Zone (AZ) in the event of failures or disruptions, ensuring uninterrupted service availability.


<br/><u>Recovery Time Objective (RTO):</u>
can vary in duration, ranging from minutes to hours, depending on the size and complexity of the system.

<br/><u>RTO Questions:</u>
financial cost? reputational cost? SLA with customers? external compliance or regulatory? 

<br/><u>Recovery Point Objective (RPO):</u>
is zero in cases where recoverable instances fail because the EBS (Elastic Block Store) volume can be recovered.

<br/><u>RPO Questions:</u>
the impact of data loss? Can lost data be recreated? How long it takes?


<br/>&nbsp;<br/>&nbsp;

## Best practices for resiliency

<u>Goal:</u>
- Fully automate failover processes to prevent data loss.
- Implement a highly available database cluster with synchronous replication across multiple Availability Zones (AZs).
- Ensure data backup is in place.

<br/><u>1</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yhq33tdjfvc500fd2ewr.PNG)
Keep a Single-AZ Amazon RDS and create a read replica in the primary region with an additional read replica in a secondary/different region.

<br/><u>2</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m9tlcjgm91creyichfe3.PNG)
Configure a Multi-AZ setup with a writer and a read replica in two different regions:

- RDS automatically performs failover to the standby database instance.
- The endpoint for your database instance remains the same after a failover.

<br/><u>3</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zq5efgypkw77bhykf63e.PNG)
Implement a strong identity foundation:

- Enforce the principle of least privilege and separate duties to enhance security.

<br/><u>4</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/klidqvrqdbfxyd86gp23.PNG)
Ensure traceability through logging and monitoring activities.

<br/><u>5</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1nenw3nps684ht7y87x2.PNG)
Apply security measures at all layers of the system.

<br/><u>6</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jn9c753ya142se4buypu.PNG)
Protect data during transit and at rest:

- Utilize encryption techniques to secure data while it is being transmitted and when it is at rest.

<br/><u>7</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cslcflnxrcf8evmqk512.PNG)
Classify and prioritize data based on sensitivity for appropriate security measures.


<br/>&nbsp;<br/>&nbsp;

## What are Static stability patterns?

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ikmh886l89jpswyvtz8d.PNG)

<u>Data plane:</u>
Refers to the physical servers where customers' Amazon EC2 instances run.
<br/>Example:

- Reading, and writing to an EBS volume,
- Getting, and putting objects in an S3 bucket
- Route 53 answering DNS queries


<br/><u>Control plane:</u>
The control plane consists of various services that interact with the data plane.
<br/>Examples 

- Providing administrative APIs used for resource management (create, read, update, delete).


<br/><u>Why is it needed?</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p12c4h7qvta4euf1tb9i.PNG)
- Static stability is crucial for stateless applications with auto-scaling capabilities.
- Deploying a stateless application allows for easy addition or removal of compute nodes using auto-scaling, ensuring scalability.

<br/>

- Without static stability, launching new EC2 instances may not be possible. 
- The system relies on the Amazon EC2 control plane for the recovery path, which includes tasks like downloading runtime configuration, registering instances with discovery services, and acquiring credentials.

<br/><u>How to apply Static stability patterns?</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hygyn09u9vo2jpigx79q.PNG)
- Aim for Availability Zone independence by keeping packet flows within the same Availability Zone instead of crossing boundaries.
- Keep network traffic local to the specific Availability Zone.
- In case of a connectivity impairment in one Availability Zone, contain the impairment within that zone.

<br/><u>Extra work for Static stability patterns:</u>

- Replicate any hard state across multiple Availability Zones for disaster recovery.
- Store periodic database backups in Amazon S3 and maintain read replicas across Availability Zone boundaries.

<br/><u>Trade-offs:</u>

- Additional complexity may increase costs.
- Data durability should be considered when evaluating the trade-offs.


<br/>&nbsp;<br/>&nbsp;

## Advanced Architecture for resilient

<br/><u>Active-active on Availability Zones:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iyihj518ta4cmyqdyq6v.PNG)
- Refers to a load-balanced service that operates simultaneously across multiple Availability Zones.
- Benefit: Ensures continuous service availability without downtime.
- Tradeoff: This can be costly to implement.


<br/><u>Active-standby on Availability Zones:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ygb62mbbf2ees7w3gp26.PNG)
- Involves setting up a relational database with an active instance in one Availability Zone and a standby instance in another.
- Benefit: Provides an affordable solution.
- Tradeoff: May experience downtime during failover as the standby instance needs to scale up to handle the traffic.


<br/><u>Regional-calls-regional:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8hyv0c5to5w5ragga6tn.PNG)
- Use case: External-facing services that operate within the same region.


<br/><u>Regional-calls-zonal and zonal-calls-zonal:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/le0dlnuxfvcgics2znip.PNG)
- Use case: Ensures network traffic remains within the same Availability Zone.
- Use case: Helps isolate impairments to specific Availability Zones.


<br/><u>Multi-AZ with static stability:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y5ld6r05ztczmhlozzwa.PNG)
- If your workload is distributed across multiple instances, deploy it to all available AZs (typically 3 or more) within the region.
- Ensures static stability and resiliency.


<br/><u>Application portfolio distribution:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/25u8qie0lph183qxu3uc.PNG)
- Distribute critical services across different regions to maintain access during disruptions.

<br/>Trade-offs: 
- Requires significant operational planning when spanning multiple regions. Functional elements may depend on downstream systems and data sources that are deployed in a single region.


<br/><u>Multi-AZ deployment (multi-Region DR):</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tqlkutb49e1bvz7xyjyh.PNG)
- Includes options like Pilot Light and Warm Standby for disaster recovery across multiple regions.

<br/><u>Multi-Region active-active:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j69o1wkqpfef9d2hxwu0.PNG)
- Enabling active-active architecture across multiple regions for enhanced resiliency and redundancy.


<br/>&nbsp;<br/>&nbsp;

## Reference

[REL11-BP05 Use static stability to prevent bimodal behavior](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_withstand_component_failures_static_stability.html)

[Building Resilient Applications in the Cloud](https://dev.to/aws-builders/building-resilient-applications-in-the-cloud-5de3)

[Introduction to AWS Resilience Hub](https://dev.to/aws-builders/introduction-to-aws-resilience-hub-4lcn?comments_sort=top)

[Understand resiliency patterns and trade-offs to architect efficiently in the cloud](https://aws.amazon.com/blogs/architecture/understand-resiliency-patterns-and-trade-offs-to-architect-efficiently-in-the-cloud/)

[Chaos Engineering](https://aws.amazon.com/solutions/resilience/chaos-engineering/)

[Validating and Improving the RTO and RPO Using AWS Resilience Hub](https://aws.amazon.com/blogs/mt/validating-and-improving-the-rto-and-rpo-using-aws-resilience-hub/)

[Establishing RPO and RTO Targets for Cloud Applications](https://aws.amazon.com/blogs/mt/establishing-rpo-and-rto-targets-for-cloud-applications/)

[How to use the AWS Resilience Hub score](https://aws.amazon.com/blogs/mt/how-to-use-the-aws-resilience-hub-score/)

[Build a resilience reporting dashboard with AWS Resilience Hub and Amazon QuickSight](https://aws.amazon.com/blogs/mt/resilience-reporting-dashboard-aws-resilience-hub/)

[Amazon RDS Multi-AZ](https://aws.amazon.com/rds/features/multi-az/)

[Cyber resilience in applications on AWS-Using AWS Security Hub](https://dev.to/aws-builders/cyber-resilience-in-applications-on-aws-using-aws-security-hub-361g)

[Enabling resiliency for a static website hosted on AWS Cloud](https://dev.to/aws-heroes/enabling-resiliency-for-a-static-website-hosted-on-aws-cloud-3nk)

[AWS Multi-Region Fundamentals - WhitePaper Summary](https://dev.to/aws-builders/aws-multi-region-fundamentals-whitepaper-summary-nbg)

[Resilience lifecycle framework](https://docs.aws.amazon.com/prescriptive-guidance/latest/resilience-lifecycle-framework/post-deployment.html)

Testing for resilience with AWS Fault Injection Simulator - DOP318

[Comprehensive guide to deploying business applications with Amazon EBS](https://d1.awsstatic.com/events/Summits/reinvent2023/STG349_Comprehensive-guide-to-deploying-business-applications-with-Amazon-EBS.pdf)