AWS Config Auto Remediation for Configuring S3 Lifecycle Rule
Tags: aws, config, ssm, s3
Published: 2023-01-22 04:01:03

Amazon Simple Storage Service (Amazon S3) is a popular storage solution offering high availability, scalability and durability. Enterprises around the world have put petabytes of data on S3 for various use cases, ranging from big data analysis to critical data backup. With the increasing number of objects stored in S3 buckets, storage cost becomes a major concern.

To ensure the objects are stored cost-effectively, S3 lifecycle configuration can be configured in every bucket so that all objects are automatically transitioned, archived or expired after a certain time. In this article, we will go through a solution that leverages AWS Config and AWS Systems Manager (SSM). This guarantees all S3 buckets to have at least one active lifecycle rule.

> **Note**: Lifecycle configuration cannot be set on buckets with MFA delete enabled.

![Architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/em0z0uk427vj2rfior78.png)

The Config rule evaluates the lifecycle rule configuration of all S3 buckets. If a bucket is found non-compliant, AWS Config applies the remediation using a SSM Automation document.

## Creating an AWS Config rule

1. In the [AWS Config console](https://console.aws.amazon.com/config/), select **Rules** in the navigation pane and click **Add rule**. If you cannot find the button, set up the AWS Config first.
![Add rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bkgeekfhrx8ol2qnky81.png)

2. Among the AWS managed rules, choose the **`s3-lifecycle-policy-check`** rule and click **Next**.
![Specify rule type](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sny9r5vtmnr4pp1xbh2u.png)
> **Note**: There is another similar rule called `s3-version-lifecycle-policy-check` which checks on versioned S3 buckets.

3. Leave all the values as default and click **Next**. Optionally, you can specify the parameters such as `targetTransitionDays` and `targetTransitionStorageClass`.
![Configure rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cz67c77zpl5yyheuva9w.png)

4. Review all the configurations and click **Add rule**.
![Review and create rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1222him7n37brqkbn5us.png)

The rule is now created and shows up on the console. If there is any noncompliant resource (i.e. S3 bucket with no lifecycle policy configured), you should see the count on the compliance column.
![Rules](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wl4jp5ey42d13da64qql.png)

## Creating a custom SSM Automation document

Since there is no existing automation for configuring S3 lifecycle rule, we have to write our own document. The automation will execute both the [`PutBucketLifecycleConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html) and [`GetBucketLifecycleConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLifecycleConfiguration.html) APIs.

```yaml
description: |
  ### Document Name - ConfigureS3BucketLifecycleRule

  ## What does this document do?
  This document is used to create or modify the lifecycle rule configuration for an Amazon S3 bucket.

  ## Input Parameters
  * BucketName: (Required) Name of the S3 bucket (not the ARN).
  * TransitionDays: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    * Default: 90
  * TransitionStorageClass: (Optional) Storage class to which the object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * NoncurrentTransitionDays: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    * Default: 30
  * NoncurrentTransitionStorageClass: (Optional) Storage class to which the noncurrent object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * AutomationAssumeRole: (Required) ARN of the role that allows Automation to perform the actions.

  ## Output Parameters
  * GetBucketLifecycleConfiguration.Output - JSON formatted response from the GetBucketLifecycleConfiguration API call
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
outputs:
  - GetBucketLifecycleConfiguration.Output
parameters:
  BucketName:
    type: String
    description: (Required) Name of the S3 bucket (not the ARN).
    allowedPattern: (?=^.{3,63}$)(?!^(\d+\.)+\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$)
  TransitionDays:
    type: Integer
    description: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    default: 90
  TransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  NoncurrentTransitionDays:
    type: Integer
    description: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    default: 30
  NoncurrentTransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the noncurrent object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  AutomationAssumeRole:
    type: String
    description: (Required) ARN of the role that allows Automation to perform the actions.
    allowedPattern: ^arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/[\w+=,.@-]+
mainSteps:
  - name: PutBucketLifecycleConfiguration
    action: "aws:executeAwsApi"
    description: |
      ## PutBucketLifecycleConfiguration
      Creates or modifies the lifecycle configuration for a S3 Bucket.
    isEnd: false
    inputs:
      Service: s3
      Api: PutBucketLifecycleConfiguration
      Bucket: "{{BucketName}}"
      LifecycleConfiguration:
        Rules:
          - Filter:
              Prefix: ""
            ID: "Default Lifecycle Rule"
            Status: Enabled
            Transitions:
              - Days: "{{ TransitionDays }}"
                StorageClass: "{{ TransitionStorageClass }}"
            NoncurrentVersionTransitions:
              - NoncurrentDays: "{{ NoncurrentTransitionDays }}"
                StorageClass: "{{ NoncurrentTransitionStorageClass }}"
    isCritical: true
    maxAttempts: 2
    timeoutSeconds: 600
  - name: GetBucketLifecycleConfiguration
    action: "aws:executeScript"
    description: |
      ## GetBucketLifecycleConfiguration
      Retrieves the S3 lifecycle configuration for a S3 Bucket.
      ## Outputs
      * Output: JSON formatted response from the GetBucketLifecycleConfiguration API call.
    timeoutSeconds: 600
    isCritical: true
    isEnd: true
    inputs:
      Runtime: python3.6
      Handler: validate_s3_bucket_lifecycle_configuration
      InputPayload:
        Bucket: "{{BucketName}}"
        TransitionDays: "{{ TransitionDays }}"
        TransitionStorageClass: "{{ TransitionStorageClass }}"
        NoncurrentTransitionDays: "{{ NoncurrentTransitionDays }}"
        NoncurrentTransitionStorageClass: "{{ NoncurrentTransitionStorageClass }}"
      Script: |-
        import boto3

        def validate_s3_bucket_lifecycle_configuration(event, context):
            s3_client = boto3.client("s3")
            bucket = event["Bucket"]
            transition_days = event["TransitionDays"]
            transition_storage_class = event["TransitionStorageClass"]
            noncurrent_transition_days = event["NoncurrentTransitionDays"]
            noncurrent_transition_storage_class = event["NoncurrentTransitionStorageClass"]

            output = s3_client.get_bucket_lifecycle_configuration(Bucket=bucket)
            updated_rules = output["Rules"]

            if any(
                any(updated_transition["Days"] == transition_days
                    and updated_transition["StorageClass"] == transition_storage_class
                    for updated_transition in updated_rule["Transitions"])
                and any(updated_noncurrent_transition["NoncurrentDays"] == noncurrent_transition_days
                        and updated_noncurrent_transition["StorageClass"] == noncurrent_transition_storage_class
                        for updated_noncurrent_transition in updated_rule["NoncurrentVersionTransitions"])
                and updated_rule["Status"] == "Enabled"
                for updated_rule in updated_rules
            ):
                return {
                    "output":
                    {
                        "message": "Bucket lifecycle configuration successfully set.",
                        "configuration": updated_rules
                    }
                }
            else:
                info = "CONFIGURATION VALUES DO NOT MATCH WITH PARAMETERS PROVIDED VALUES TransitionDays: {}, TransitionStorageClass: {}, NoncurrentTransitionDays: {}, NoncurrentTransitionStorageClass: {}".format(
                    transition_days,
                    transition_storage_class,
                    noncurrent_transition_days,
                    noncurrent_transition_storage_class
                )
                raise Exception(info)
    outputs:
      - Name: Output
        Selector: $.Payload.output
        Type: StringMap
```

The same document can also be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/ConfigureS3BucketLifecycleRule.yaml.

1. In the [AWS Systems Manager console](https://console.aws.amazon.com/systems-manager/), select **Documents** in the navigation pane, click **Create document** and choose **Automation**.
![Create document](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b622w2jifnhywy2wsxjr.png)

2. For **Name**, enter `ConfigureS3BucketLifecycleRule`. Switch to the **Editor** tab, click **Edit** and replace the content with the above document. Click **Create automation**.
![Create automation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ioa6iiwzuzg47c5z52p.png)![Create automation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3v2ehbaeaok3ao7i85yx.png)

## Creating an IAM role for the SSM automation

1. In the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** in the navigation pane and click **Create role**.
![Create role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/em1kkaad9vnyj6aq4xsv.png)

2. For **Use case**, choose **Systems Manager**. Click **Next**.
![Select trusted entity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7sh67cww20qm61vx630v.png)

3. Choose **`AmazonSSMAutomationRole`** and click **Next**.
![Add permissions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x57k86nq8fiucf261lw6.png)

4. For **Role name**, enter `s3-configure-bucket-lifecycle-rule`. Click **Create role**.
![Name, review and create](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/42i7h8nehzpslqoy7hwg.png)![Name, review and create](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4sscml6j0wxbpzrt2qws.png)

5. Click **View role**.
![View role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cf1jrgz0q2d0xp3m5zsi.png)

6. Click **Add permissions** and choose **Create inline policy**.
![s3-configure-bucket-lifecycle-rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vy3okrasn87k68amzmdo.png)

7. Switch to the **JSON** tab, replace the content with the below document. Click **Review policy**.```json
{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "s3:PutLifecycleConfiguration",
                    "s3:GetLifecycleConfiguration"
                ],
                "Resource": "*",
                "Effect": "Allow"
            }
        ]
}
```![Create policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7lt5f4yjnuw0blufy70p.png)

8. For **Name**, enter `s3-configure-bucket-lifecycle-rule`. Click **Create policy**.
![Create policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x2e2rw2ahobiz591hyzm.png)

## Setting up an automatic remediation for the AWS Config rule

1. On the **Rules** page of the [AWS Config console](https://console.aws.amazon.com/config/), choose the **`s3-lifecycle-policy-check`** rule and click **View details**.
![View details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p3es3ufght8kzrtsixfn.png)

2. Click **Actions** and choose **Manage remediation**.
![Manage remediation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/isf7dodgknv4lq6yo349.png)

3. For **Select remediation method**, choose **Automatic remediation**. For **Remediation action details**, choose **ConfigureS3BucketLifecycleRule**. For **Resource ID parameter**, choose **BucketName**. For **`AutomationAssumeRole`**, enter `arn:aws:iam::<<aws-account-id>>:role/s3-configure-bucket-lifecycle-rule`. Leave other values as default and click **Save changes**. Optionally, you can specify the parameters such as **`TransitionDays`** and **`TransitionStorageClass`**.
![Edit: Remediation action](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2nsdaaxw8anh8ycwd0hr.png)![Edit: Remediation action](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h1xj054mbo3p0pzvolf5.png)

The remediation is now configured. If there was any noncompliant resource before, it will become compliant after some time. In the future, all new S3 buckets will also have the lifecycle rule configured upon creation.
![Resources in scope](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4hlu7q4i602yky300a7i.png)

## Advanced usage

The entire solution can be deployed with an AWS CloudFormation template, which can be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/main.yaml.

For enterprise use, this template can be deployed into multiple accounts easily using an AWS CloudFormation StackSet.

> **Note**: If the solution is deployed in multiple regions, it is advised to separate the IAM role creation from the template as IAM resources are global constructs.

## References
- https://docs.aws.amazon.com/config/latest/developerguide/s3-lifecycle-policy-check.html
- https://docs.aws.amazon.com/systems-manager/latest/userguide/create-ssm-console.html
- https://docs.aws.amazon.com/config/latest/developerguide/remediation.html#setup-autoremediation
- https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console