Automated Mass Tagging in AWS Across Accounts and Organizations
Tags: aws, organization, lambda, tagging
Published: 2023-08-17 16:44:47

# Tagging strategy: easier said...

In the expansive world of AWS, tagging resources stands out as both a straightforward task and an essential one. On the surface, it's about assigning a label, a seemingly simple action. Yet, the implications of this action are profound. Tags are not just mere identifiers; they're pivotal tools in organizing, managing, and optimizing your cloud environment, because they cater to a range of organizational needs, such as:

* **Expense Tracking**: for instance, with [**Cost Allocation Tags**](https://aws.amazon.com/blogs/aws-cloud-financial-management/cost-allocation-blog-series-2-aws-generated-vs-user-defined-cost-allocation-tag/), you can monitor specific costs tied to a particular project or department.
    
* **Infrastructure Automation**: tags can trigger **Automated Infrastructure Activities**. Think of an instance that's tagged as 'development' being automatically shut down outside of working hours to save costs.
    
* **Project Phases**: with **Workload Lifecycle** tags, you can easily identify whether a particular resource is in the 'testing', 'development', or 'production' phase.
    
* **Issue Resolution**: **incident management** tags can help in quickly identifying resources that might be affected during an outage or incident.
    
* **Maintenance**: **update management** tags can indicate when a resource was last patched or updated, ensuring timely maintenance.
    
* **Operational Insights**: for a clear view of your operations, **Operational Observability** tags can denote the health or status of resources.
    
* **Data Protection**: **risk and security management** tags can highlight resources that contain sensitive data, ensuring they have tighter security controls.
    
* **Access Management**: **identity and access tags** can dictate who within your organization can access specific resources, reinforcing security protocols.
    

A well-defined tagging strategy is paramount. AWS itself recognizes the significance of this and has published [an extensive whitepaper](https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/tagging-best-practices.html) detailing best practices and guidelines. This strategy isn't just about knowing what to tag, but understanding the 'why' and 'how' behind each tag.

## ...than done

So, once we've established our tagging strategy, is it smooth sailing from there? Well, as the saying goes, "easier said than done." Indeed, a strategy is only as good as its **execution** plan. A comprehensive strategy must be paired with a **pragmatic** action plan detailing its implementation.

From our list of tagging use cases, one aspect becomes abundantly clear: tags, while seemingly simple tools, cater to a **myriad** of distinct purposes. These **purposes**, in turn, address the needs of **diverse teams** within an organization. Whether it's Finance, Operations, Security, or Development teams, each has its **unique requirements**. These teams might possess different skill sets, operate on varying timelines, and even employ distinct **tools** for their tagging activities. The challenge then is **coordination**: how do these teams work in tandem without stepping on each other's toes?

# Organizational complexity

To truly grasp the intricacies of tagging, let's delve into a **real-world scenario** I encountered. In this setup, there's a team, aptly named "**Cloud Center**", responsible for managing the AWS **Organizations**. Their tasks encompass creating, overseeing, and auditing the AWS accounts affiliated with the organization.

Within this Organization, there are several Organizational Units (**OUs**) mirroring the company's internal structure. Each OU houses multiple **projects**, and each project might have separate **accounts** for development, testing, and production.

Each OU is backed by a **Platform Team**, providing project teams with essential tools, such as CI/CD **pipelines** and Infrastructure as Code (**IaC**) execution tools like **Terraform**. Then, every project has a **separate team**, which is diverse, comprising roles like backend engineers, DevOps specialists, QAs, and more. Notably, many team members are often **consultants or contractors** dedicated to specific projects rather than company employees.

Additionally, there are **shared OUs** and accounts dedicated to operational services, like **networking**, Transit Gateway, Network Firewall, and DNS, or **security**\-centric tasks: each of these is managed by a different team.

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/59q4fwhdatzc62s82cf2.png)

On **top** of all that, this Organization belongs to a **larger corporate group**. At the group level, there's a need to monitor the spending of each subsidiary company. This requires **data extraction** from each Organization, necessitating the **tagging of AWS accounts themselves**, with unique labels and values consistent across the entire corporate group.

# The mass-tagging hierarchy

In such a multifaceted environment, expecting every individual to simply read a tagging strategy manual and apply it flawlessly is **wishful thinking**. Each team has its tagging objectives aligned with its goals. However, keeping track of everyone's tagging needs would be a Herculean task. While enforcing a [Tagging Policy](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies.html) can provide some structure, manually **reconciling** the requirements of so many teams would be a colossal **drain** on time and resources.

The reality of the matter is that tags, in many instances, aren't particularly volatile entities. In an ideal setup, they act as labels assigned during the creation of a resource. Once in place, these tags seldom change, except for specific use cases. Given this nature, **it's counterproductive to burden individuals** with a task that, with the right precautions, can be seamlessly automated. After all, machines are inherently better suited for **repetitive** and mundane tasks than humans.

This realization led to the adoption of a **multi-tiered mass-tagging strategy**. Each "tier" or "level" employed a recurring **Lambda** function to tag all resources under its **purview**. Care was taken to ensure that tags from one level didn't overwrite or remove those from another.

## Cloud Center Command

At the **topmost tier**, the Cloud Center took on the responsibility of tagging AWS accounts directly. This was primarily for billing, finance, and cost allocation purposes. They utilized tags that were **globally unique** within the corporate group, along with additional tags indicating the **OU, project, and environment** specifics. Here's how the process was streamlined:

* every night, an **EventBridge**\-scheduled **Lambda** function would activate. This function would:
    
1. **read** all the tags (both key and value) for each account


        def list_accounts():
            existing_accounts = [
                account
                for accounts in _org_client.get_paginator("list_accounts").paginate()
                for account in accounts['Accounts']
            ]
            return existing_accounts
        
        def get_account_tags( account_id):
            formatted_tags = _org_client.list_tags_for_resource(
                ResourceId=account_id)
            return formatted_tags
        
        def handler(event, context):
            for account in list_accounts():
            account_id = account.get('Id')
        
            try:
                tags = get_account_tags(account_id)
            except Exception as ce:
                logger.error(
                    f'Exception retrieving tags in Organization for account {account_id}: {ce}')
                continue

        
2. create a message for each account in an **SQS** queue

        
        sqs.send_message(
            QueueUrl=sqs_queue_url,
            DelaySeconds=15,
            MessageAttributes={
                'Account': {
                    'DataType': 'String',
                    'StringValue': account_id
                },
                'Tags': {
                    'DataType': 'String',
                    'StringValue': json.dumps(response_json)
                },
                'Region': {
                    'DataType': 'String',
                    'StringValue': reg
                }
            },
            MessageBody=(
                f'Tag value for account {account_id} in region {reg}'
            )
        )
        

* each message in the queue would then **trigger a second Lambda** function. This function would:
    
1. read the message content
        

        for record in event['Records']:
            account_id = record['messageAttributes']['Account']['stringValue']
            tags_raw = record['messageAttributes']['Tags']['stringValue']
            reg = record['messageAttributes']['Region']['stringValue']
            receipt_handle = record['receiptHandle']
            tags = json.loads(tags_raw)['Tags']

     
2. **assume an IAM role in the target account**
        
3. save the tag list in an **SSM parameter**
        
4. list resources to be tagged using the `resourcegroupstaggingapi`
        

        client = create_boto3_client(account_id, 'resourcegroupstaggingapi', assume_role(account_id), reg)
        map = client.get_resources(ResourcesPerPage=50)
        list = get_resources_to_tag(map['ResourceTagMappingList'], tagkey, tagvalue)
        
        [...]
        
        def get_resources_to_tag(map, tagkey, tagvalue):
            resourcelist = []
            for resource in map:
                logger.debug(f'Resource: {resource}')
                if resource['ResourceARN'].startswith('arn:aws:cloudformation'):
                    logger.debug(
                        f'Resource {resource} is a cloudformation stack, we do not need to tag it')
                    continue
                to_be_tagged = True
                for tag in resource['Tags']:
                    if tag['Key'] == tagkey and tag['Value'] == tagvalue:
                        to_be_tagged = False
                        logger.debug(
                            f'Found tag {tagkey} with value {tagvalue} in resource, no need to retag')
                        break
                if to_be_tagged == True:
                    logger.debug(
                        f'NOT FOUND tag {tagkey} with value {tagvalue} in resource, need to tag')
                    resourcelist.append(resource['ResourceARN'])
            return resourcelist
        

5. finally, apply the tags.
       

This approach ensured that the Cloud Center team maintained tags in a centralized manner, eliminating the need for disparate synchronization efforts.

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><em>For those interested, you can find a Python version of these Lambda functions </em><a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/theonlymonica/aws-multi-level-tagging" style="pointer-events: none"><strong><em>here</em></strong></a><em>.</em></div>
</div>

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/226d9cwg0sik89swa49b.png)

## Platform Team Playbook

The **second "tier"** in this tagging hierarchy is occupied by the **Platform Teams of each OU**, and their approach mirrors that of the Cloud Center, albeit with some tailored modifications.

In the case of the Platform Teams, their management account has read delegation over the Organization. The nightly process for them unfolds as follows:

1. **Account Listing from Organization**: Triggered by **EventBridge** at a different time than the Cloud Center's process, a **Lambda** function initiates and reads the **list of accounts** specific to its OU from the Organization.
    
2. **Mass Tagging (If Necessary)**: If the Platform Team has its specific tags to apply, it employs a mass-tagging approach **identical** to the Cloud Center's. It's worth noting that **not all Platform Teams have this requirement**. Some use this technique to assign tags that indicate, for instance, which EBS volumes need **backups** or which non-production EC2 instances can be **shut down** during nights and weekends.
    
3. **Terraform Pipeline Integration**: Given that these Platform Teams provide project teams with Terraform execution pipelines, they adopt a methodology (detailed in [this article](https://letsmake.cloud/automating-the-injection-of-cicd-runtime-information-into-terraform-code)) that **dynamically** instructs the AWS provider in Terraform to "ignore" certain tags. This list of "ignored" tags is a merger of the Cloud Center's tags (read from the SSM Parameter saved by the Cloud Center's Lambda function) and their own.
    
    ```
    provider "aws" {
    
      ignore_tags {
        keys = ["cost_centre","environment","territory","service","billing_team"]
      }
    
      [...]
    }
    ```
    

## Project Team Precision

The **final tier** in this tagging hierarchy is the **project teams**. Their primary focus is on their specific projects, and they shouldn't be burdened with the complexities of the overarching tagging strategy. While **transparency** is essential, and indeed, tag information is openly shared (given that tags are visible and not shrouded in secrecy), it's not the project teams' **responsibility** to manage or be overly concerned with them.

These teams have the liberty to add **project-specific tags** using their primary tool, Terraform. However, there's a catch: they can **only** use Terraform through the **pipeline** provided by their respective Platform Team. This constraint is in place because individual user accounts have very limited permissions, typically read-only. This restriction ensures that resources are not proliferated haphazardly without version control on Git, avoiding the pitfalls of [ClickOps](https://docs.cloudposse.com/glossary/clickops/).

The beauty of the pipeline's design is its runtime instruction to Terraform to ignore specific tags. This feature acts as a **safeguard**. Even if a team member inadvertently adds a tag in Terraform that matches an existing tag but with a different value, the pipeline ensures that the original value remains untouched and the new value is disregarded.

# Making Sense of the Tagging Puzzle

Tagging in AWS might seem like a small task, but as we've seen, it's a big deal. Getting from a plan on paper to actually tagging everything right is no walk in the park. But with a good system in place and everyone **on the same page**, it becomes a lot easier.

What's the **main lesson** here? Keep things **automated** and work together. Machines are great at repetitive tasks, so let's let them handle that. And when teams **collaborate**, the whole tagging process becomes smoother and more efficient.