Explaining boto3: How to use any AWS service with Python
Tags: python, aws, programming, serverless
Published: 2021-07-19 14:09:03

**A deep dive into boto3 and how AWS built it**.

AWS defines boto3 as a Python Software Development Kit to create, configure, and manage AWS services. In this article, we'll look at how boto3 works and how it can help us interact with various AWS services.

![](https://dashbird.io/wp-content/uploads/2021/07/Screenshot-2021-07-19-at-11.59.36-1024x765.png)

Photo by [Kindel Media](https://www.pexels.com/@kindelmedia?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels) from [Pexels](https://www.pexels.com/photo/desk-industry-technology-computer-8566472/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)

**Boto3 under the hood**
------------------------

Both, [AWS CLI](https://github.com/aws/aws-cli) and [boto3](https://github.com/boto/boto3) are built on top of [botocore](https://github.com/boto/botocore) --- a low-level Python library that takes care of everything needed to send an API request to AWS and receive a response back.

**Botocore:**

-   handles **session**, credentials, and configuration,
-   gives fine-granular **access** to all operations (*ex. ListObjects, DeleteObject*) within a specific service (*ex. S3*),
-   takes care of **serializing** input parameters, **signing requests**, and **deserializing** response data into Python dictionaries,
-   provides low-level** clients** and high-level **resource** abstractions to interact with AWS services from Python.

You can think of botocore as a **package that allows us to forget about underlying JSON specifications** and use **Python (boto3) when interacting with AWS APIs**. 

* * * * *

**Clients vs. Resources**
-------------------------

In most cases, we should use boto3 rather than botocore. Using boto3, we can choose to either interact with lower-level **clients** or higher-level object-oriented **resource** abstractions. The image below shows the relationship between those abstractions. 

![boto3, aws-cli, and botocore based on S3 as an example](https://dashbird.io/wp-content/uploads/2021/07/1_4ItsL7ZglQiAy_Xt-MSqVA.png)

Level of abstraction in boto3, aws-cli, and botocore based on S3 as an example --- image by author

To understand the difference between those components, let's look at a simple example that will demonstrate the difference between an S3 **client** and an S3 **resource**. We want to list all objects from the images directory, i.e., all objects with the `prefix images/.`

![difference between S3 client and an S3 resource](https://lh6.googleusercontent.com/Pi26HyPEfhz-nN5su6ImCFalsQQA430Td6QUKnOLkgiNXNHCbvVSWX7FxkhqspelTQDFkxViiRm5d08zLs9WHuP5CxYVgKnmQNvKIyBoeHfzjBh_5luNsRtiXbM9ss__GvG7mxZl)

Already by looking at this simple example, you can probably spot the difference:

-   with a **client**, you directly interact with **response dictionary **from a deserialized API response,
-   in contrast, with the **resource**, you interact with standard **Python classes **and objects rather than raw response dictionaries.

You can investigate the functionality of **resource **objects using `help()` and `dir()`:

![](https://lh4.googleusercontent.com/vGCxLH7mF1u-gJZayzS1VTZp26O3tl9GBtOHBCHLcdmegS8cd6T2dJFYSOjuNA-18ykrlAd3YLeC2uhKncuWm15vnrJ6waq1kjRpMsn8UitHklaPYdP1pOqvrcN9hQoejhFRmyyr)

Overall, the **resource** abstraction results in a more readable code (*interacting with Python objects rather than parsing response dictionaries*). It also handles many low-level details such as pagination. **Resource** methods usually return a generator so that you can lazily iterate over a large number of returned objects without having to worry about pagination or running out of memory.

***Fun fact:**** Both, client and resource code, are dynamically generated based on JSON models describing various AWS APIs. For clients, AWS uses ****JSON service description****, and for resource a ****resource description**** as a basis for auto-generated code. This facilitates quicker updates and provides a consistent interface across all ways you can interact with AWS (*CLI, boto3, management console*). The only real difference between the JSON service description and the final boto3 code is that PascalCase operations are converted to a more Pythonic snake_case notation.*

* * * * *

### **Why is resource often much easier to use than client**?

Imagine that you need to list thousands of objects from an S3 bucket. We could try the same approach we used in the initial code example. The only problem is that `s3_client.list_objects_v2()` method will allow us to only list a [maximum of one thousand objects](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html?highlight=list_objects_v2#S3.Client.list_objects_v2). To solve this problem, we could leverage **pagination**:

![leverage pagination
](https://lh4.googleusercontent.com/c3731OskDNUTYrReXcGij7ne283dXE9yudFRP1m5HTbGkEQfZ3cDJiBvyddf-zoW4u_oMLQt0pixfW28n1Qsdy5RKGW1Cvy6BeqYwzAPyZE2rXjb6eO4wiUPwAW1taiW5jjvzhgb)

While the paginator code is easy enough, **resource** abstraction gets the job done in just two lines of code:

![pagination in two lines](https://lh3.googleusercontent.com/GUfUDsKcewPMkcMZvgALefu6X8wzRyqnHdVzg0sg_jiUB00iQwZOx-XJ2rkdlnpeXefokDybhl0SZPl8cAh24KmY6fW_Szwvy9jAZq_qX4bPf4mvYYSWviZGJCovIvtGOqqBr_Lj)

* * * * *

### **Why you will still use clients for some of your work**?

Despite the benefits of resource abstractions, **clients** provide more functionality, as they map almost 1:1 with the AWS service APIs. Thus, you will most likely end up using both, **client** and **resource**, depending on a specific use case.

Apart from a difference in functionality, **resources** are **not thread-safe**, so if you plan to use multithreading or multiprocessing to speed up AWS operations such as file uploads, you should use clients rather than resources. More on that [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-or-multiprocessing-with-resources).

**Note:** There is a way to access client methods directly from a resource object: `s3_resource.meta.client.some_client_method()`.

* * * * *

**Waiters**
-----------

Waiters are polling the status of a specific resource until it reaches a state that you are interested in. For instance, when you create an EC2 instance using boto3, you may want to wait till it reaches a "Running" state until you can do something with this EC2 instance. Here is a sample code that shows this specific example:

![Boto3: using waiter to poll a new EC2 instance for a running state](https://dashbird.io/wp-content/uploads/2021/07/1_B8Z0jjV0h3XM3axZAhonDQ.png)

Boto3: using waiter to poll a new EC2 instance for a running state--- image by the author

Note that ImageId from the above example is different for each AWS region. You can find the ID of the AMI by following the "Launch instances" wizard in the AWS console:

![Finding AMI ID ](https://dashbird.io/wp-content/uploads/2021/07/1_4rBTdlg6i_DvLSrfEbIV7A.png)

Finding the AMI ID --- image by the author

### **A more common waiter example --- wait until a specific S3 object arrives in S3**

Let's be honest. How often are you launching new instances? **Probably not that often.** So let's build a more realistic waiter example. Imagine that your **ETL process is waiting** until a specific file **arrives in an S3 bucket**. In the example below, we wait until somebody from the marketing department will upload a file with current campaign costs. While you could implement the same with AWS Lambda using an S3 event trigger, the logic below is not tied to Lambda and can run anywhere.

![Using waiter in S3 resource](https://dashbird.io/wp-content/uploads/2021/07/1_jTDZxwW0DPnG0qkiE4IPog.png)

Using waiter in S3 resource --- image by author

Or the same using a more configurable **client** method:

![Using waiter in S3 client ](https://dashbird.io/wp-content/uploads/2021/07/1_dkeBmz3ADRepc-GX2zDKoQ.png)

Using waiter in S3 client --- image by author

As you can see from the code snippet above, using the client's waiter abstraction, we can specify:

-   **MaxAttempts**: how many times should we check whether a specific object arrived --- this will prevent zombie processes and will fail if the object hasn't arrived during the time we would expect it to arrive,
-   **Delay**: the number of seconds to wait between each attempt. 

As soon as the file arrives in S3, the script stops waiting, which allows you to do something with this newly arrived data.

* * * * *

**Collections**
---------------

Collections indicate a group of resources such as a group of S3 objects in a bucket or a group of SQS queues. They allow us to perform various actions on a group of AWS resources in a single API call. **Collections can be used to:**

-   get **all S3** **objects** with a specific object prefix:

![S3 objects with a specific object prefi](https://lh3.googleusercontent.com/ESL_mx6Q5dh82UHNLFzGG07kZoSjvzdjEqGuNIKkUdkUnuiQWSHCfTcZJmpRHm6ecZOcUS_OjKHcZuZtcID4A7-dULoeFKEwyKrkMqW7DAr5w6u_YB0VlDSpA_JXpPb2JRuhdWfG)

-   get all S3 objects with a specific content type, for example, to **find all CSV files**:

![S3 object type](https://lh4.googleusercontent.com/r_dYtWQ3G2VAgLxAIieqhvCpC6bIhkEgbaZQ4cAWyVx1ZkFxTwrkUWrWzNREzpYeYJplucl2sf705XYiAZ0INSqVmzLQfEamWdRhidcf-k-xjgcgwBZ4MFKJmGXUuX8zakJGXbRR)

-   get all S3 object versions:

![s3 object version](https://lh3.googleusercontent.com/HiLGANu_OoBE5DzS8lUnZ2pESbCNMsJzCIkNcOZDvDp_POCxuR2kn3hNGo-V8ZlBS0LEQeOl0-baTgV8pygY0oY3weasyXS7VSX12wTEZrrx44GcGrivL-C6vikQBCbubV7tmBkb)

-   specify a **chunk-size **of objects to iterate over, for instance when the default page size of 1000 objects is too large for your application:

![iterate s2 objects](https://lh5.googleusercontent.com/sbovBCQxYpWb_bJzyj_pLrLvhQs2kraI330LgYA-bcjO8Q3WIGIeHyv888fzNo_xoRzpP4I4xrJFoMmkm45mz-xqg3BFCoMD_12yQze29bQ6tvpR3HCmEfpQGJ0sK26ehGzK6AU5)

-   **delete all objects** in a single API call (*be careful about that!):*

![s3 API call](https://lh6.googleusercontent.com/qhtgaVNEMFi6VaZC5sAaaS-Nv2peh0Ozna5vI-buJAXt7TVoGzFA37iwrHmesUI9YC-0nwPNJn1zSwfO2_fuY5wBJ2aBKC-Bne8G98ZOwmAQ7ijjhipxjjOPbz5xkYUoUN46cZe4)

A more common operation is to delete all objects with a specific prefix:

![delete s3 objects](https://lh6.googleusercontent.com/PZInAzH60jRbb3z3MKZRShVpb29hsC7sU91Ox-558IP7MaMUK2G9YR6eSGpslmhoV7L-NG0hJKRMv4MZNvcCLCA5hN-VQ26fQNvSXzw9fVUV44nJkSQa-Pf9hw7KmtJMeMTgmb4i)

* * * * *

**Sessions: How to pass IAM credentials to your boto3 code?**
-------------------------------------------------------------

There are many ways you can pass access keys when interacting with boto3. Here is the order of places where boto3 tries to find credentials:

1 - Explicitly passed to `boto3.client()`, `boto3.resource()` or `boto3.Session()`:

![IAM credentials to boto3](https://lh3.googleusercontent.com/OX4RqUCvoo4T1NUkZ9tF1i69GBRALTKb5lgiocjt-C6vTCjEt5RZsvm5_UltwVrYTI0Y_0bVTv8psGa7K__GxBbJ1hqoQRh6e3-n7srXwHmab00cLIGmnZ5PYSJl-KBc0hvTT7AQ)

![IAM credentials to boto3](https://lh3.googleusercontent.com/8-oNRYRp0Soeit71L8Nq4Ky-FIdMKWiF6S4wTFaIGhQLvbaVetsE0VVFBkZh3R_5s-K8HWGJRmzc_VcLecyCm0LeKJkzOY6UWWmL71pNSy9-hsoH3LirQpCdMmOkmEK-AVIU9Ugn)

2 - Set as environment variables:

![iam credentials to boto3 variables](https://lh4.googleusercontent.com/0gfYI1b2hi0U5JF1VQslpOlPFe3JHNMgGN8sxUhLyxBUjYsW6JoNyI6XkQJMqYYXmJgU8kcigoKpYNgAG1qL5UrOOeJawXb_rXM0t3fEbi2dDvFvIeafkkLStm_2eLLlpWdipkOX)

3 - Set as credentials in the `~/.aws/credentials` file (*this file is generated automatically using aws configure in the AWS CLI*):

![iam credentials to boto3 AWS CLI](https://lh5.googleusercontent.com/mhj4FECMMscqo3aY3S34DvpXaAZvQTYCz_uXjq9YcGvJKdEo4Xhu3r89bEZ9xC4JafojlnTWlgzho_dufKn4kpaKBWrhYUpby3d6E6fcHNXHCY9QWLf4vOvVNTdF8WthKQ0TT7IZ)

4 - If you attach** IAM roles** with proper permissions to your AWS resources, you don't have to pass credentials at all but rather assign a policy with required permission scopes. Here is how it looks like in AWS Lambda:

![IAM roles AWS Lambda](https://lh5.googleusercontent.com/J0nkQX1xI791WdrE2R-G9ui6M7fY8aKQWoTgaRphkfgBotnqlE9BBz4PAkz0DnbaspvbNsMlZlB00NPMGBywQdM3LgUYHMOW2T-5Lj0IH1GMjlN3gqB6uZAwh5nWBBwE3uuX314A)

![IAM roles AWS Lambda](https://lh4.googleusercontent.com/_2vY6k3mi-avJy3522BgAQaAUM4_S3FjfHbs2GJpRy5BdwHQTqB8bA9CFaZaQM4zP5grZqDURGZXz2GpPhkC96nrUoDOaxCv13_SH0BBRwctiipaPl8Wee2gx4fzfDWNeuMdspV_)

This means that with **IAM roles** attached to resources such as Lambda functions, you don't need to manually pass or configure any long-term access keys. Instead, IAM roles are dynamically generating **temporary access keys**, making the process more secure. 

If you want to leverage **AWS Lambda **with **Python** and **boto3 **for a specific use case, have a look at the links below:

-   How to build [event-driven ETL](https://dashbird.io/blog/7-reasons-why-you-should-consider-a-data-lake/) and [event-driven data testing](https://dashbird.io/blog/data-swamps-event-driven-data-testing/),
-   How to [extract text from images using Amazon Rekognition](https://dashbird.io/blog/can-data-lakes-accelerate-ml-data-pipelines/),
-   How to [build decoupled services using SNS, SQS and Kinesis](https://dashbird.io/blog/kinesis-sqs-sns-comparison/),
-   How to [use NoSQL DynamoDB to read and write data](https://dashbird.io/blog/nosql-database-data-engineering/),
-   How to [manage credentials using Secrets Manager](https://dashbird.io/blog/aws-secrets-manager-python/).

A useful feature of AWS Lambda is that boto3 is already preinstalled in all Python runtime environments. This way, you can run any of the examples from this article directly in your Lambda function. Just make sure to add proper policy corresponding to the service you want to use in your Lambda's IAM role:

![Creating a function in AWS Lambda](https://dashbird.io/wp-content/uploads/2021/07/1_zM9-5HSD38jfNA0tZpoEGg-1024x736.png)

Creating a function in AWS Lambda--- image by author

![Testing boto3 from AWS Lambda](https://dashbird.io/wp-content/uploads/2021/07/1_HQX0mcj0JhHLFE5ZPIhcxA-958x1024.png)

Testing boto3 from AWS Lambda --- image by author

If you plan to run a number of Lambda functions in production, you may explore [Dashbird](https://dashbird.io/) --- an observability platform that will help you [monitor and debug your serverless workloads](https://dashbird.io/serverless-observability/). It's particularly valuable for building [automated alerts on failure](https://dashbird.io/failure-detection/), [grouping related resources](https://dashbird.io/blog/group-aws-lambda-functions-dashbird-project/) based on a **project** or domain, providing an **overview of all serverless resources**, interactively **browsing through the logs**, and **visualizing operational bottlenecks**.

The tool is completely free to use and only takes [2 minutes to set up](https://dashbird.io/docs/quickstart/setting-up-dashbird/) -- and you can start exploring your data immediately.

![serverless resources](https://dashbird.io/wp-content/uploads/2021/07/0_0_uROQ2_Ip_QEC5g.png)

Overview of all serverless resources provided by Dashbird --- image courtesy by [Dashbird.io](https://dashbird.io/)

* * * * *

**How to change a default boto3 session?**
------------------------------------------

Boto3 makes it easy to change the default session. For instance, if you have several profiles (*such as one for dev and one for prod AWS environment*), you can switch between those using a single line of code:

![default to boto3](https://lh3.googleusercontent.com/hJgb1GV9usgMdvKZ9ujCQwlbwNoxYbCDOOOTe8WDOtro4wgsIpFMHjRBAirk6UULaUhy8lfFFq-R7xPjfjE4lu102K8BZWLww5p3J_VJlFYDesYiG__naU4ZNMKUFNMjRpBlD9a7)

Alternatively, you can attach credentials directly to the default session, so that you don't have to define them separately for every new client or resource.

![default to boto3](https://lh3.googleusercontent.com/G1ZDUeTVufbKTHyyuUQ6MP2NrDpC3Zh7cFHBTuqASEdhtQsmLl-aSPS2iEZ4LIErfgRyOglwI01HNoc5i6DJEG1wBzI8dzmLEjgiHSHcIuOrT_k2OgQ0XqysHdmPIU5pEfmorXJn)

* * * * *

### **Conclusion**

In this article, we looked at how to use boto3 and how it is **built internally**. We examined the differences between **clients** and **resources** and investigated how each of them handles **pagination**. We explored how **waiters** can help us poll for specific status of AWS resources before proceeding with other parts of our code. We also looked at how **collections** allow us to perform actions on multiple AWS objects. Finally, we explored different ways of providing credentials to boto3 and how those are handled using IAM roles and user-specific access keys.

**Thank you for reading!**

* * * * *

*References and further reading:*

[AWS re:Invent 2014 | (DEV307) Introduction to Version 3 of the AWS SDK for Python (Boto)](https://www.youtube.com/watch?v=Cb2czfCV4Dg)

[Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)

[How I manage Credentials in Python using AWS Secrets Manager?](https://dashbird.io/blog/aws-secrets-manager-python/)

[Performance monitoring for AWS Lambda](https://dashbird.io/blog/performance-monitoring-for-aws-lambda/)