How to copy files to s3 using boto3
Tags: python, aws, boto3, devops
Published: 2020-03-01 23:43:46

This article was also posted on [razcodes.dev](https://razcodes.dev/how-to-copy-files-to-s3-using-boto3.html "razcodes.dev").

[Boto](https://github.com/boto/boto3 "Boto") is a the AWS SDK for Python. It's a library that allows you to interact with the different AWS services. You can use it either on a computer/server to run all sorts of automation or to write lambda functions in your AWS account.

In this article I will go over a simple case scenario of copying a file from the local machine to an s3 bucket. This can be easily adapted into creating automatic backups in the cloud.

AWS offers a free tier and if you are interested in their 12 month free you can [create an account](https://aws.amazon.com/free "AWS Free Tier").

## Creating the bucket

You can create the bucket either from the console or using the [AWS CLI](https://aws.amazon.com/cli/ "AWS CLI"). I used the console for this one, so I logged into my AWS account. Under _Services_ chose _S3_, then '**Create Bucket**'.

Buckets in S3 must have unique names across AWS so you will only be able to give it a name that does not already exists. The interface will tell you that. I named mine _razbackupbucket_ , so chose a name, then click **Create**. 

## Provisioning credentials

In order for us to use the AWS services we will need to setup access credentials. To do that, under _Services_ again go to _IAM_. 

Here under _Users_ click on _Add User_. For the username I chose 'svc-s3' (the name is more for you than anything). I also selected _Programmatic access only_, as the user will not need access to the console. Click _Next: Permissions_.

For permissions I selected '**Attach existing policies directly**', then when you type  s3 in the filter, you can select **AmazonS3FullAccess**. Next: Tags. Next: Review. **Create user**.

It is very important on this screen to either download the .csv file or copy the _Access key ID_ and _Secret access key_ because once you press **Close**, you will never be able to see the Secret access key ever again.

## Setup Python environment

On the computer now, in your work folder, create a new directory where everything will be created. I like to have virtual environments for every project and to keep things separated, so we will create one. After that we will install boto3 as well as python-dotenv to store out credentials properly as environment variables.

```bash
mkdir s3-project
cd s3-project
python3 -m venv venv
source ./venv/bin/activate
pip install boto3
pip install python-dotenv
```

We will now create 2 files, one called '_app.py_', that will have our code, and one called '_.env_' where we can store the credentials.

```bash
touch app.py
touch .env
```

I also copied a new file in that folder called _s3.png_ that I will use to copy to my bucket.

The _.env_ file looks like this. Make sure you replace the values with the ones you got from the previous step.

```bash
AWS_ACCESS_KEY_ID=your-access-key-id
AWS_SECRET_ACCESS_KEY=your-secret-access-key
```

And finally here is the code in _app.py_ that will take the image file and upload it to the S3 bucket.

```python
import boto3 
import os 
from dotenv import load_dotenv

load_dotenv() # this loads the .env file with our credentials

file_name = 's3.png' # name of the file to upload
bucket_name = 'razbackupbucket' # name of the bucket

s3_client = boto3.client(
    's3',
    aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
    aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY')
)

response = s3_client.upload_file(file_name, bucket_name, file_name)
```

Notice that in the last line, we have the filename referenced 2 times. That is because the first one, we refer to the actual file on the disk and the second time, we chose what the name will be once it's uploaded to s3.

You can now run your program with the virtual environment still activated.

```python
python app.py
```

If you go back and check your s3 bucket and refresh, you will see the new file in there. If not, this is a good time to track back and see what did not go according to plan.

More information on boto and S3 can be found [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html "Boto S3").

Thank you for reading this and please let me know if you have any questions.