AWS S3 in Elixir with ExAws
Tags: aws, s3, elixir, exaws
Published: 2019-12-04 16:06:25

---
title: AWS S3 in Elixir with ExAws
published: true
date: 2019-12-04 16:00:00 UTC
tags: aws, s3, elixir, exaws
description: We setup the AWS account, configure ExAws, put, list, get and delete objects. Upload large files with multipart uploads, generate presigned urls and process large S3 objects on the fly.
canonical_url: https://www.poeticoding.com/aws-s3-in-elixir-with-exaws/
---


{% youtube gSGu5wXeS1c %}


In this article we see how to store and retrieve files on [AWS S3](https://aws.amazon.com/s3/) using Elixir and with the help of [ExAws](https://github.com/ex-aws/ex_aws).

We start by setting up an AWS account and credentials, configure an Elixir application and see the basic upload and download operations with small files.

Then, we see how to deal with large files, making multipart uploads and using presigned urls to create a download stream, processing data on the fly.

## Create an IAM user, configure permissions and credentials

If you don't have an Amazon Web Services account yet, you can create it on [https://aws.amazon.com/](https://aws.amazon.com/) and use the [free tier for the first 12 months](https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Categories=categories%23storage&awsf.Free%20Tier%20Types=tier%2312monthsfree&all-free-tier.q=s3&all-free-tier.q_operator=AND) where you have up to 5GB of free S3 storage.

Be sure you **check all the limits of the free tier** before you start using the service. Always take a look at the billing page to keep track of the usage.

To access to AWS S3 resources, first we need to create an AWS IAM (Identity and Access Management) user with limited permissions.

Once logged into the [AWS console](https://console.aws.amazon.com/console/home), go to the _users_ section of the _security credentials_ page and click on _Add user_.

[![Menu on the top-right side of the AWS console](https://www.poeticoding.com/wp-content/uploads/2019/11/1_1_aws_security_credentials_menu-1.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_1_aws_security_credentials_menu-1.jpg)

Menu on the top-right side of the AWS console

[![Create a new IAM user](https://www.poeticoding.com/wp-content/uploads/2019/11/1_2_aws_security_credentials_users.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_2_aws_security_credentials_users.jpg)

Create a new IAM user

When creating a user, we need to set a username and most importantly enable the _Programmatic access_: this means the user can programmatically access to the AWS resources via API.

[![Username and Programmatic access](https://www.poeticoding.com/wp-content/uploads/2019/11/1_3-add_user.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_3-add_user.jpg)

Username and Programmatic access

Then we set the permissions, attaching the `AmazonS3FullAccess` policy and limiting the user to just the S3 service.

[![AmazonS3FullAccess policy](https://www.poeticoding.com/wp-content/uploads/2019/11/1_4-permissions.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_4-permissions.jpg)

AmazonS3FullAccess policy

Now, this policy is fine for this demo, but it's still too broad: a user, or an app, can access to all the buckets, files and settings of S3.

By creating a custom policy, we can limit the user permissions to only the needed S3 actions and buckets. More on this at [AWS User Policy Examples](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-policies-s3.html)

Once the user is created, we can download the _Access Key Id_ and the _Secret Access Key._ You must **keep these keys secret** because whoever has them can access to your AWS S3 resources.

[![IAM user Access key ID and Secret access key](https://www.poeticoding.com/wp-content/uploads/2019/11/1_5-credentials-page-1.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_5-credentials-page-1.jpg)

IAM user Access key ID and Secret access key

To create an S3 bucket using the AWS console, go to the S3 section and click on _Create bucket_, set a bucket name (I've used `poeticoding-aws-elixir)` and be sure to block all the public access.

[![Bucket name and region](https://www.poeticoding.com/wp-content/uploads/2019/11/1_6_2-bucket-name-and-region.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_6_2-bucket-name-and-region.jpg)

Bucket name and region

[![Block all public access](https://www.poeticoding.com/wp-content/uploads/2019/11/1_6_3-block-all-public-access.jpg)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_6_3-block-all-public-access.jpg)

Block all public access

## Configure `ex_aws` and environment variables

Let's create a new Elixir application and add the dependencies to make [ex\_aws](https://github.com/ex-aws/ex_aws) and [ex\_aws\_s3](https://github.com/ex-aws/ex_aws_s3) work

```elixir
# mix.exs
def deps do
  [
    {:ex_aws, "~> 2.1"},
    {:ex_aws_s3, "~> 2.0"},
    {:hackney, "~> 1.15"},
    {:sweet_xml, "~> 0.6"},
    {:jason, "~> 1.1"},
  ]
end

```

ExAws, by default, uses [hackney](https://github.com/benoitc/hackney) HTTP Client to make requests to AWS.

We create the `config/config.exs` configuration file, where we set access id and secret access keys

```elixir
# config/config.exs

import Config

config :ex_aws,
  json_codec: Jason,
  access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
  secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"}

```

The default ExAws JSON codec is [Poison](https://github.com/devinus/poison). If we want to use another library, like [Jason](https://github.com/michalmuskala/jason), we need to explicitly set the `jason_codec` property.

We don't want to write our keys in the configuration file. First, because who has access to the code can see them, second because we want to make them easy to change.

We can use environment variables: by passing `{:system, "AWS_ACCESS_KEY_ID"}` and `{:system, "AWS_SECRET_ACCESS_KEY"}` tuples the application gets the keys from the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.

In case you are on a Unix/Unix-like system (like MacOs or Linux), you can set these environment variables in a script

```bash
# .env file
export AWS_ACCESS_KEY_ID="your access key"
export AWS_SECRET_ACCESS_KEY="your secret access key"

```

and load them with `source`

```bash
$ source .env
$ iex -S mix

```

**Keep this script secret**. If you are using _git_ remember to put this script into _.gitignore_ to avoid to commit this file.

If you don't want to keep these keys in a script, you can always pass them when launching the application or `iex`

```bash
$ AWS_ACCESS_KEY_ID="..." \
  AWS_SECRET_ACCESS_KEY="..." \
  iex -S mix

```

In case you're on a Windows machine, you can set the environment variables using the command prompt or the PowerShell

```powershell
# Windows CMD
set AWS_ACCESS_KEY_ID="..."

# Windows PowerShell
$env:AWS_ACCESS_KEY_ID="..."

```

## Listing the buckets

Now we have everything ready: credentials, application dependencies and `ex_aws` configured with environment variables. So let's try the first request.

```elixir
# load the environment variables
$ source .env

# run iex
$ iex -S mix

iex> ExAws.S3.list_buckets()
%ExAws.Operation.S3{
  http_method: :get,
  parser: &ExAws.S3.Parsers.parse_all_my_buckets_result/1,
  path: "/",
  service: :s3,
  ...,
}

```

The `ExAws.S3.list_buckets()` function doesn't send the request itself, it returns an `ExAws.Operation.S3` struct. To make a request we use `ExAws.request` or `ExAws.request!`

```elixir
iex> ExAws.S3.list_buckets() |> ExAws.request!()

%{
  body: %{
    buckets: [
      %{
        creation_date: "2019-11-25T17:48:16.000Z",
        name: "poeticoding-aws-elixir"
      }
    ],
    owner: %{ ... }
  },
  headers: [
    ...
    {"Content-Type", "application/xml"},
    {"Transfer-Encoding", "chunked"},
    {"Server", "AmazonS3"},
    ...
  ],
  status_code: 200
}

```

`ExAws.request!` returns a map with the HTTP response from S3. With [get\_in/2](https://hexdocs.pm/elixir/Kernel.html#get_in/2) we can get just the bucket list

```elixir
ExAws.S3.list_buckets()
|> ExAws.request!()
|> get_in([:body, :buckets])

[%{creation_date: "2019-11-25T17:48:16.000Z", name: "poeticoding-aws-elixir"}]

```

## put, list, get and delete

With `ExAws`, the easiest way to upload a file to S3 is with [ExAws.S3.put\_object/4](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#put_object/4)

```elixir
iex> local_image = File.read!("elixir_logo.png")
<<137, 80, 78, 71, 13, 10, 26, 10, 0, 0, ...>>
    
iex> ExAws.S3.put_object("poeticoding-aws-elixir", "images/elixir_logo.png", local_image) \
...> |> ExAws.request!()

%{
  body: "",
  headers: [...],
  status_code: 200
}

```

The first argument is the _bucket name_, then we pass the _object key_ (the path) and the third is the file's content, `local_image`. As a fourth argument we can pass a [list of options](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#t:put_object_opts/0) like storage class, meta, encryption etc.

Using the AWS management console, on the S3 bucket's page, we can see the file we've just uploaded.

[![Uploaded file visible on AWS Management console](https://www.poeticoding.com/wp-content/uploads/2019/11/1_7-file-present-in-the-bucket.png)](https://www.poeticoding.com/wp-content/uploads/2019/11/1_7-file-present-in-the-bucket.png)

Uploaded file visible on AWS Management console

We list the bucket's objects with [ExAws.S3.list\_objects](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#list_objects/2)

```elixir
iex> ExAws.S3.list_objects("poeticoding-aws-elixir") \
...> |> ExAws.request!() \
...> |> get_in([:body, :contents]) \

[
  %{
    e_tag: "\"...\"",
    key: "images/elixir_logo.png",
    last_modified: "2019-11-26T14:40:34.000Z",
    owner: %{ ... }
    size: "29169",
    storage_class: "STANDARD"
  }
]

```

Passing the bucket name and object key to [ExAws.S3.get\_object/2](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#get_object/2), we get the file's content.

```elixir
iex> resp = ExAws.S3.get_object("poeticoding-aws-elixir", "images/elixir_logo.png") \
...> |> ExAws.request!()
    
%{
  body: <<137, 80, 78, 71, 13, 10, 26, ...>>,
  headers: [
    {"Last-Modified", "Tue, 26 Nov 2019 14:40:34 GMT"},
    {"Content-Type", "application/octet-stream"},
    {"Content-Length", "29169"},
    ...
  ],
  status_code: 200
}

```

The request returns a response map with the whole file's content in `:body`.

```elixir
iex> File.read!("elixir_logo.png") == resp.body
true

```

We can delete the object with [ExAws.S3.delete\_object/2](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#delete_object/3).

```elixir
iex> ExAws.S3.delete_object("poeticoding-aws-elixir", "images/elixir_logo.png") \
...> |> ExAws.request!()

%{
  body: "",
  headers: [
    {"Date", "Tue, 26 Nov 2019 15:04:35 GMT"},
    ...
  ],
  status_code: 204
}

```

After listing again the objects we see, as expected, that the bucket is now empty.

```elixir
iex> ExAws.S3.list_objects("poeticoding-aws-elixir") 
...> |> ExAws.request!() 
...> |> get_in([:body, :contents])

[]

```

## Multipart upload and large files

The image in the example above is just ~30 KB and we can simply use `put_object` and `get_object` to upload and download it, but there are some limits:

- with these two functions the file is fully kept in memory, for both upload and download.
- [put\_object uploads the file in a single operation](https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html) and we can upload only objects up to 5 GB in size.

S3 and ExAws client support [multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html). It means that a file is divided in parts (5 MB parts by default) which are sent separately and in parallel to S3! In case the part's upload fails, ExAws retries the upload of that 5 MB part only.

With multipart uploads we can upload objects from 5 MB to 5 TB - ExAws uses file streams, avoiding to keep the whole file in memory.

Let's consider `numbers.txt`, a [relatively large txt file](https://www.poeticoding.com/elixir-stream-and-large-http-responses-processing-text/#text-response) we've already seen in [another article - Elixir Stream and large HTTP responses: processing text](https://www.poeticoding.com/elixir-stream-and-large-http-responses-processing-text/) ( you can download from this url `https://www.poeticoding.com/downloads/httpstream/numbers.txt`).

`numbers.txt` size is **125 MB**, much smaller than the 5GB limit imposed by the single PUT operation. But to me this file is large enough to benefit from a multipart upload!

```elixir
iex> ExAws.S3.Upload.stream_file("numbers.txt") \
...> |> ExAws.S3.upload("poeticoding-aws-elixir", "numbers.txt") \
...> |> ExAws.request!()

# returned response
%{
  body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<CompleteMultipartUploadResult>...",
  headers: [
    {"Date", "Tue, 26 Nov 2019 16:34:08 GMT"},
    {"Content-Type", "application/xml"},
    {"Transfer-Encoding", "chunked"},
  ],
  status_code: 200
}

```

- First we create a file stream with [ExAws.S3.Upload.stream\_file/2](https://hexdocs.pm/ex_aws_s3/ExAws.S3.Upload.html#stream_file/2)
- The stream is passed to [ExAws.S3.upload/4](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#upload/4), along with _bucket name_ and _object key_
- `ExAws.request!` initialize the multipart upload and uploads the parts

To have an idea of what ExAws is doing, we can enable the debug option in the `ex_aws` configuration

```elixir
# config/config.exs
   
config :ex_aws,
 debug_requests: true,
 json_codec: Jason,
 access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
 secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"}

```

We should see multiple parts being sent at the same time

```bash
17:11:24.586 [debug] ExAws: Request URL: "...?partNumber=2&uploadId=..." ATTEMPT: 1
17:11:24.589 [debug] ExAws: Request URL: "...?partNumber=1&uploadId=..." ATTEMPT: 1

```

## Multipart upload timeout

When the file is large, the upload could take time. To upload the parts concurrently, ExAws uses Elixir [Tasks](https://hexdocs.pm/elixir/Task.html) - the default timeout for part's upload is set to 30 seconds, which could not be enough with a slow connection.

```bash
** (exit) exited in: Task.Supervised.stream(30000)
    ** (EXIT) time out

```

We can change the timeout by passing a new `:timeout` to `ExAws.S3.upload/4`, 120 seconds in this example.

```elixir
ExAws.S3.Upload.stream_file("numbers.txt")
|> ExAws.S3.upload(
  "poeticoding-aws-elixir", "numbers.txt", 
  [timeout: 120_000]) 
|> ExAws.request!()

```

## Download a large file

To download a large file it's better to avoid `get_object`, which holds the whole file content in memory. With [ExAws.S3.download\_file/4](https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#download_file/4) instead, we can download the data in chunks saving them directly into a file.

```elixir
ExAws.S3.download_file(
  "poeticoding-aws-elixir", 
  "numbers.txt", "local_file.txt"
) 
|> ExAws.request!()

```

## presigned urls and download streams - process a file on the fly

Unfortunately we can't use `ExAws.S3.download_file/4` to get a download stream and process the file on the fly.

However, we can generate a [presigned url](https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) to get a unique and temporary URL and then download the file with a library like [mint](https://github.com/elixir-mint/mint) or [HTTPoison](https://github.com/edgurgel/httpoison).

```elixir
iex> ExAws.Config.new(:s3) \
...> |> ExAws.S3.presigned_url(:get, "poeticoding-aws-elixir", "numbers.txt")

{:ok, "https://...?X-Amz-Credential=...&X-Amz-Expires=3600"}

```

By default, the URL expires after one hour - with the `:expires_in` option we can set a different expiration time (in seconds).

```elixir
iex> ExAws.Config.new(:s3) \
...> |> ExAws.S3.presigned_url(:get, "poeticoding-aws-elixir", 
   "numbers.txt", [expires_in: 300]) # 300 seconds

{:ok, "https://...?X-Amz-Credential=...&X-Amz-Expires=300"}

```

Now that we have the URL, we can use Elixir Streams to process the data on the fly and calculate the sum of all the lines `numbers.txt`. [In this article you find the HTTPStream's code and how it works](https://www.poeticoding.com/elixir-stream-and-large-http-responses-processing-text/).

```elixir
# generate the presigned URL
ExAws.Config.new(:s3) 
|> ExAws.S3.presigned_url(:get, "poeticoding-aws-elixir", "numbers.txt")

# returning just the URL string to the next step
|> case do
  {:ok, url} -> url
end

# using HTTPStream to download the file in chunks
# getting a Stream of lines
|> HTTPStream.get()
|> HTTPStream.lines()

## converting each line to an integer
|> Stream.map(fn line-> 
  case Integer.parse(line) do
    {num, _} -> num
    :error -> 0
  end
end)

## sum the numbers
|> Enum.sum()

|> IO.inspect(label: "result")

```

In the first two lines we generate a presigned url. Then, with [HTTPStream.get](https://github.com/poeticoding/httpstream_articles/blob/master/lib/http_stream.ex#L3) we create a stream that lazily downloads the file chunk by chunk, transforming chunks into lines with [HTTPStream.lines](https://github.com/poeticoding/httpstream_articles/blob/master/lib/http_stream.ex#L56), mapping the lines into integers and summing all the numbers. The result should be _12468816_.