File upload to AWS S3 or S3 compatible bucket from Phoenix LiveView using Elixir
Tags: elixir, webdev, phoenix
Published: 2023-12-10 07:53:23

In this article, I will discuss how to implement LiveView file uploader from client directly to the DigitalOcean's Spaces (S3) bucket using Elixir and LiveView in your Phoenix application (but you can also use this tutorial to upload to AWS S3 or any S3-compatible bucket). 

> This article is not endorsed or paid by DigitalOcean. It is just the lack of resources on integration with other object storage buckets (than AWS S3). But also, DigitalOcean’s Spaces (disclaimer: in my opinion) a lot easier to use and has more straightforward pricing.

My configs are:

    Elixir 1.15.7
    OTP 26
    LiveView 0.20.1
    Phoenix 1.7.10
    Phoenix LiveView 0.18.18
    Ecto 3.11
    Dockerized PostgreSQL
    All running on my MacBook Air M2 8GB Ram

If you want to upload files from the server, please check out [ex_aws](https://github.com/ex-aws/ex_aws), [ex_aws_s3](https://github.com/ex-aws/ex_aws_s3),  and [AWS S3 in Elixir with ExAws](https://www.youtube.com/watch?v=gSGu5wXeS1c). Uploading from the server is very straightforward compared to uploading from the client mainly because `ex_aws` already includes all necessary request settings. If you want the user to first upload the file to the server (maybe you want to change the file a bit before storing it or generating thumbnail of the picture), then consider also using [Waffle](https://hexdocs.pm/waffle/Waffle.html) that comes with seamless integration with `ex_aws`.

However, if we want user to upload files and then store it in the S3 bucket without any changes to the file, it would be more efficient to enable users upload  the file directly
to the S3 bucket (user -> s3) instead of uploading it to the server so that the server uploads it to the S3 (user -> our server -> s3). And this is what this article focuses on.

This article is built entirely on the following resources: 
- [External uploads guide from LiveView docs](https://hexdocs.pm/phoenix_live_view/uploads-external.html)
- [Phoenix LiveView Uploads Deep Dive — 2020 article by Chris McCord (creator of Phoenix framework)](https://www.phoenixframework.org/blog/phoenix-live-view-upload-deep-dive)
- [Liveview File Uploads to S3 — YouTube video by LiveView Mastery](https://www.youtube.com/watch?v=RMyJ3qo7gVg)
- Slightly modified S3 uploader (originally coming from Chris McCord) by LiveView Mastery. [gist](https://gist.github.com/plicjo/5e5ced381f2b71d69d98b3e48885aacf)

Uploading file directly	from the client (with LiveView) is challenging because we need to do multipart HTTP request and add security verification parameters that are stored in ENV of our  Phoenix application. Luckily, Chris McCord has got us covered. 

Let’s do initial setup.

1. Create an account in DigitalOcean and create Spaces Object Storage bucket. You will need to choose the location of the server, name of the bucket. 
2. Inside of Spaces bucket page, go to “Settings” and add CORS configuration. For the sake of tutorial, you can put `*` to the origin to allow everyone an access then allow all methods, but you must immediately change the CORS configuration to the one that suits your needs best, and NEVER have `*` in the origin field
3. Then generate access key and store it somewhere safe. We will need access key so that our Phoenix application can connect and access the bucket.
4. Add the Spaces (S3) bucket credentials to the `config.exs`:


```elixir
config :your_app, 
	access_key_id: “fake”, 
	secret_access_key: "fake", 
	bucket: "fake", 
	region: "your_s3_region"
```

> Note: NEVER store your credentials in plain text or variable. You must handle the secrets through ENV variables, but that is not the focus of this article. Please refer to official docs on how to handle secrets in Elixir.

Now, let’s talk about overall architecture of things. We are going to upload the file to the Spaces (S3) bucket and will receive an URL to the file, then we are going to store that URL in database in the `string` field of the schema. For example, let’s say I am building an online menu for the restaurant. I will have `menu_items` schema and I want to upload pictures of prepared menu item there that the consumer will get. I will have to add `field :image, :string` (that will store the image URL from my Spaces bucket) to the schema `menu_items`.

When it comes implementing the file upload to the Spaces (S3) bucket in LiveView, there are several steps we have to do:

1. Get secrets from our Phoenix application and pass it to the client
2. Implement the file upload to the Spaces (S3) bucket from the client by adding some JavaScript
3. Store the file URL in database

Let’s jump into the code.

First, inside of liveview, we are going to enable the user to upload the file in the form and add a function that will fetch the secrets from Phoenix application and prepare required headers for the request client-side JavaScript will do.

In the LiveView, we are going to use `allow_upload` to allow the user upload. Then, we will add a function to `allow_upload` that will pass our secrets from Phoenix application to the client. We are going to call that function `presign_upload`. 

```elixir
 def mount(_, _, socket) do 
   socket =
      socket
      |> allow_upload(:image,
        accept: ~w(.jpg .jpeg .png),
        max_entries: 1,
        external: &presign_upload/2
      )

    {:ok, socket}
  end


  defp presign_upload(entry, %{assigns: %{uploads: uploads}} = socket) do
    meta = S3Uploader.meta(entry, uploads)
    {:ok, meta, socket}
  end
```

`presign_upload` will basically fetch secrets from out Phoenix application and pass them to the client. It will also prepare the headers that the S3 requires.

There is a dependency-free [S3Uploader](https://gist.github.com/chrismccord/37862f1f8b1f5148644b75d20d1cb073) written by Chris McCord. If you go to the official LiveView [docs](https://hexdocs.pm/phoenix_live_view/uploads-external.html#direct-to-s3) on uploading directly to S3, you will notice that `presign_upload()` there has some logic which is not ideal to have on the LiveView. So, we are going to use LiveView mastery’s [S3Uploader](https://gist.github.com/plicjo/5e5ced381f2b71d69d98b3e48885aacf) which is functionally the same as Chris McCord’s but handles the logic so that we in our LiveView our `presign_upload` looks like this:

```elixir
  defp presign_upload(entry, %{assigns: %{uploads: uploads}} = socket) do
    meta = S3Uploader.meta(entry, uploads)
    {:ok, meta, socket}
  end
```

Instead of this:

```elixir
defp presign_upload(entry, socket) do
  uploads = socket.assigns.uploads
  bucket = "phx-upload-example"
  key = "public/#{entry.client_name}"

  config = %{
    region: "us-east-1",
    access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
  }

  {:ok, fields} =
    SimpleS3Upload.sign_form_upload(config, bucket,
      key: key,
      content_type: entry.client_type,
      max_file_size: uploads[entry.upload_config].max_file_size,
      expires_in: :timer.hours(1)
    )

  meta = %{uploader: "S3", key: key, url: "http://#{bucket}.s3-#{config.region}.amazonaws.com", fields: fields}
  {:ok, meta, socket}
end
```

NOTE: To upload to the DigitalOcean’s Spaces bucket, you need to change the URLs on lines: 95, 113, 127 from `"https://#{bucket()}.s3.#{region()}.amazonaws.com”` to `https://#{bucket()}.#{region()}.digitaloceanspaces.com/`.


We add file uploading functionality for the user using `<.live_file_input upload={@uploads.image} />`

So, our LiveView HEEx will minimally look like this:

```elixir
def render(assigns) do
    ~H"""
    <div>
      <form
        phx-submit="save"
        phx-change="validate"
      >

	   <label for="name">name</label>
          <input
            type="text"
            name=“menu_item[name]"
            phx-update="ignore"
            id="name"
          />

	  <label for={@uploads.image.ref}>image</label>
          <.live_file_input upload={@uploads.image} />

        <button
          type="submit"
        >
          Submit
        </button>

	</form>
       </div>
	“””
end
```

> Important: You must bind phx-submit and phx-change on the form.

Otherwise the file upload will not work. Refer to [docs](https://hexdocs.pm/phoenix_live_view/uploads.html).

When user will hit “Submit” button, client must get the meta data (with our secrets) and initiate the upload directly to the Spaces (S3) bucket. According to the official [external upload guide from docs](https://hexdocs.pm/phoenix_live_view/uploads-external.html), we can implement that by adding this code to the `app.js`:

```javascript
let Uploaders = {}

Uploaders.S3 = function(entries, onViewError){
  entries.forEach(entry => {
    let formData = new FormData()
    let {url, fields} = entry.meta
    Object.entries(fields).forEach(([key, val]) => formData.append(key, val))
    formData.append("file", entry.file)
    let xhr = new XMLHttpRequest()
    onViewError(() => xhr.abort())
    xhr.onload = () => xhr.status === 204 ? entry.progress(100) : entry.error()
    xhr.onerror = () => entry.error()
    xhr.upload.addEventListener("progress", (event) => {
      if(event.lengthComputable){
        let percent = Math.round((event.loaded / event.total) * 100)
        if(percent < 100){ entry.progress(percent) }
      }
    })

    xhr.open("POST", url, true)
    xhr.send(formData)
  })
}

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: {_csrf_token: csrfToken}
})
```

Now, we have a LiveView with the form where you can upload a file, and on submit button, client will automatically upload the file to the Spaces (S3). Now let’s handle the part how to actually store the file url.

Let’s handle validations and submission of the form. In the LiveView we need to add: 
```elixir
 def handle_event("validate”, _, socket) do
    {:noreply, socket}
  end

  def handle_event("save", %{“menu_item” => menu_item_params}, socket) do

       uploaded_files = consume_uploaded_entries(socket, :image, fn _, entry ->
        {:ok, S3Uploader.entry_url(entry)}
      end)

    menu_item_params = case Enum.empty?(uploaded_files) do
      true -> menu_item_params
      false -> Map.put(menu_item_params, "image", uploaded_files |> List.first())
    end

    socket =
      case Context.create_menu_item(menu_item_params) do
        {:ok, _} ->

          socket
          |> put_flash(:info, "menu_item created")
          |> push_patch(to: “/success”)

        _ ->
          socket
          |> put_flash(:error, "Could not create a new menu_item")
      end

  end
```

When the user hits the submit button, image is uploaded by the client-side JS code with metadata handed over by `S3Uploader` via `presign_upload()`. The metadata from `S3Uploader` also includes a name of the file (uuid) to use for saving it to the bucket for the file uploaded. On our LiveView part of handling the submission, we are using `consume_uploaded_entries()` and generating the URL the file that would have if uploaded to the Spaces S3 bucket. Because the file is the same and its `uuid` is the same, `S3Uploader.entry_url(entry)` will result in the URL (with the name of the file) that is identical to the one passed to client-side JS. 

Next, we are going to check if any file was uploaded, then update the `menu_item_params` map. `uploaded_files` will be a list of URLs if we had many files uploaded, but as we permit only one file upload in `allow_upload`, our list will be either empty or have only one item.

Then, we are going to pass the parameters to the Context to save it in our database and handle the result of operation.

If the user has uploaded the file, `menu_item_params` will look something like this:

```elixir
%{
	“name” => “delicious uzbek plov”,
	“image” => “https://tutorial-bucket.ams3.digitaloceanspaces.com/2d1622d5-328f-4b8b-b41d-44370bafe222.jpg”
}
```

There are also couple of small improvements we can add. 

First, we can enable auto-upload. When the user chooses the file, our client-side JS will start uploading to the Spaces (S3) bucket right away. We can implement that with one line of code (thanks Chris). We will need to pass `auto_upload: true` to `allow_upload` function on mount:

```elixir
 def mount(_, _, socket) do 
   socket =
      socket
      |> allow_upload(:image,
        accept: ~w(.jpg .jpeg .png),
        max_entries: 1,
	auto_upload: true,
        external: &presign_upload/2
      )

    {:ok, socket}
  end
```

Next, we can show the preview as well as progress of file upload to the user, by adding these lines to the view:

```elixir
          <%= for entry <- @uploads.image.entries do %>
            <.live_img_preview entry={entry} width="75" />
            <div class="py-5">
                      <%= entry.progress %>%
            </div>
          <% end %>
```

What to do with the progress is up to your imagination.

If you have any suggestions or critique, please let me know.