Directly upload large files to the cloud with Anvil
Tags: webdev, python, programming, tutorial
Published: 2023-03-30 16:16:55

You might need to build a web app that allows users to upload very large files. If you were building an [Anvil](https://anvil.works/build?utm_source=crosspost:dev.to:/blog/direct-s3-upload) app, you would normally use a [FileLoader](https://anvil.works/docs/client/components/basic#fileloader?utm_source=crosspost:dev.to:/blog/direct-s3-upload) component to capture the user's file then store the file in a [Data Table](https://anvil.works/docs/tables?utm_source=crosspost:dev.to:/blog/direct-s3-upload). But if your file is many gigabytes in size, you might see timeouts while uploading or exceed the storage limits of your Data Tables. Instead, it makes sense to store giant files directly in a cloud storage service, such as Amazon S3. 

Fortunately, Anvil makes it easy to use third-party cloud services on the back end, and Javascript libraries on the front end. In this example, we'll walk through how to use the use the [`uppy.io` JavaScript widget](https://uppy.io/) and [Amazon `boto3` library](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) to upload files from Anvil directly to an S3 bucket.


> New to Anvil? Anvil is a platform for building full-stack web applications with nothing but Python. This is a fairly advanced how-to guide -- you might want to [start with a quick tutorial](https://anvil.works/learn/tutorials/feedback-form?utm_source=crosspost:dev.to:/blog/direct-s3-upload).


Check out the app we're going to build by [cloning the source code](https://anvil.works/build?utm_source=crosspost:dev.to:/blog/direct-s3-upload#clone:HEQEC5XZDJO3DONU%3dIT2JKSLWJOXJKFPX3SMLUBCI).

We'll build the app in five steps:

1. First, we'll [create a new Anvil app](#step-1-create-an-anvil-app).
2. We'll then [set up an Amazon S3 bucket](#step-2-set-up-an-s3-bucket) with the proper permissions.
3. Next, we'll [add the Uppy widget](#step-3-add-the-uppy-widget-to-the-app) to our app.
4. We then need to [get a presigned URL](#step-4-get-a-presigned-url-from-s3-using-boto3) using Amazon's `boto3` library.
5. Finally, we'll [provide the presigned URL to Uppy](#step-5-provide-the-presigned-url-to-uppy) which will upload the file to S3.

## Step 1: Create an Anvil app

First, log into [Anvil](https://anvil.works/login?utm_source=crosspost:dev.to:/blog/direct-s3-upload) and create a new blank app. We'll need to make use of our app's [App Secrets](https://anvil.works/docs/security/encrypting-secret-data) to store the AWS access keys that we'll acquire in the next step, so ahead and add the App Secrets service to your app.

![Adding the App Secrets service in the Anvil Editor](https://anvil.works/blog/img/upload-large-files-to-s3/add-app-secrets.png)

## Step 2: Set up an S3 bucket

If you don't already have an S3 bucket, you'll need to set one up. You can find a get started guide here: [Getting Started with Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html). 

Your S3 bucket will need to have [cross-origin access enabled](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html). The following cross-origin policy is sufficient to allow uploads from browsers – you can paste it into the CORS policy text box in the S3 console:

```json
[{"AllowedHeaders": ["content-type"], "AllowedMethods": ["POST", "PUT"], "AllowedOrigins": ["*"], "ExposeHeaders": []}]
```

You'll also need an [AWS Access Key](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/) and a [Secret Access Key](https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/) that has permission to write to your S3 bucket.

Save your access keys in your app's [App Secrets](https://anvil.works/docs/security/encrypting-secret-data?utm_source=crosspost:dev.to:/blog/direct-s3-upload). Store the AWS access key ID in a secret named `aws_key_id` and the secret key in a secret named `aws_secret_key`. 

![AWS keys added as App Secrets in an Anvil app](https://anvil.works/docs/how-to/img/upload-large-files-to-s3/aws-app-secrets.png)

## Step 3: Add the Uppy widget to the app

Uppy is a third-party JavaScript that displays a form where users can drag and drop files to upload to S3. We can add the widget to an Anvil app using [Native Libraries](https://anvil.works/docs/client/javascript#using-native-javascript-libraries?utm_source=crosspost:dev.to:/blog/direct-s3-upload) and the [`anvil.js` module](https://anvil.works/docs/client/javascript?utm_source=crosspost:dev.to:/blog/direct-s3-upload). 

First, add the following line to your app's Native Libraries:

```html
<link href="https://releases.transloadit.com/uppy/v3.3.1/uppy.min.css" rel="stylesheet">
```

Next, go to the Design View of your Form, add a ColumnPanel, and name it `uppy_target`. This is where the Uppy widget will go. 

Switch to Code View and add the following import statements to your Form's code: 

```python
from anvil.js import window, import_from, get_dom_node
import anvil.js
```

We'll need to set up two functions for the Uppy widget to use: 

* `get_upload_parameters` will return AWS credentials from a Server Module that will give Uppy temporary permission to access the S3 bucket. 

* `on_complete` will be called by Uppy when the upload completes, whether or not it's successful.

For now, the functions won't do anything, but we'll fix that in Steps 4 and 5.

```python
  def get_upload_parameters(self, file_descr):
    # This function is called to get the pre-signed URL
    # for the S3 upload.
    pass
    
    
  def on_complete(self, result, *args):
    # This function is called when the upload is finished,
    # successful or not.
    pass
```

Next, in the Form's `__init__` method, add the following code: 

```python
    mod = anvil.js.import_from("https://releases.transloadit.com/uppy/v3.3.1/uppy.min.mjs")
    self.uppy = mod.Uppy().use(mod.Dashboard, {
      'inline': True,
      'target': get_dom_node(self.uppy_target)
    })
    self.uppy.use(mod.AwsS3, {
      'getUploadParameters': self.get_upload_parameters
    })
    self.uppy.on('complete', self.on_complete)
```

If you run the app now, you'll see an empty Uppy uploader. Nothing will happen if you try to upload a file because we haven't set up the `get_upload_parameters` function yet. In the next step, we'll use `boto3` to return the AWS credentials we need.

![The uppy widget added in an app](https://anvil.works/docs/how-to/img/upload-large-files-to-s3/uppy-widget.png)

## Step 4: Get a presigned URL from S3 using boto3

The AWS access key we set up in [Step 2](#step-2-set-up-an-s3-bucket) will provide access to S3, but it's a powerful credential for this very reason. We don't want to [expose it to the client](https://anvil.works/docs/security#client-vs-server-code?utm_source=crosspost:dev.to:/blog/direct-s3-upload) and give anyone unlimited access to our S3. 

Instead, we can generate a presigned URL, which grants temporary access to upload one file to the target S3 bucket. We can use the `boto3` library from a Server Module to generate the presigned URL.

> We can trust the server code with powerful AWS credentials, because we have control over the server. Because the client is controlled by anyone who visits our app, we can't trust it with anything except a time-limited, single-purpose presigned URL.  Read more about client and server security [here](https://anvil.works/articles/client-vs-server?utm_source=crosspost:dev.to:/blog/direct-s3-upload).


In a Server Module, add the required import statements and set up variables for your S3 bucket and AWS region:

```python
import boto3, string
from datetime import datetime
from random import SystemRandom

BUCKET_NAME="anvil-upload-demo" # Replace with your S3 bucket name
REGION_NAME="eu-west-2" # Replace with your AWS region
```

Next, we'll need to create a function to generate the presigned URL and make it callable from the client. This function will be called in the `get_upload_parameters` function we created in the previous step:

```python
@anvil.server.callable
def get_presigned_url(filename, content_type):
  s3_client = boto3.client(
      's3',
      aws_access_key_id=anvil.secrets.get_secret('aws_key_id'),
      aws_secret_access_key=anvil.secrets.get_secret('aws_secret_key'),
      region_name=REGION_NAME
  )

  # Because the filename is user-supplied, we use the current date and a random string to ensure
  # a unique filename for each upload
  random_string = ''.join(SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(10))
  filename_to_upload = f"{datetime.now().isoformat()}-{random_string}-{filename}"
  
  r = s3_client.generate_presigned_post(BUCKET_NAME, filename_to_upload)

  # Return the data in the format Uppy needs
  return {
    'url': r['url'],
    'fields': r['fields'],
    'method': 'POST'
  }
```

## Step 5: Provide the presigned URL to Uppy

When Uppy is ready to upload a file, it will call the `get_upload_parameters` function. We can now change the function so that it calls the `get_presigned_url` server function and returns the results (the pre-signed URL and associated data) to Uppy.

Because we're using this function as a callback from Javascript, we need to add the [`@anvil.js.report_exceptions` decorator](https://anvil.works/docs/client/javascript/accessing-javascript#using-python-functions-as-callbacks?utm_source=crosspost:dev.to:/blog/direct-s3-upload). This makes sure that if the code throws an exception, it appears correctly in the Anvil console.

Go back to your Form's client code and update `get_upload_parameters`:

```python
  @anvil.js.report_exceptions
  def get_upload_parameters(self, file_descr):
    return anvil.server.call('get_presigned_url', file_descr['name'], file_descr['type'])
```

Provided you've configured your bucket and access keys correctly, you should now be able to upload files to S3. 

Optionally, we can configure the `on_complete` function to report feedback from Uppy. We can add this to an alert to display to the user:

```python
  @anvil.js.report_exceptions
  def on_complete(self, result, *args):
    successful = result['successful']
    failed = result['failed']
    alert(f"{len(successful)} succeeded, {len(failed)} failed")
```

![The final app](https://anvil-website-static.s3.eu-west-2.amazonaws.com/blog/upload-large-files-to-s3/s3-upload-finished-app.gif)

## That's it!

That's all we need to upload files directly from our user's browser into S3.

You can copy the sample app [here](https://anvil.works/build?utm_source=crosspost:dev.to:/blog/direct-s3-upload#clone:HEQEC5XZDJO3DONU%3dIT2JKSLWJOXJKFPX3SMLUBCI). Of course, you'll need your own S3 bucket and access keys for it work.

For our next steps, consult the [Uppy documentation](https://uppy.io/docs/) for how to customise the Uppy upload widget, or the [boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) for how to handle your file now it's safely uploaded into S3.


### Using other cloud storage services

Amazon S3 isn't the only cloud storage service out there. But because it was the first, a lot of other services use the same API. You can adapt the sample code from this example to use other services, such as Cloudflare R2, DigitalOcean Spaces or Backblaze B2.

Just check your preferred service's documentation for how to use it with `boto3`. (For example, here's [CloudFlare's page on configuring boto3](https://developers.cloudflare.com/r2/examples/boto3/) to work with their R2 service.)


---

## Build your own app with Anvil 

If you're new here, welcome! [Anvil](https://anvil.works?utm_source=crosspost:dev.to:/blog/direct-s3-upload) is a platform for building full-stack web apps with nothing but Python. No need to wrestle with JS, HTML, CSS, Python, SQL and all their frameworks – just **build it all in Python**.

[Sign up for Anvil!](https://anvil.works/build?utm_source=crosspost:dev.to:/blog/direct-s3-upload)