Upload and Delete file from Amazon S3 Bucket in Go using Presigned URLs
Tags: go, aws, webdev
Published: 2024-04-22 06:25:50

## Introduction
This article will demonstrate how to securely upload and delete files from Amazon S3 storage using signed urls using the golang sdk.
Amazon Simple Storage Service (Amazon S3) is an object storage service. 

## Prerequsites
* An aws s3 bucket where we will be storing our files. 
* An access key and secret access key that have been granted to a user with the necessary permission to upload and delete objects from the s3 bucket.

## Benefits of Using Presinged URLS
* Controlled Access - set an expiration time for the URL, ensuring temporary access
* Security - accesss is limited to specific s3 objects, reducing the risk of unauthorized access 

### Demonstration 
We will create a REST api that has two routes, `upload` route that receives a file that will be resized and uploaded to the bucket and `delete` which will take in the filename to be deleted and delete it.

#### Setup the API
Create a go project using `go mod init <name_of_project>`. Once done create a `main.go` file and create the following functions.

```{go}
func main() {
	fmt.Println("hello world")
	setupRoutes()
}

func setupRoutes() {
	http.HandleFunc("/upload", UploadHandler())
	http.HandleFunc("/delete", DeleteHandler())
	http.ListenAndServe(":8080", nil)
}

func UploadHandler() http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {

	}
}

func DeleteHandler() http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		
		
	}
}
```

We have setup an http routes to handle the delete and upload action for our project 

#### Setup Environment Variables 
We are going to load our environment variables from a `.env` file. Create `.env` file in the root of the project and fill it with bucket details in this format

```{json}
AWS_BUCKET_NAME="<bucket-name-here>"
AWS_REGION="<aws-region-here>"
AWS_S3_BUCKET_ACCESS_KEY="<aws-bucket-access-key-here>"
AWS_S3_BUCKET_SECRET_ACCESS_KEY="<aws-bucket-secret-key-here>"
```
Once environment variables are setup we need load them into our project. For this this i will use [viper](https://github.com/spf13/viper)

Create a folder named utils. Inside the folder create a file named config.go

Update the file to be 
```{golang}
package utils

import (
	"log"
	"path/filepath"

	"github.com/spf13/viper"
)

type Config struct {
	AWS_BUCKET_NAME                 string `mapstructure:"AWS_BUCKET_NAME"`
	AWS_REGION                      string `mapstructure:"AWS_REGION"`
	AWS_S3_BUCKET_ACCESS_KEY        string `mapstructure:"AWS_S3_BUCKET_ACCESS_KEY"`
	AWS_S3_BUCKET_SECRET_ACCESS_KEY string `mapstructure:"AWS_S3_BUCKET_SECRET_ACCESS_KEY"`
}

func LoadViperEnvironment(path string) (config Config, err error) {
	viper.SetConfigFile(filepath.Join(path, ".env"))
	viper.SetConfigType("env")

	// Read the config file
	if err := viper.ReadInConfig(); err != nil {
		log.Fatalf("Error reading config file, %s", err)
	}

	err = viper.Unmarshal(&config)
	if err != nil {
		log.Fatalf("Unable to decode into struct, %v", err)
	}

	return config, nil
}

```
We will use the code above to load the environment variables into our routes by updating `setupRoutes()` in main.go
```{go}
func setupRoutes() {
	// Load configurations from environment variables from .env
	// at root of project
	cwd, err := os.Getwd()
	if err != nil {
		log.Fatalf("Error getting current working directory: %v", err)
	}
	config, err := utils.LoadViperEnvironment(cwd)
	if err != nil {
		log.Fatalf("Error loading config: %s", err)
	}
	http.HandleFunc("/upload", UploadHandler(config))
	http.HandleFunc("/delete", DeleteHandler(config))
	http.ListenAndServe(":8080", nil)
}
```

We updated setupRoutes to import the app configs from the .env file and then pass those configs to the separate routes where they will be used when creating s3 bucket presigned urls. 

#### Setup S3 bucket Client 
Create another folder in the root directory named `repo` inside repo we will create a file named `repo.go`.

For this stage of the project I will need to download the necessary modules from the golang sdk

Write the following commands in the terminal to get the sdk modules.

```{bash}
go get github.com/aws/aws-sdk-go-v2
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/service/s3
```
First lets add the following code into the `repo.go` file 
```{go}
type Repo struct {
	s3Client          *s3.Client
	s3PresignedClient *s3.PresignClient
}

func NewS3Client(accessKey string, secretKey string, s3BucketRegion string) *Repo {
	options := s3.Options{
		Region:      s3BucketRegion,
		Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
	}

	client := s3.New(options, func(o *s3.Options) {
		o.Region = s3BucketRegion
		o.UseAccelerate = false
	})

	presignClient := s3.NewPresignClient(client)
	return &Repo{
		s3Client:          client,
		s3PresignedClient: presignClient,
	}
}

func (repo Repo) PutObject(bucketName string, objectKey string, lifetimeSecs int64) (*v4.PresignedHTTPRequest, error) {
	request, err := repo.s3PresignedClient.PresignPutObject(context.TODO(), &s3.PutObjectInput{
		Bucket: aws.String(bucketName),
		Key:    aws.String(objectKey),
	}, func(opts *s3.PresignOptions) {
		opts.Expires = time.Duration(lifetimeSecs * int64(time.Second))
	})
	if err != nil {
		log.Printf("Couldn't get a presigned request to put %v:%v. Here's why: %v\n",
			bucketName, objectKey, err)
	}
	return request, err
}

func (repo Repo) DeleteObject(bucketName string, objectKey string, lifetimeSecs int64) (*v4.PresignedHTTPRequest, error) {
	request, err := repo.s3PresignedClient.PresignDeleteObject(context.TODO(), &s3.DeleteObjectInput{
		Bucket: aws.String(bucketName),
		Key:    aws.String(objectKey),
	}, func(opts *s3.PresignOptions) {
		opts.Expires = time.Duration(lifetimeSecs * int64(time.Second))
	})
	if err != nil {
		log.Printf("Couldn't get a presigned request to delete %v:%v. Here's why: %v\n",
			bucketName, objectKey, err)
	}

	return request, err
}

func (repo Repo) UploadFile(file image.Image, url string) error {
	var buf bytes.Buffer
	err := jpeg.Encode(&buf, file, nil)
	if err != nil {
		return nil
	}
	body := io.Reader(&buf)
	request, err := http.NewRequest(http.MethodPut, url, body)
	if err != nil {
		return err
	}

	request.Header.Set("Content-Type", "image/jpeg")

	client := &http.Client{}
	resp, err := client.Do(request)
	if err != nil {
		log.Println("Error sending request:", err)
		return err
	}
	defer resp.Body.Close() 
	return err
}

func (repo Repo) DeleteFile(url string) error {
	request, err := http.NewRequest(http.MethodDelete, url, nil)
	if err != nil {
		return err
	}
	response, err := http.DefaultClient.Do(request)
	if err != nil {
		return err
	}

	log.Printf("Delete Request Response status code: %v", response.StatusCode)
	return err
}
```

1. Create a struct `Repo` that holds our s3client and s3PresingedClient.
2. `NewS3Client` takes in accesskey, secretkey, s3bucket region and returns the defined repo struct with all containing the s3 clients
3. The `PutObject` function that creates a presigned url for uploading files into the bucket. The function required the bucketname, objectkey(filename), lifetime(how long will the presinged url will be valid)
4. The `DeleteObject` function creates a presigned url for deleting a file from the bucket. The function required bucketname, objectkey(filename) and lifetime to create return the url. 
5. `UploadFile` function takes in the url generate from the `Putobject` function and perform a put request populated by a file.
6. `DeleteFile` function takes in the url generated from `DeleteObject` function and makes a delete request to the url. 

### Hooking Up everything
 
We will now update our routes to use the configuration from .env and upload and delete files using the repo functions

#### Upload Action
```{go}
func UploadHandler(config utils.Config) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {

		file, _, err := r.FormFile("image")
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			fmt.Fprintf(w, "Error reading file: %v", err)
			return
		}
		defer file.Close()

		imageDoc, _, err := image.Decode(file)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			fmt.Fprintf(w, "Error encoding image: %v", err)
			return
		}

		repo := repo.NewS3Client(
			config.AWS_S3_BUCKET_ACCESS_KEY,
			config.AWS_S3_BUCKET_SECRET_ACCESS_KEY,
			config.AWS_REGION,
		)

		// Create singed url used for uploading file
		presignedurl, err := repo.PutObject(config.AWS_BUCKET_NAME, 'filename', 60)
		if err != nil {
			log.Fatalf("Error generating presigned url for put object: %s", err)
		}
		// Uploading document to S3
		err = repo.UploadFile(file, presignedurl.URL)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			fmt.Fprintf(w, "Error uploading image to S3: %v", err)
			return
		}

		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "Successfully uploaded image: %s", filename)
	}
}
```
We take the file from the request and decode it before uploading it to the bucket

#### Delete Action
```{go}
func DeleteHandler(config utils.Config) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		// Read request body
		type RequestBody struct {
			Filename string `json:"filename"`
		}
		var data RequestBody
		if r.Body != nil {
			json.NewDecoder(r.Body).Decode(&data)
		}

		if data.Filename == "" {
			w.WriteHeader(http.StatusInternalServerError)
			fmt.Fprintf(w, "Filename must be provided")
			return
		}

		repo := repo.NewS3Client(
			config.AWS_S3_BUCKET_ACCESS_KEY,
			config.AWS_S3_BUCKET_SECRET_ACCESS_KEY,
			config.AWS_REGION,
		)

		presignedurl, err := repo.DeleteObject(config.AWS_BUCKET_NAME, data.Filename, 60)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			log.Fatalf("Error generating presigned url for put object: %s", err)
			return
		}

		err = repo.DeleteFile(presignedurl.URL)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			log.Fatalf("Error deleting file from s3 bucket: %v", err)
			return
		}

		w.WriteHeader(200)
		fmt.Fprintf(w, "Successfully deleted file: %s", data.Filename)
	}
}
```

The delete action works similar to the upload handler difference is we expect a json of the format
```
{
 "filename": "<filename-here>",
}
```
The file the specified name will be deleted. 

## Conclusion 
The article has goes through the process of deleting and uploading files by using presigned url. Further Reading Material on this 
* [aws-sdk-tutorial](https://aws.github.io/aws-sdk-go-v2/docs/getting-started/)
* [Github-Repo](https://github.com/Sean-Miningah/s3BucketWithGo)