Golang Tutorial: Upload Bulk Files to aws S3 Using Golang
Tags: tutorial, aws, s3, go
Published: 2022-10-27 13:47:47

AWS S3 or Amazon S3 is a storage system from Amazon to store and retrieve files from anywhere on the web. It is a highly scalable, reliable, fast, inexpensive data storage system from Amazon. It provides easy to use developer kit to store and retrieve files.

In this tutorial you will learn how to upload bulk files to S3 bucket aws golang. We will use the [Go AWS SDK](https://aws.amazon.com/sdk-for-go/) to upload files to AWS S3.

The AWS SDK for Go requires Go 1.15 or later versions. You can view your current version of Go by running the following command.

```bash
go version
```

For information about installing or upgrading your version of Go, see [https://golang.org/doc/install](https://golang.org/doc/install)

Before you can use the AWS SDK for Go V2, you must have an Amazon account. See [How do I create and activate a new AWS account?](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) for details

Download the AWS Golang SDK for using in your codebase using below command

```bash
go get github.com/aws/aws-sdk-go/aws
```

We will cover this tutorial step by step to upload bulk files to AWS S3 Bucket server using Golang with example code.

## Load require packages

We will create main.go file and import necessary packages we need to upload the files into AWS S3 Bucket.

```go
package main

import (
    "bytes"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)
```

## Custom struct for receiver

We will be using a Session struct so we can use same it as receiver, and we will pass this receiver to our main upload function,

Via using a custom struct we’re eliminating the duplication from our code. We’ll be using only single session for our all bulk files to upload into s3 bucket aws golang.

```go
type Session struct {
    S3Session *session.Session
}
```

## Set up main configuration and AWS session

We will setup configuration using AWS S3 REGION and SECRET ID and SECRET KEY and create single AWS session to upload multiple files to AWS S3. Then we will call method upload() and pass AWS session instance and file details to upload file to AWS S3 server.

```go
func main() {
    paths := []string{"", ""}
    credential := credentials.NewStaticCredentials(
        os.Getenv("SECRET_ID"),
        os.Getenv("SECRET_KEY"),
        "",
    )

    awsConfig := aws.Config{
        Region:      aws.String(os.Getenv("REGION")),
        Credentials: credential,
    }

    s, err := session.NewSession(&awsConfig)
    if err != nil {
        log.Println("failed to create S3 session:", err.Error())
        return
    }

    se := Session{s}

    err = se.upload(paths)
    if err != nil {
        log.Println(err.Error())
        return
    }

}
```

## Create Upload Method to Upload Bulk Files to AWS S3

We will create method upload() to upload files to AWS S3 server. We will open the file one by one and store into buffer and then put the file to AWS S3 using PutObject() method from S3. We will also specify options in the PutObjectInput when uploading the file. We will also enable AES256 encryption on files using ServerSideEncryption options of s3.PutObjectInput struct.

```go
func (s Session) upload(paths []string) error {
    for _, path := range paths {
        upFile, err := os.Open(path)
        if err != nil {
            log.Printf("failed %s, error: %v", path, err.Error())
            continue
        }
        defer upFile.Close()

        upFileInfo, err := upFile.Stat()
        if err != nil {
            log.Printf("failed to get stat %s, error: %v", path, err.Error())
            continue
        }
        var fileSize int64 = upFileInfo.Size()
        fileBuffer := make([]byte, fileSize)
        upFile.Read(fileBuffer)

        // uploading
        _, err = s3.New(s.S3Session).PutObject(&s3.PutObjectInput{
            Bucket:               aws.String(os.Getenv("BUCKET_NAME")),
            Key:                  aws.String(path),
            ACL:                  aws.String("public-read"), // could be private if you want it to be access by only authorized users
            Body:                 bytes.NewReader(fileBuffer),
            ContentLength:        aws.Int64(int64(fileSize)),
            ContentType:          aws.String(http.DetectContentType(fileBuffer)),
            ContentDisposition:   aws.String("attachment"),
            ServerSideEncryption: aws.String("AES256"),
            StorageClass:         aws.String("INTELLIGENT_TIERING"),
        })

        if err != nil {
            log.Printf("failed to upload %s, error: %v", path, err.Error())
            continue
        }

        url := "https://%s.s3-%s.amazonaws.com/%s"
        url = fmt.Sprintf(url, os.Getenv("BUCKET_NAME"), os.Getenv("REGION"), path)
        fmt.Printf("Uploaded File Url %s\n", url)
    }

    return nil
}
```

you can checkout more article like this on my [personal blog](https://programmingeeksclub.com)

[Full Code](https://programmingeeksclub.com/golang-tutorial-upload-bulk-files-to-s3-bucket-aws-golang/)