The easiest way to embed static files into a binary file in your Golang app (no external dependencies)
Tags: go, tutorial, beginners, productivity
Published: 2020-01-31 21:08:01

## Introduction

Hello, everyone! 👋 I have a very interesting find for you. But first, I'm going to tell you a little background story, so...

Earlier, I tried [go.rice](https://github.com/GeertJohan/go.rice) and [go-bindata](https://github.com/jteeuwen/go-bindata), then I used [packr](https://github.com/gobuffalo/packr) and switched to [pkger](https://github.com/markbates/pkger) from the same author (_Hi, Mark Bates, if you're suddenly reading this_). Yep, I was satisfied with this packages and quite happy!

But when I saw this method somewhere on the Internet, I **immediately** herded it into all of my Golang apps (who needed it, of course).

### ...and what's problem with ready-made packages?

First of all, it's external packages with **their** dependencies. It's good when you need to quickly make a working prototype OR don't really care about the size of your code base.

Documentation may not be complete or cover _actually_ what you need at this moment. I have been frustrated to have to **wait** a long time (_sometimes, up to several weeks_) to get an answer OR a bug fix.

> I have nothing against the authors of these great Go packages! I personally support many of _Open Source_ repositories on [GitHub](https://github.com/koddr) and understand they _very_ well! 😉

### Disclaimer

This article was written **before** Go 1.16 released, which added awesome features for adding files to the final binary.

### Why I need this?

Good question, by the way, because many programmers do not always understand **advantages** of this approach to packaging a product (the final program, which you sell to your customer).

> ☝️ Especially, who came from scripting languages like _JavaScript_, _Python_, _PHP_ or else.

I highlight the following:

- It's **safe**. Files, which added to a binary, are impossible (or extremely _difficult_) to extract back.

- It's **convenient**. There's no need to always remember to include a folder with static files, when you give production-ready product to your customer.

![embed static files](https://dev-to-uploads.s3.amazonaws.com/i/3l0ne6zl8vza2lk0vzmq.jpg)

## OK! So what am I suggesting? 🤔

My suggest will be very simple and elegant: _why not write our own packer for static files using only built-in Go tools?_ Yes, **gophers** have already figured out _what I'm talking about_... Go generate function!

> ⭐️ Please, read first [this](https://blog.golang.org/generate) awesome blog post by **Rob Pike** (22 Dec 2014) for more understand _how it works_! Unfortunately, I can't explain this principle better and shorter than Rob did...

### No time for further reading? 🚀

I created repository with full code example on my GitHub especially for you:

{% github koddr/example-embed-static-files-go no-readme %}

Just `git clone` and read instructions from `README`.

## Project structure

Let's `go mod init ...` and create some files 👌

```console
$ tree .
.
├── Makefile
├── go.mod
├── cmd
│   └── app
│       └── main.go          <-- main file of your Go app
├── internal
│   └── box                  <-- template and functions for create blob
│       ├── box.go
│       └── generator.go
└── static                   <-- folder with static files
    └── index.html
```

Add most used commands to `Makefile`:

```makefile
.PHONY: generate

# ⚠️
generate:
	@go generate ./...
	@echo "[OK] Files added to embed box!"

security:
	@gosec ./...
	@echo "[OK] Go security check was completed!"

build: generate security
	@go build -o ./build/server ./cmd/app/*.go
	@echo "[OK] App binary was created!"

run:
	@./build/server
```

> ⚠️ I strongly recommend to use [Go security checker](https://github.com/securego/gosec) for your code! Maybe I'll write a separate article about this great package later.

### Create methods for use on code (`box.go`)

```go
//go:generate go run generator.go

package box

type embedBox struct {
	storage map[string][]byte
}

// Create new box for embed files
func newEmbedBox() *embedBox {
	return &embedBox{storage: make(map[string][]byte)}
}

// Add a file to box
func (e *embedBox) Add(file string, content []byte) {
	e.storage[file] = content
}

// Get file's content
func (e *embedBox) Get(file string) []byte {
	if f, ok := e.storage[file]; ok {
		return f
	}
	return nil
}

// Embed box expose
var box = newEmbedBox()

// Add a file content to box
func Add(file string, content []byte) {
	box.Add(file, content)
}

// Get a file from box
func Get(file string) []byte {
	return box.Get(file)
}
```

All magic included into first line: `//go:generate ...`. This call Go compiler to use methods from this file for code generate by another.

### Create template and functions for blob file (`generator.go`)

```go
//+build ignore

package main

import (
	"bytes"
	"fmt"
	"go/format"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
	"strings"
	"text/template"
)

const (
	blobFileName string = "blob.go"
	embedFolder  string = "../../static"
)

// Define vars for build template
var conv = map[string]interface{}{"conv": fmtByteSlice}
var tmpl = template.Must(template.New("").Funcs(conv).Parse(`package box

// Code generated by go generate; DO NOT EDIT.

func init() {
	{{- range $name, $file := . }}
    	box.Add("{{ $name }}", []byte{ {{ conv $file }} })
	{{- end }}
}`),
)

func fmtByteSlice(s []byte) string {
	builder := strings.Builder{}

	for _, v := range s {
		builder.WriteString(fmt.Sprintf("%d,", int(v)))
	}

	return builder.String()
}

func main() {
	// Checking directory with files
	if _, err := os.Stat(embedFolder); os.IsNotExist(err) {
		log.Fatal("Configs directory does not exists!")
	}

	// Create map for filenames
	configs := make(map[string][]byte)

	// Walking through embed directory
	err := filepath.Walk(embedFolder, func(path string, info os.FileInfo, err error) error {
		relativePath := filepath.ToSlash(strings.TrimPrefix(path, embedFolder))

		if info.IsDir() {
			// Skip directories
			log.Println(path, "is a directory, skipping...")
			return nil
		} else {
			// If element is a simple file, embed
			log.Println(path, "is a file, packing in...")

			b, err := ioutil.ReadFile(path)
			if err != nil {
				// If file not reading
				log.Printf("Error reading %s: %s", path, err)
				return err
			}

			// Add file name to map
			configs[relativePath] = b
		}

		return nil
	})
	if err != nil {
		log.Fatal("Error walking through embed directory:", err)
	}

	// Create blob file
	f, err := os.Create(blobFileName)
	if err != nil {
		log.Fatal("Error creating blob file:", err)
	}
	defer f.Close()

	// Create buffer
	builder := &bytes.Buffer{}

	// Execute template
	if err = tmpl.Execute(builder, configs); err != nil {
		log.Fatal("Error executing template", err)
	}

	// Formatting generated code
	data, err := format.Source(builder.Bytes())
	if err != nil {
		log.Fatal("Error formatting generated code", err)
	}

	// Writing blob file
	if err = ioutil.WriteFile(blobFileName, data, os.ModePerm); err != nil {
		log.Fatal("Error writing blob file", err)
	}
}
```

At this point, we set `//+build ignore` on first line, because this code needed only for code generation process and don't be in main build.

![generate it](https://dev-to-uploads.s3.amazonaws.com/i/uolc0eqrwaxf7ls1ecj6.jpg)

## We're almost done! 👍

Let's go through the block _file generation process_ itself in more detail.

So, running command:

```console
$ make generate
```

...it created for us `blob.go` file (on current project folder) with this code:

```go
package box

// Code generated by go generate; DO NOT EDIT.

func init() {
	box.Add("/index.html", []byte{35, 32, ...}
	// ...
}
```

Basically, we converted the files in the `./static` folder to a slice of bytes and now we can easily add them (in this clean form) to our binary executable! 🎉

## Time to use it all in your code

```go
package main

import (
	"log"
	"net/http"
	"text/template"

	"github.com/koddr/example-embed-static-files-go/internal/box"
)

type PageData struct {
	Title       string
	Heading     string
	Description string
}

func main() {
	// Define embed file (for a short)
	index := string(box.Get("/index.html"))

	// Template
	tmpl := template.Must(template.New("").Parse(index))

	// Handle function
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		data := PageData{
			Title:       "The easiest way to embed static files into a binary file",
			Heading:     "This is easiest way",
			Description: "My life credo is 'If you can not use N, do not use N'.",
		}

		// Execute template with data
		if err := tmpl.Execute(w, data); err != nil {
			log.Fatal(err)
		}
	})

	// Start server
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(err)
	}
}
```

![result](https://dev-to-uploads.s3.amazonaws.com/i/xrrjdorf4vgg081hbnfb.png)

> ☝️ Always use `/` for looking up! For example: `/images/logo.jpg` is actually `./static/images/logo.jpg`.

## Exercises 🤓

- Improve `box.Get()` function for getting all files from embed folder at once call (_for example, by using wildcard_). Like `box.Get("/images/*")` or similar.
- Exclude path to directory of static files to _flags_ and when call it on `go:generate`.

## Final words

I understand, that it would be much better to take a ready-made library and use its _API_... but then you would never get to know this wonderful tool and write your first code generator!

Invest in yourself and your horizons! 🎁

## Photo by

[Title] Vitor Santos https://unsplash.com/photos/GOQ32dlahDk
[1] Brunno Tozzo https://unsplash.com/photos/t32lrFimPlU
[2] Christin Hume https://unsplash.com/photos/Hcfwew744z4

<a name=""></a>
## P.S.

If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻

And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!

My projects that need your help (and stars) 👇

- 🔥 [gowebly][gowebly_url]: A next-generation CLI tool for easily build amazing web applications with Go on the backend, using htmx & hyperscript and the most popular atomic/utility-first CSS frameworks on the frontend.
- ✨ [create-go-app][cgapp_url]: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.
- 🏃 [yatr][yatr_url]: Yet Another Task Runner allows you to organize and automate your routine operations that you normally do in Makefile (or else) for each project.
- 📚 [gosl][gosl_url]: The Go Snippet Library provides snippets collection for working with routine operations in your Go programs with a super user-friendly API and the most efficient performance.
- 🏄‍♂️ [csv2api][csv2api_url]: The parser reads the CSV file with the raw data, filters the records, identifies fields to be changed, and sends a request to update the data to the specified endpoint of your REST API.
- 🚴 [json2csv][json2csv_url]: The parser can read given folder with JSON files, filtering and qualifying input data with intent & stop words dictionaries and save results to CSV files by given chunk size.

<!-- Footer links -->

[gowebly_url]: https://github.com/gowebly
[cgapp_url]: https://github.com/create-go-app
[yatr_url]: https://github.com/koddr/yatr
[gosl_url]: https://github.com/koddr/gosl
[csv2api_url]: https://github.com/koddr/csv2api
[json2csv_url]: https://github.com/koddr/json2csv