Define a protobuf message and generate Go code
Tags: grpc, beginners, go, java
Published: 2020-02-20 10:47:35

Hello everyone! Let’s start the hands-on section of the gRPC course. The target of the whole section is to build a "pc book" web service that will allow us to manage and search for laptop configurations.

Here's the link to the [full gRPC course playlist on Youtube](https://www.youtube.com/playlist?list=PLy_6D98if3UJd5hxWNfAqKMr15HZqFnqf)
Github repository: [pcbook-go](https://github.com/techschool/pcbook-go) and [pcbook-java](https://github.com/techschool/pcbook-java)
Gitlab repository: [pcbook-go](https://gitlab.com/techschool/pcbook) and [pcbook-java](https://gitlab.com/techschool/pcbook-java)

# Protocol buffer basics

{% youtube YzypniHHU3w %}

In this lecture, we will learn how to write a simple protocol-buffer message with some basic data types, install Visual Studio Code plugins to work with protobuf, and finally we will install protocol-buffer compiler and write a Makefile to run code generation for Go.

But before start, make sure that you already have Go and Visual Studio Code up and running properly on your computer. If not, you can watch my tutorial video on how to install Go and setup Visual Studio code:

{% youtube jRLV-1GVET4 %}

The tutorial will guide you, step by step, to install Go, add the `bin` folder to your `PATH`, install Visual Studio Code, customise its theme and setup Go extensions to work with it.

Once everything is ready, you can come back here and continue this lecture.

## Install vscode plugins

Alright, let's start by creating a new project. First, I will create a simple hello-world program in `main.go` file and run it, just to make sure that Go is working properly.

```go
package main

import "fmt"

func main() {
    fmt.Println("Hello world")
}
```

Then create a new folder named `protoc`, and add a `processor_message.proto` file under it. 

```bash
pcbook
├── proto
│   └── processor_message.proto
└── main.go
```

Vscode will ask us to install the extensions for the proto file. So let's go to the marketplace and search for `ext:proto`

![Search for ext:proto](https://dev-to-uploads.s3.amazonaws.com/i/yvmevdswbn62arr7vyrw.png)

There are 2 extensions shown at the top that we should install: `clang-format` and `vscode-proto3`. Let’s click install for both of them.

## How to define a protobuf message

Now come back to our proto file. This file will contain the message definition of the CPU of a laptop.

We start with `syntax = "proto3"`. 

At the moment, there are 2 versions of protocol buffer on [Google's official documentation](https://developers.google.com/protocol-buffers/docs/overview): `proto2` and `proto3`. For simplicity, we will only use `proto3` (the newer version) in this course.

The syntax is pretty simple, just use the `message` keyword followed by the name of the message. Then inside the message block, we define all of its fields as shown in this picture:

![How to write protubuf message](https://dev-to-uploads.s3.amazonaws.com/i/y02baw92xgehu2zz75iz.png)

Note that the name of the message should be UpperCamelCase, and the name of the field should be lower_snake_case.

There are many built-in [scalar-value data types](https://developers.google.com/protocol-buffers/docs/proto3#scalar) that we can use, for instance: `string`, `bool`, `byte`, `float`, `double`, and many other integer types. We can also use our own data types, such as enums or other messages.

Each message field should be assigned a unique tag. And the tag is more important than the field name because protobuf will use it to serialise the message.

A tag is simply an arbitrary integer with the smallest value of 1, and the biggest value of 2<sup>29</sup> - 1, except for numbers from 19000 to 19999, as they're reserved for internal protocol buffers implementation.

Note that tags from 1 to 15 take only 1 byte to encode, while those from 16 to 2047 take 2 bytes. So you should use them wisely, like: saving tags from 1 to 15 for very frequently occurring fields. 

And remember that the tags don't need to be in-order (or sequential), but they must be unique for the same-level fields of the message.

## Define the CPU message

Now let's get back to our proto file and define the CPU message. 

```protobuf
syntax = "proto3";

message CPU {
  string brand = 1;
  string name = 2;
  uint32 number_cores = 3;
  uint32 number_threads = 4;
  double min_ghz = 5;
  double max_ghz = 6;
}
```

The CPU will have a brand of type `string`, such as "Intel", and a name also of type `string`, for example "Core i7-9850".

We need to keep track of how many cores or threads the CPU has. They cannot be negative, so let's use `uint32` here.

Next, it has the minimum and maximum frequency, for example 2.4 Ghz or something like that. So we can use `double` type here.

## Generate Go codes

Now we've finished our first protobuf message. How can we generate Go codes from it?

First, we need to install protocol buffer compiler (or `protoc`). On macOS, we can easily do that with the help of [Homebrew](https://brew.sh/).

You can install Homebrew with this simple command:

```bash
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```

Once Homebrew is installed, can run this command to install `protoc`:

```bash
brew install protobuf
```

We can check if it's working or not by running the `protoc` command.

Next we will go to [grpc.io](https://grpc.io/docs/quickstart/go/) to copy and run 2 commands to install 2 libraries: the `golang grpc` library and the `protoc-gen-go` library.

```bash
go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go
```

Now we're all set! I will create a new folder named `pb` to store the generated Go codes.

```bash
pcbook
├── proto
│   └── processor_message.proto
├── pb
└── main.go
```

Then run this command to generate the codes:

```bash
protoc --proto_path=proto proto/*.proto --go_out=plugins=grpc:pb
```

Our proto file is located inside the `proto` folder, so we tell `protoc` to look for it in that folder.

With the `go_out` parameter, we tell `protoc` to use the `grpc` plugins to generate Go codes, and store them inside the `pb` folder that we've created before.

Now if we open that folder in vscode, we will see a new file `processor_message.pb.go`.

```bash
pcbook
├── proto
│   └── processor_message.proto
├── pb
│   └── processor_message.pb.go
└── main.go
```

Look inside, there's a CPU struct and all fields with the correct data types as we defined in our protocol buffer file.

```go
const _ = proto.ProtoPackageIsVersion3

type CPU struct {
	Brand                string   `protobuf:"bytes,1,opt,name=brand,proto3" json:"brand,omitempty"`
	Name                 string   `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
	NumberCores          uint32   `protobuf:"varint,3,opt,name=number_cores,json=numberCores,proto3" json:"number_cores,omitempty"`
	NumberThreads        uint32   `protobuf:"varint,4,opt,name=number_threads,json=numberThreads,proto3" json:"number_threads,omitempty"`
	MinGhz               float64  `protobuf:"fixed64,5,opt,name=min_ghz,json=minGhz,proto3" json:"min_ghz,omitempty"`
	MaxGhz               float64  `protobuf:"fixed64,6,opt,name=max_ghz,json=maxGhz,proto3" json:"max_ghz,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

func (m *CPU) Reset()         { *m = CPU{} }
func (m *CPU) String() string { return proto.CompactTextString(m) }
func (*CPU) ProtoMessage()    {}
func (*CPU) Descriptor() ([]byte, []int) {
	return fileDescriptor_466578cecc6db379, []int{0}
}

func (m *CPU) XXX_Unmarshal(b []byte) error {
	return xxx_messageInfo_CPU.Unmarshal(m, b)
}
func (m *CPU) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
	return xxx_messageInfo_CPU.Marshal(b, m, deterministic)
}
func (m *CPU) XXX_Merge(src proto.Message) {
	xxx_messageInfo_CPU.Merge(m, src)
}
func (m *CPU) XXX_Size() int {
	return xxx_messageInfo_CPU.Size(m)
}
func (m *CPU) XXX_DiscardUnknown() {
	xxx_messageInfo_CPU.DiscardUnknown(m)
}

var xxx_messageInfo_CPU proto.InternalMessageInfo

func (m *CPU) GetBrand() string {
	if m != nil {
		return m.Brand
	}
	return ""
}

func (m *CPU) GetName() string {
	if m != nil {
		return m.Name
	}
	return ""
}

func (m *CPU) GetNumberCores() uint32 {
	if m != nil {
		return m.NumberCores
	}
	return 0
}

func (m *CPU) GetNumberThreads() uint32 {
	if m != nil {
		return m.NumberThreads
	}
	return 0
}

func (m *CPU) GetMinGhz() float64 {
	if m != nil {
		return m.MinGhz
	}
	return 0
}

func (m *CPU) GetMaxGhz() float64 {
	if m != nil {
		return m.MaxGhz
	}
	return 0
}
```

There are some special fields used internally by gRPC to serialise the message, but we don't need to care about them. Some useful getter functions are also generated. So it looks great!

## Write a Makefile

The command that we used to generate codes is pretty long, so it’s not very convenient to type when we update the proto file and want to regenerate the codes. So let's create a Makefile with a short and simple command to do that.

```bash
pcbook
├── proto
│   └── processor_message.proto
├── pb
│   └── processor_message.pb.go
├── main.go
└── Makefile
```

In this Makefile, we add a `gen` task to run code generation command, a `clean` task to remove all generated go files whenever we want, and a `run` task to run the `main.go` file as well.

```Makefile
gen:
	protoc --proto_path=proto proto/*.proto --go_out=plugins=grpc:pb

clean:
	rm pb/*.go 

run:
	go run main.go
```

We can try them in the terminal. 

![Run make commands](https://dev-to-uploads.s3.amazonaws.com/i/93ch2dw8p10e3qpi6b4e.png)

When we run `make clean`, the generated files will be deleted. 

When we run `make gen`, the files will be regenerated in `pb` folder. 

And finally, when we run `make run`, "Hello world" is printed.

## What's next

OK, so now you know how to define a simple protocol buffer message and generate Go code from it. In the next lecture, we will dig deeper and learn more advanced features of protobuf.

Thanks for reading! Happy coding, and see you later!

---

If you like the article, please [subscribe to our Youtube channel](http://bit.ly/techschoolguru) and [follow us on Twitter](https://twitter.com/techschoolguru) for more tutorials in the future.

---
If you want to join me on my current amazing team at Voodoo, check out [our job openings here](https://bit.ly/voodoojobs). Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.