Go - Protocol Buffer, Protoc tool & generated structs
Tags: go, grpc, protoc
Published: 2021-08-05 16:23:52

## Protocol Buffers with Go

So, today I'm having a look at protocol buffers with generated output in `Go`. Let's start at the top with:

**Q. What is a protocol buffer?**
Well, google says that it's _a flexible, efficient, automated mechanism for serializing structured data, like XML but smaller, faster and simpler_.

**Q. That's great, thanks. Now what does it mean?**
Well, seemingly we can _define once how we'd like our data to be structured_. Then we just simply _use the specially generated source code to read and write the data a variety of streams in a variety of languages_. From this we can summarize that they allow us to define the `data contract` between systems or services. Whereas data in formats like XML or JSON are still text-based protocol buffers are binary.

**Q. What are the benefits of this?**
- Stronger interfaces are a good thing.
- Smaller size than text based alternatives are another advantage.
- typically they are speed optimised.
- type and order are in place so ambiguity is minimised.
- data access classes generated are considered to be easier to work with.

>A tight interface with our data is absolutely essential for designing microservices well.

> Protocol buffers in Go can be used on different transport layers such as HTTP/2 or AMQP (Advanced Message Queuing Protocol) where the binary data being transferred can only be understood between the client and the server.


![but first](https://i.makeagif.com/media/3-08-2016/-S2aPN.gif)

## Protocol Buffer language

I know this is a bit wordy with no code yet and it can feel a bit TL;DR but stick with me. A protocol buffer is a file with a minimalist language syntax we use to define our data structures. We compile this file and it generates a new file for a given programming language - in our case this means generating `.go` files. The language gives us types that we will use to create interfaces. This is called a `protobuf` which is one of the terms of the technology. Let's see an example of this for version 3.

```sh
syntax 'proto3';

# note that we have type, name, index position in the file for each entry
message SomeInterface {
    int id = 1;
    int code = 2;
    string name =3;
    string nickname = 4;
}
```

This file when compiled will give us Go style structures.

Key types:
- Scalar values.
- Enumerations and repeated values.
- nested fields.

### Enumerations

A numerical ordering of a given set of elements, defaults from 0 to _n_. This is fairly standard and should remind you of an `iota` in Go. Where it differs is by using `option allow_alias = true` we can have the same numeric value assigned to more than one option. essentially allowing duplicates.

```sh
syntax = 'proto3';

message Move{
    enum Moves{
        ROCK = 0
        PAPER = 1
        SCISSORS = 2
    }
}
```

### Repeated

Repeated values are much like an array in JSON. We use the `[values, values, values]` syntax.

### Nested fields

We can see below how to define nested types. Below we have a person who can have multiple contact numbers.

```sh
syntax = 'proto3';

message MyPerson {
    string name = 1;
    string nickname = 2;
    repeated ContactType contacts = 3;
}

message ContactType {
    string desc =1;
    string number =2;s
}
```

## Protoc and Protocol Buffer compilation

- Install the protoc tool for your system.
- save your protobuf files with a `.proto` extension.
- compile to target a particular language, for us that is the mighty Go!
- Import structs from generated file and add the required data

## Here we go, the generating part....

So now you want to generate the `Go` files, this can be real headache. I must have dropped a few pounds in the swear jar here but if you're trying to do this (or you're me several months from now) remember to put the `option go_package = "github.com/USERNAME/MODULE_NAME";` in the `.proto` file.

By now this will look like: 

```sh
syntax = 'proto3';

option go_package = "github.com/USERNAME/MODULE_NAME";

package protofiles;

message MyPerson {
    string name = 1;
    string nickname = 2;
    repeated ContactType contacts = 3;
}

message ContactType {
    string desc =1;
    string number =2;s
}
```

to do the generation of our files we can run the following in the CLI. 
```sh
protoc --go_out=. --go_opt=Mpaths=source_relative  FILENAME.proto
```

Afterwards, you'll have generated a new folder set in the protofiles directory and in the project name subfolder you'll find a `person.pb.go`. This autogenerated code should look similar to below.

```go
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// 	protoc-gen-go v1.27.1
// 	protoc        v3.17.3
// source: mock.proto

package protocol_buffers

import (
	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
	reflect "reflect"
	sync "sync"
)

const (
	// Verify that this generated code is sufficiently up-to-date.
	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
	// Verify that runtime/protoimpl is sufficiently up-to-date.
	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)

type MyPerson struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Name     string         `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
	Nickname string         `protobuf:"bytes,2,opt,name=nickname,proto3" json:"nickname,omitempty"`
	Contacts []*ContactType `protobuf:"bytes,3,rep,name=contacts,proto3" json:"contacts,omitempty"`
}

func (x *MyPerson) Reset() {
	*x = MyPerson{}
	if protoimpl.UnsafeEnabled {
		mi := &file_mock_proto_msgTypes[0]
		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
		ms.StoreMessageInfo(mi)
	}
}

func (x *MyPerson) String() string {
	return protoimpl.X.MessageStringOf(x)
}

func (*MyPerson) ProtoMessage() {}

func (x *MyPerson) ProtoReflect() protoreflect.Message {
	mi := &file_mock_proto_msgTypes[0]
	if protoimpl.UnsafeEnabled && x != nil {
		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
		if ms.LoadMessageInfo() == nil {
			ms.StoreMessageInfo(mi)
		}
		return ms
	}
	return mi.MessageOf(x)
}

// Deprecated: Use MyPerson.ProtoReflect.Descriptor instead.
func (*MyPerson) Descriptor() ([]byte, []int) {
	return file_mock_proto_rawDescGZIP(), []int{0}
}

func (x *MyPerson) GetName() string {
	if x != nil {
		return x.Name
	}
	return ""
}

func (x *MyPerson) GetNickname() string {
	if x != nil {
		return x.Nickname
	}
	return ""
}

func (x *MyPerson) GetContacts() []*ContactType {
	if x != nil {
		return x.Contacts
	}
	return nil
}

type ContactType struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Desc   string `protobuf:"bytes,1,opt,name=desc,proto3" json:"desc,omitempty"`
	Number string `protobuf:"bytes,2,opt,name=number,proto3" json:"number,omitempty"`
}

func (x *ContactType) Reset() {
	*x = ContactType{}
	if protoimpl.UnsafeEnabled {
		mi := &file_mock_proto_msgTypes[1]
		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
		ms.StoreMessageInfo(mi)
	}
}

func (x *ContactType) String() string {
	return protoimpl.X.MessageStringOf(x)
}

func (*ContactType) ProtoMessage() {}

func (x *ContactType) ProtoReflect() protoreflect.Message {
	mi := &file_mock_proto_msgTypes[1]
	if protoimpl.UnsafeEnabled && x != nil {
		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
		if ms.LoadMessageInfo() == nil {
			ms.StoreMessageInfo(mi)
		}
		return ms
	}
	return mi.MessageOf(x)
}

// Deprecated: Use ContactType.ProtoReflect.Descriptor instead.
func (*ContactType) Descriptor() ([]byte, []int) {
	return file_mock_proto_rawDescGZIP(), []int{1}
}

func (x *ContactType) GetDesc() string {
	if x != nil {
		return x.Desc
	}
	return ""
}

func (x *ContactType) GetNumber() string {
	if x != nil {
		return x.Number
	}
	return ""
}

var File_mock_proto protoreflect.FileDescriptor

var file_mock_proto_rawDesc = []byte{
	0x0a, 0x0a, 0x6d, 0x6f, 0x63, 0x6b, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x70, 0x72,
	0x6f, 0x74, 0x6f, 0x66, 0x69, 0x6c, 0x65, 0x73, 0x22, 0x6f, 0x0a, 0x08, 0x4d, 0x79, 0x50, 0x65,
	0x72, 0x73, 0x6f, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01,
	0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x1a, 0x0a, 0x08, 0x6e, 0x69, 0x63, 0x6b,
	0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x6e, 0x69, 0x63, 0x6b,
	0x6e, 0x61, 0x6d, 0x65, 0x12, 0x33, 0x0a, 0x08, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x73,
	0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x66, 0x69,
	0x6c, 0x65, 0x73, 0x2e, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x52,
	0x08, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x73, 0x22, 0x39, 0x0a, 0x0b, 0x43, 0x6f, 0x6e,
	0x74, 0x61, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x64, 0x65, 0x73, 0x63,
	0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x64, 0x65, 0x73, 0x63, 0x12, 0x16, 0x0a, 0x06,
	0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x6e, 0x75,
	0x6d, 0x62, 0x65, 0x72, 0x42, 0x23, 0x5a, 0x21, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63,
	0x6f, 0x6d, 0x2f, 0x65, 0x6c, 0x65, 0x73, 0x71, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f,
	0x6c, 0x2d, 0x62, 0x75, 0x66, 0x66, 0x65, 0x72, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f,
	0x33,
}

var (
	file_mock_proto_rawDescOnce sync.Once
	file_mock_proto_rawDescData = file_mock_proto_rawDesc
)

func file_mock_proto_rawDescGZIP() []byte {
	file_mock_proto_rawDescOnce.Do(func() {
		file_mock_proto_rawDescData = protoimpl.X.CompressGZIP(file_mock_proto_rawDescData)
	})
	return file_mock_proto_rawDescData
}

var file_mock_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_mock_proto_goTypes = []interface{}{
	(*MyPerson)(nil),    // 0: protofiles.MyPerson
	(*ContactType)(nil), // 1: protofiles.ContactType
}
var file_mock_proto_depIdxs = []int32{
	1, // 0: protofiles.MyPerson.contacts:type_name -> protofiles.ContactType
	1, // [1:1] is the sub-list for method output_type
	1, // [1:1] is the sub-list for method input_type
	1, // [1:1] is the sub-list for extension type_name
	1, // [1:1] is the sub-list for extension extendee
	0, // [0:1] is the sub-list for field type_name
}

func init() { file_mock_proto_init() }
func file_mock_proto_init() {
	if File_mock_proto != nil {
		return
	}
	if !protoimpl.UnsafeEnabled {
		file_mock_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
			switch v := v.(*MyPerson); i {
			case 0:
				return &v.state
			case 1:
				return &v.sizeCache
			case 2:
				return &v.unknownFields
			default:
				return nil
			}
		}
		file_mock_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
			switch v := v.(*ContactType); i {
			case 0:
				return &v.state
			case 1:
				return &v.sizeCache
			case 2:
				return &v.unknownFields
			default:
				return nil
			}
		}
	}
	type x struct{}
	out := protoimpl.TypeBuilder{
		File: protoimpl.DescBuilder{
			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
			RawDescriptor: file_mock_proto_rawDesc,
			NumEnums:      0,
			NumMessages:   2,
			NumExtensions: 0,
			NumServices:   0,
		},
		GoTypes:           file_mock_proto_goTypes,
		DependencyIndexes: file_mock_proto_depIdxs,
		MessageInfos:      file_mock_proto_msgTypes,
	}.Build()
	File_mock_proto = out.File
	file_mock_proto_rawDesc = nil
	file_mock_proto_goTypes = nil
	file_mock_proto_depIdxs = nil
}
```

I'll be using these in a further examples where we get to write some Go and use this, but we've seen the phaff and pain required to setup and generate your files. I hope if you're dabbling in this area at all that is somewhat useful to you. 

Best wishes.