Protobuf Quick Refresher
Published: 2023-07-23 22:07:11

Protocol Buffers, for those needing a quick refresher, is a language-agnostic binary format developed by Google. The format is leveraged to structure and store data, and is quite popular due to its compact size and fast serialization/deserialization properties. 

In our day-to-day work, we use protobufs to define message structures and services, and the protobuf compiler `protoc` generates data access classes for us in our language of choice, which in our case is Python.

## Protobuf Schema Definition (.proto files)

Let's start by writing a .proto file, which will be the blueprint for our data.

```protobuf
syntax = "proto3";

package tutorial;

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;
}
```
In the example above, `Person` is a message type, with various fields like `name`, `id`, and `email`. Notice the number assignment to the fields, these numbers are used to identify your fields in the message binary format, they won't be reassigned.

We've also defined an enum `PhoneType` and a nested message `PhoneNumber` within `Person`, which is then used in the repeated field `phones` (equivalent to a list in Python).

## Generating Python Code

Once we have our .proto file, we can generate Python code using the `protoc` compiler. 

```bash
protoc --python_out=. ./person.proto
```
This generates a `person_pb2.py` file which includes Python classes for our defined messages, enums, and services.

## Serializing and Deserializing Data

Next, let's create, serialize, and then deserialize a `Person` message in Python.

```python
import person_pb2

# Create a new Person
person = person_pb2.Person()
person.name = "John Doe"
person.id = 1234
person.email = "jdoe@example.com"

phone = person.phones.add()
phone.number = "555-4321"
phone.type = person_pb2.Person.HOME

# Serialize to a byte string
person_bytes = person.SerializeToString()

# Deserialize to a new person
new_person = person_pb2.Person()
new_person.ParseFromString(person_bytes)
```

Here, we've created an instance of `Person`, filled in its fields, then serialized it to a byte string using `SerializeToString()`. Then, we've taken that byte string and deserialized it back into a `Person` object using `ParseFromString()`.

## Reading and Writing to Files

You can also write these byte strings directly to files or read them from files:

```python
# Write to a file
with open("person.bin", "wb") as fd:
    fd.write(person_bytes)

# Read from a file
with open("person.bin", "rb") as fd:
    new_person = person_pb2.Person()
    new_person.ParseFromString(fd.read())
```



## Optional Fields and Default Values

In protobuf 3, all fields are optional, and fields not set take their default values. For scalar types (like int32, bool, string), default values are the "zero value" for the type - 0 for integers, false for bools, and empty string for strings. Enum fields have a default value of the first value listed in the enum's type definition.

Here's an example:

```protobuf
syntax = "proto3";

message Weather {
  string location = 1;
  int32 temperature = 2;
  bool is_raining = 3;
}
```

In Python:

```python
import weather_pb2

weather = weather_pb2.Weather()
weather.location = "San Francisco"

print(weather.temperature)  # prints: 0
print(weather.is_raining)  # prints: False
```

In the example, we didn't set `temperature` or `is_raining`, but they still have default values.

## Upgradeability

Protobuf shines when it comes to upgradeability, i.e., the ability to evolve your data structures over time without breaking existing systems. Essentially, you can add new fields to your messages or stop using old ones, and your protobuf data will remain backwards and forwards compatible.

Here's a simple example of a `Person` message:

### Version 1
Let's say we start with this message definition:

```protobuf
syntax = "proto3";

message Person {
  string name = 1;
}
```
And in our Python code, we create a `Person` and serialize it:

```python
import person_pb2

person = person_pb2.Person()
person.name = "John Doe"
person_bytes = person.SerializeToString()
```

### Version 2
Then, at a later time, we update our message to include a new field `age`:

```protobuf
syntax = "proto3";

message Person {
  string name = 1;
  int32 age = 2;  // New field
}
```

Even though our Python code doesn't know about the `age` field yet, it can still deserialize the `Person` bytes:

```python
old_person = person_pb2.Person()
old_person.ParseFromString(person_bytes)
print(old_person.name)  # prints: John Doe
```

This example illustrates backward compatibility, where newer message versions can be read by older software. The older version simply ignores fields it doesn't recognize.

### Version 3
Let's say we then decide to remove the `age` field:

```protobuf
syntax = "proto3";

message Person {
  string name = 1;  // 'age' field removed
}
```
And we serialize a new `Person` object:

```python
new_person = person_pb2.Person()
new_person.name = "Jane Doe"
new_person_bytes = new_person.SerializeToString()
```

The Python code from version 1 (which expects the `name` field and doesn't know about `age`) can still read this:

```python
v1_person = person_pb2.Person()
v1_person.ParseFromString(new_person_bytes)
print(v1_person.name)  # prints: Jane Doe
```

This illustrates forward compatibility, where older message versions can be read by newer software.

Key takeaway: Whenever you change a message type by adding or removing fields, be mindful not to reuse the old field numbers. This will ensure that your updated code maintains compatibility with older versions.

## Remote Procedure Calls (RPCs) and Services in Protobuf

In addition to defining message types, Protobuf also allows us to define services, which specify a set of RPCs (Remote Procedure Calls) that can be implemented on a server and called on a client. This makes Protobuf a great choice for designing the interface of microservices.

Here's how we could define a simple service:

```protobuf
syntax = "proto3";

message HelloRequest {
  string greeting = 1;
}

message HelloResponse {
  string reply = 1;
}

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloResponse);
}
```

In this `.proto` file, we define a `Greeter` service with a single RPC `SayHello`. This RPC takes a `HelloRequest` message and returns a `HelloResponse` message.

To generate service interfaces and bindings for Python, we need to use the `grpc_tools` package, which is an extension of `protoc` with GRPC-specific features.

```bash
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. ./greeter.proto
```

This will generate a `greeter_pb2.py` file (as before), but also a `greeter_pb2_grpc.py` file, which includes GRPC-specific code.

Now we can implement this service on the server side:

```python
import grpc
from concurrent import futures
import greeter_pb2
import greeter_pb2_grpc

class Greeter(greeter_pb2_grpc.GreeterServicer):
    def SayHello(self, request, context):
        return greeter_pb2.HelloResponse(reply='Hello, ' + request.greeting)

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
greeter_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
server.add_insecure_port('[::]:50051')
server.start()
# keep running...
```

And call it from the client side:

```python
import grpc
import greeter_pb2
import greeter_pb2_grpc

channel = grpc.insecure_channel('localhost:50051')
stub = greeter_pb2_grpc.GreeterStub(channel)
response = stub.SayHello(greeter_pb2.HelloRequest(greeting='world'))
print(response.reply)  # prints: Hello, world
```

This way, we can define both the structure of our data and the interface of our services using Protobuf, keeping our microservices clean, consistent, and easy to work with.

## Conclusion

You can refer to the [Protocol Buffers Language Guide](https://developers.google.com/protocol-buffers/docs/proto3) for more detailed information.