go-doudou series 02: How to develop gRPC service with go-doudou
Tags: go, grpc, microservices
Published: 2022-11-21 00:44:07

Since v2.0.0 version, `go-doudou` has been supported to develop gRPC service. The approach or workflow is the same as previous versions that is define methods of a golang interface in svc.go file first, then let go-doudou generate as much as code for you. `go-doudou` has a smooth learning curve, which make it very friendly to novice, especially to developers with other programming language background like Java, Nodejs or Python. `go-doudou` is easy, but very productive.

Here I will use a small demo to show you how the developping workflow looks like including some best practices. We will implement a [Largest Remainder Method](https://en.wikipedia.org/wiki/Largest_remainder_method) gRPC service with go-doudou, and then we will see how to test it. Full code can be cloned from [https://github.com/unionj-cloud/go-doudou-tutorials/tree/master/go-stats](https://github.com/unionj-cloud/go-doudou-tutorials/tree/master/go-stats).

## Preparation

### Intall Go
You should install go v1.16 or above.

### Install gRPC Compiler and Plugins

#### Install Compiler protoc
To install the Protobuf compiler protoc, please refer to [official documentation](https://grpc.io/docs/protoc-installation/), here are the installation commands for common operating systems:

- Ubuntu system:
```shell
$ apt install -y protobuf-compiler
$ protoc --version # Make sure to install v3 and above
```

- Mac system, you need to install [Homebrew](https://brew.sh/):
```shell
$ brew install protobuf
$ protoc --version # Make sure to install v3 and above
```

If Homebrew fails to be installed on Windows systems or Mac systems, you need to download the installation package from github, unzip it, and configure the environment variables yourself.

The latest protoc download address for Windows system: [https://github.com/protocolbuffers/protobuf/releases/download/v21.7/protoc-21.7-win64.zip](https://github.com/protocolbuffers/protobuf/releases/download/v21.7/protoc-21.7-win64.zip)

Mac system Intel latest protoc download address: [https://github.com/protocolbuffers/protobuf/releases/download/v21.7/protoc-21.7-osx-x86_64.zip](https://github.com/protocolbuffers/protobuf/releases/download/v21.7/protoc-21.7-osx-x86_64.zip)

Please find other packages in [github releases](https://github.com/protocolbuffers/protobuf/releases).

#### Install Plugins

1. Install the plugin:
```shell
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2
```

2. Configure environment variables:
```shell
$ export PATH="$PATH:$(go env GOPATH)/bin"
```

Please go to [https://grpc.io/docs/languages/go/quickstart/](https://grpc.io/docs/languages/go/quickstart/) to find the latest version number.

### Install go-doudou

- To install `go-doudou` CLI

If go version < 1.17,
```shell
go get -v github.com/unionj-cloud/go-doudou/v2@v2.0.3
```

If go version >= 1.17, recommend to use below command to install go-doudou cli globally
```shell
go install -v github.com/unionj-cloud/go-doudou/v2@v2.0.3
```

- To download go-doudou as dependency for your module
```shell
go get -v -d github.com/unionj-cloud/go-doudou/v2@v2.0.3
```
<br />If you meet 410 Gone error, try to run below command, then run install command again:
```shell
export GOSUMDB=off
```

After installation, if you meet `go-doudou: command not found` error, please configure `$HOME/go/bin` to `~/.bash_profile` file, for example:
```shell
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:/usr/local/go/bin
PATH=$PATH:$HOME/go/bin

export PATH
```

Verify go-doudou has been installed, run below command:
```shell
➜  go-doudou -v
go-doudou version v2.0.3
```

## Initialize Project
Now we are ready to start coding. First we should initialize our project.
```shell
go-doudou svc init go-stats -m go-doudou-tutorials/go-stats
```

`go-stats` is project root folder path, go-doudou will create directories recursively if any directory not exists. `go-doudou-tutorials/go-stats` is module name, specified by `-m` flag.

`go-doudou` generated these folder and files as below.
```shell
➜  go-stats git:(master) ✗ tree -L 2
.
├── Dockerfile
├── go.mod
├── svc.go
└── vo
    └── vo.go

1 directory, 4 files
```

In `svc.go` file there has been declared an interface for you to define methods to map protobuf rpcs. `vo` package is for you to define structs to map protobuf messages.

## Design Service
Then we have a look the `svc.go` file.
{% embed https://gist.github.com/wubin1989/64b134a8df849123a2ac94f7fd56af6b %}

There are two `//go:generate` directives, line 12 is for RESTful service, line 13 is for gRPC service, for convenience. If you use goland (my favourite IDE), you can just single click the green arrow to run go-doudou command.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1d04qsyo8btyt419baf5.png)



PageUsers is an example method, now we get rid of it and write our own.
{% embed https://gist.github.com/wubin1989/90cca5497a0c71ff3a1e66850ef135b0 %}

We also need to define PercentageReqVo and PercentageRespVo in vo package. Note, struct type input parameters and result parameters here in each methods must be declared in vo package as go-doudou CLI only scan the package to fetch informations for generating protobuf messages.
{% embed https://gist.github.com/wubin1989/47d949ba3e70d7fc04735ee1767df20f %}

`go-doudou name` in line 7 is a small CLI tool to create or update json tag of each fields in each structs, default is lower camel case.

## Generate Code
Now we can run `go-doudou svc grpc` command to generate proto file, server and client stub code.
```shell
➜  go-stats git:(master) ✗ tree -L 3 -a
.
├── .dockerignore
├── .env
├── .gitignore
├── Dockerfile
├── cmd
│   └── main.go
├── config
│   └── config.go
├── db
│   └── db.go
├── go.mod
├── svc.go
├── svcimpl.go
├── transport
│   └── grpc
│       ├── annotation.go
│       ├── gostats.pb.go
│       ├── gostats.proto
│       └── gostats_grpc.pb.go
└── vo
    └── vo.go

13 directories, 16 files
```

We can see that there are some more folders and files generated. 

- `.dockerignore` : ignore `**/*.local` files
- `.env` : dotenv configuration file
- `cmd` : there is `main.go` file as entry point
- `config` : mapping environment variables to config struct
- `db` : for database connection, useless here
- `svcimpl.go` : implement your own business logic
- `transport` : proto file, server and client stub code

Before we dive into the code, we should run `go mod tidy` to download dependencies. Then we start the program to make sure everything is working correctly until now.
{% embed https://gist.github.com/wubin1989/0a0fa4ce9a6709deab3d3b663927097e %}

go-doudou is relying on `go.uber.org/automaxprocs` for effectly using resources, so you can see line 2.

Let's open `transport/grpc/gostats.proto` file and explain the code.
{% embed https://gist.github.com/wubin1989/8174de7994a3306609abc3fa8fe79d25 %}

As we can see, all of the field names in each messages are lower camel case, which is not conventional snake case, because all our json tags in vo package are lower camel case. Of course you can use snake case both sides, but you must make them consistent with each other, otherwise you maybe have to convert protoc generated struct to vo struct manually (if you will use vo structs).

`go-doudou` will wrap all result parameters except error into one message like LargestRemainderRpcResponse when generating proto file and copy all comments into it. 

`go-doudou` only supports protobuf v3 grammer.

## Implement Stub
We will implement our business logic in `svcimpl.go` file now. Let's see what's in it now.
{% embed https://gist.github.com/wubin1989/71487835df46fb512f005d790e5105d6 %}

Code line 13 assigns a *GoStatsImpl type nil to interface type pb.GoStatsServiceServer to make sure pointer type GoStatsImpl is always implementing pb.GoStatsServiceServer interface.

We can add any fields to GoStatsImpl if necessary like external service client instance, database connection instance, any queue or pool instance, etc. There is a package level factory function NewGoStats for you to inject any dependencies and create a pointer type GoStatsImpl instance for later use.

Now let's implement LargestRemainderRpc method. Here is a piece of best practice that you don't have to implement pb.GoStatsServiceServer directly, you can choose to implement GoStats at first, then call each implemented GoStats methods in corresponding pb.GoStatsServiceServer method implementations as `go-doudou` doesn't support grpc-gateway and grpc-web, implementing GoStats first can let you reuse code to add RESTful endpoints when necessary in the future.

So we prefer to run `go-doudou svc http -c` to generate RESTful related code at first even if you don't need RESTful at the moment.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nggdgsdqhssyor2hl90k.png)



The current project structure looks like this.
```go
➜  go-stats git:(master) ✗ tree -L 3   
.
├── Dockerfile
├── client
│   ├── client.go
│   ├── clientproxy.go
│   └── iclient.go
├── cmd
│   └── main.go
├── config
│   └── config.go
├── db
│   └── db.go
├── go.mod
├── go.sum
├── gostats_openapi3.go
├── gostats_openapi3.json
├── svc.go
├── svcimpl.go
├── transport
│   ├── grpc
│   │   ├── annotation.go
│   │   ├── gostats.pb.go
│   │   ├── gostats.proto
│   │   └── gostats_grpc.pb.go
│   └── httpsrv
│       ├── handler.go
│       ├── handlerimpl.go
│       └── middleware.go
└── vo
    └── vo.go

8 directories, 21 files
```

We can see `go-doudou` generated `httpsrv` package that is for implementing RESTful. We can simply remove it or leave it there. Then let's see what changed in `svcimpl.go` file.
{% embed https://gist.github.com/wubin1989/b3de5d9c03d5a8d9d5aceba90cd9f742 %}

We can see line 17 and line 36-42 are new code. Line 17 is for making sure pointer type GoStatsImpl is always implementing GoStats interface. Line 36-42 are stub code for us to finish.
{% embed https://gist.github.com/wubin1989/497e0aa448b8e55c4aab5a0a961e7cad %}

`go-doudou` provides us a helper function LargestRemainder in `github.com/unionj-cloud/go-doudou/v2/toolkit/numberutils` package. We omit the algorithm explain here as it is not the concern.

Now we can implement LargestRemainderRpc method like this:
{% embed https://gist.github.com/wubin1989/55517a5f4c861ee10c51fc755eb33e9d %}

We don't need to worry about manually converting from protoc generated struct to vo struct as each fields in one has the same json tag with the other, so that we can use DeepCopy function in `github.com/unionj-cloud/go-doudou/v2/toolkit/copier` package to help us do the conversion.


## Test Service
We start our service again. Don't forget to run `go mod tidy` as we generated new code and imported new dependencies while we were coding.
```go
➜  go-stats git:(master) ✗ go mod tidy && go run cmd/main.go
2022/11/23 13:18:13 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
                           _                    _
                          | |                  | |
  __ _   ___   ______   __| |  ___   _   _   __| |  ___   _   _
 / _` | / _ \ |______| / _` | / _ \ | | | | / _` | / _ \ | | | |
| (_| || (_) |        | (_| || (_) || |_| || (_| || (_) || |_| |
 \__, | \___/          \__,_| \___/  \__,_| \__,_| \___/  \__,_|
  __/ |
 |___/
2022-11-23 13:18:13 INF ================ Registered Services ================
2022-11-23 13:18:13 INF +------------------------------------------+----------------------+
2022-11-23 13:18:13 INF |                 SERVICE                  |         RPC          |
2022-11-23 13:18:13 INF +------------------------------------------+----------------------+
2022-11-23 13:18:13 INF | go_stats.GoStatsService                  | LargestRemainderRpc  |
2022-11-23 13:18:13 INF | grpc.reflection.v1alpha.ServerReflection | ServerReflectionInfo |
2022-11-23 13:18:13 INF +------------------------------------------+----------------------+
2022-11-23 13:18:13 INF ===================================================
2022-11-23 13:18:13 INF Grpc server is listening at [::]:50051
2022-11-23 13:18:13 INF Grpc server started in 1.001238ms
```

Personally, I prefer to use [evans](https://github.com/ktr0731/evans) to test gRPC.
```go
➜  go-stats git:(master) ✗ evans -r repl -p 50051

  ______
 |  ____|
 | |__    __   __   __ _   _ __    ___
 |  __|   \ \ / /  / _. | | '_ \  / __|
 | |____   \ V /  | (_| | | | | | \__ \
 |______|   \_/    \__,_| |_| |_| |___/

 more expressive universal gRPC client


go_stats.GoStatsService@127.0.0.1:50051> show service
+----------------+---------------------+-----------------+-----------------------------+
|    SERVICE     |         RPC         |  REQUEST TYPE   |        RESPONSE TYPE        |
+----------------+---------------------+-----------------+-----------------------------+
| GoStatsService | LargestRemainderRpc | PercentageReqVo | LargestRemainderRpcResponse |
+----------------+---------------------+-----------------+-----------------------------+

go_stats.GoStatsService@127.0.0.1:50051> service GoStatsService

go_stats.GoStatsService@127.0.0.1:50051> call LargestRemainderRpc
<repeated> data::value (TYPE_INT32) => 20
<repeated> data::key (TYPE_STRING) => apple
<repeated> data::value (TYPE_INT32) => 30
<repeated> data::key (TYPE_STRING) => banana
<repeated> data::value (TYPE_INT32) => 40
<repeated> data::key (TYPE_STRING) => pear
<repeated> data::value (TYPE_INT32) => 
places (TYPE_INT32) => 2
{
  "data": [
    {
      "key": "apple",
      "percent": 22.22,
      "percentFormatted": "22.22%",
      "value": 20
    },
    {
      "key": "banana",
      "percent": 33.33,
      "percentFormatted": "33.33%",
      "value": 30
    },
    {
      "key": "pear",
      "percent": 44.45,
      "percentFormatted": "44.45%",
      "value": 40
    }
  ]
}
```

See, we input apple 20kg, banana 30kg, pear 40kg and we want 2 digits after dot, then we get expected result that 22.22 + 33.33 + 44.45 = 100.


## Summary
In this tutorial, we learned basic skill about how to develop gRPC service with go-doudou microservice framework. `go-doudou` not only can help you build your gRPC service to implement your business logic, but also contains full of service government features to help you build entire microservice system. `go-doudou` is young, but very promising, hope more and more developers can join us to make contributions.