GO Protobuf Encoding/Decoding integrated with C
Tags: go, c, json, proto
Published: 2019-10-24 10:29:02

Protobuf messages are gaining popularity across many domains. There are libraries like nanopb in C to support the protobuf encoding/decoding. However they are slow and also pose significant challenge in implementation. GoLang protobuf encoding/decoding is fast and provides support of unmarshalling data from JSON to proto format.

We will be creating a C shared library containing functions that read from json file, protobuf encode and decode.

__Full Code can be found here:__ {% github brundhasv/GoC-Protobuf %}

#GoLang Code
Lets start with _Student.proto_. I am using proto3 for this example:
```
syntax="proto3";

package main;

message Parents {
      string dad=1;
      string mom=2;
}

enum Gender {
    Female = 0;
    Male = 1;
}

message Student {
      string name = 1;
      int32 age = 2;
      Parents parents=3;
      Gender gender=4;
}

message Students {
      repeated bytes StudentEntry=1;
}
```
As you can see above proto includes enum, nested proto and repeated bytes. __Student__ represents a student's data, each student's data will be encoded and added to __Students__ StudentEntry and all the encoded entries will be encoded again.

Below is input JSON file _student_list.json_ which is used to populate proto fields. We have to use the same names and nested levels for the unmarshal of Json to proto.
```JSON
{
"0" : {
    "Name": "Elliot",
    "Age" : 15,
    "Parents" : {
            "Dad" : "George",
            "Mom" : "Elena"
    },
    "Gender" : 0
  },
"1" : {
    "Name": "David",
    "Age" : 14,
    "Parents" : {
            "Dad" : "Keith",
            "Mom" : "Sarah"
    },
    "Gender" : 1
   }
}
```
##Encoding Protobuf from JSON

To integrate with C, library C is used to convert GoData to C datatypes. Comment _export_ is used to export functions to C lib. 
We will create a new GoLang program file __student_en_dc.go__ and add encode and decode functions to it.
Json file is input parameter of this function __get_student_enbuf__ passed from C. This function returns encoded buffer and its length to C.
```
//export get_student_enbuf
func get_student_enbuf(student_json_file *C.char) (unsafe.Pointer,C.int) {
       .
       .
       return C.CBytes(students_en),C.int(int32(len(students_en)))
```

First step in encoding is to Unmarshall JSON to proto. As there are more than one student entries, I have added entries using number keys. This cannot be directly unmarshalled, we would have to extract json data of each student and then unmarshal. I am first unmarshalling json file data to an interface. Below is the code: 
```
jsonbytes, _ := ioutil.ReadAll(jsonFile)
var parsed_proto map[string]interface{}
json.Unmarshal(jsonbytes, &parsed_proto)
```
Then parse each interface to obtain student json data and unmarshall it into our proto. Aftermath proto is encoded as shown.
```
    for i=0;i<len(parsed_proto);i++ {
            selectedjb, err := json.Marshal(parsed_proto[strconv.Itoa(i)])
            if err != nil {
                log.Println("Error in reading json index",err)
                return nil,0
            }
            var parsed_jf map[string]interface{}
            //Each student data to proto
            json.Unmarshal(selectedjb, &parsed_jf)
            student := &Student{}
            //Proto Encoding of student
            json.Unmarshal(selectedjb, student)

```
Encoded data is added to StudentEntry of Students. Then Students data is encoded. We have the encoded buffer.
```
            students.StudentEntry=append(students.StudentEntry,student_en)
    }
    students_en, err := proto.Marshal(students)
```

##Decoding protobuf

_Students_ is decoded first following all the _student_ entries.

Function __decode_student_enbuf__ takes encoded data and its length as input passed from C and returns decoded buffer to C.
```
//export decode_student_enbuf
func decode_student_enbuf(data *C.char,leng C.int) *C.char {
           .
           .
           return C.CString(decoded_student)
}
```
Before decode, *char is converted to GoBytes and then unmarshalled to proto.
```
new_students := &Students{}
student_en:=C.GoBytes(unsafe.Pointer(data),leng)
err := proto.Unmarshal(student_en, new_students)
```
Decoded _students_ data is used to obtain each _StudentEntry_ and is decoded. Decoded student is appended to a string and is returned to C.
```
for i=0;i<len(new_students.StudentEntry);i++ {
        new_student := &Student{}
        err = proto.Unmarshal(new_students.StudentEntry[i], new_student)
        decoded_student=decoded_student+"\n"+proto.MarshalTextString(new_student)
```
#Building C-Shared file
Lets compile proto to generate _Student.pb.go_.Then build c-shared file.
```
protoc --go_out=. Student.proto
go build  -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go
```
_student.so_ and _student.h_ files are generated. 
Following lines would be present in the generated header file. Golang function _get_student_enbuf_ returns tuple(more than one return parameters), this is converted to struct when header files are generated as seen below.
```
/* Return type for get_student_enbuf */
struct get_student_enbuf_return {
        void* r0;
        int r1;
};

extern struct get_student_enbuf_return get_student_enbuf(char* p0);

extern char* decode_student_enbuf(char* p0, int p1);
```

#C Code 
From C side, generated header file is included. Students list JSON file is passed  to function _get_student_enbuf_, returned encoded buffer and its length is populated in struct stu. The same are passed to decode function _decode_student_enbuf_, the returned decoded string is printed.
Finally both encoded and decoded buffers are freed.
```
#include <stdio.h>
#include "student.h"
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>

struct get_student_enbuf_return stu;

int main() {
    stu = get_student_enbuf("student_list.json");
    int stu_list_len = stu.r1;
    if(stu.r0!=NULL) {
        char *decoded_stu = decode_student_enbuf((char*)stu.r0,stu.r1);
        if(decoded_stu != NULL){
                printf("%s",decoded_stu);
                free((char*)decoded_stu);
        }
    free((char*)stu.r0);
    }
    return 0;
}
```

##Compiling and executing C
Link with the created _student.so_, create object _student_ and run the object.
```
gcc -o student student_cgo.c ./student.so

./student
name: "Elliot"
age: 15
parents: <
  dad: "George"
  mom: "Elena"
>

name: "David"
age: 14
parents: <
  dad: "Keith"
  mom: "Sarah"
>
gender: Male

```

__Full Code can be found here:__ {% github brundhasv/GoC-Protobuf %}