I am trying to merge all json data from paginated api responses with Go, this is how it looks like
Tags: go, restapi, json
Published: 2023-08-31 12:49:43

## Background
Once upon a time, I need to consume list of users data from an api which uses json response. As I predicted earlier, the api provider creates a pagination responses when the consumer interacts with list. Unfortunately I need to consume all users which means I have to hit every possible pagination query parameter to fetch all users. 

here's the api endpoint

`.../api/v2/users?page=1&per_page=4`

and here's the api response
```
{
  "meta": {
    "page": 1,
    "per_page": 5,
    "total_pages": 100
  },
  "data": [
    {
      "id": "ef5b4299-b8d0-40c2-8442-2fdc90e342ae",
      "firstname": "Cale",
      "lastname": "Schmidt",
      "email": "Liana8@gmail.com"
    },
    {
      "id": "c0328632-1f18-4cf6-ad53-17278ffcc7bb",
      "firstname": "Norval",
      "lastname": "Feest",
      "email": "Eleanora.Gleason55@gmail.com"
    },
    {
      "id": "0c467516-444f-4d01-b40d-82f7a646820a",
      "firstname": "Muriel",
      "lastname": "Berge",
      "email": "Clarissa_Simonis44@hotmail.com"
    },
    {
      "id": "2942a4b0-eac4-463c-87e3-6c7d0406cc81",
      "firstname": "Cletus",
      "lastname": "Considine",
      "email": "Casper.Kris77@gmail.com"
    },
    {
      "id": "2942a4b0-eac4-463c-87e3-6c7d0406cc81",
      "firstname": "Cletus",
      "lastname": "Considine",
      "email": "Casper.Kris77@gmail.com"
    }
  ]
}
```

First thing that comes to my mind when i face this problem is i go to first page and get last page information and then i will iterate to reach every possible page and merge every user from api response. To improve performance then I need to do it asynchronously for each page. As a result the user data I manage to merge is not sequential and I'm ok with that.

## Code
I developed a module which provide the functionality as i mention earlier. You can get it in this [repository](https://github.com/Mhakimamransyah/go-pagination-aggregate).

### Usage
Create a struct that representing json response which bind api response to a static data type.

```
type Meta struct {
    Page    int `json:"page"`
    PerPage int `json:"per_page"`
    TotalPages int `json:"total_pages"`
}

type User struct {
    Uuid  string `json:"id" gorm:"column:id;primaryKey"`
    FirstName string `json:"firstname" gorm:"column:firstname"`
    LastName  string `json:"lastname" gorm:"column:lastname"`
    Email     string `json:"email" gorm:"column:email"`
}

type Response struct {
    Meta Meta   `json:"meta"`
    User []User `json:"data"`
}

func (resp *Response) GetBoundary() int {
	return resp.Meta.TotalPages
}
```
In case you're asking about tag "gorm" , yes i will use it. 
Then i create module instance with some configurations like this
```
type Meta struct {
    Page    int `json:"page"`
    PerPage int `json:"per_page"`
    TotalPages int `json:"total_pages"`
}

type User struct {
    Uuid  string `json:"id" gorm:"column:id;primaryKey"`
    FirstName string `json:"firstname" gorm:"column:firstname"`
    LastName  string `json:"lastname" gorm:"column:lastname"`
    Email     string `json:"email" gorm:"column:email"`
}

type Response struct {
    Meta Meta   `json:"meta"`
    User []User `json:"data"`
}

func (resp *Response) GetBoundary() int {
	return resp.Meta.TotalPages
}

func main() {

   var jsonPages Response

  pag, err := paginationaggregator.NewPaginationAggregator( 
    &paginationaggregator.PaginationAggregatorConfig{
      URL:"http://127.0.0.1:3000/api/v2/users?per_page=5&page=%d",
      Client: &http.Client{},
      JsonPage: &jsonPages,
      Concurrent: 5,
    },
  )

}
```
As you can see, the library provides `Concurrent` configuration. This configuration means if you need to make 100 requests for each page then there will be 20 batch asynchronous requests or each batch will make 5 asynchronous requests until all page fetched. I think it will prevent api server considers all these requests as DDOS attacks and prevents requests from hitting rate limiters.

### Insert user to database
After successfully retrieving all user then I need to insert it into database only if user email does not exist. Rather than waiting for all user successfully retrieved, the library provides a function so we can process it in each batch.

```
type Meta struct {
   Page       int `json:"page"`
   PerPage    int `json:"per_page"`
   TotalPages int `json:"total_pages"`
}

type User struct {
   Uuid      string `json:"id" gorm:"column:id;primaryKey"`
   FirstName string `json:"firstname" gorm:"column:firstname"`
   LastName  string `json:"lastname" gorm:"column:lastname"`
   Email     string `json:"email" gorm:"column:email"`
}

type Response struct {
   Meta Meta   `json:"meta"`
   User []User `json:"data"`
}

func (resp *Response) GetBoundary() int {
   return resp.Meta.TotalPages
}

var Mysql *gorm.DB

func main() {

  var jsonPages Response

  Mysql = connectMysql()
  defer func() {
     sql, _ := Mysql.DB()
     sql.Close()
  }()

  pag, err := paginationaggregator.NewPaginationAggregator(
   &paginationaggregator.PaginationAggregatorConfig{
      URL:"http://127.0.0.1:3000/api/v2/users?per_page=5&page=%d",
      Client: &http.Client{},
      JsonPage: &jsonPages,
      Concurrent: 5,
      ConcurrentBatch: processingBatch,
     },
  )

  if err != nil {
     fmt.Println(err)
     return
  }

  res, err := pag.Get()

  if err != nil {
     fmt.Println(err)
     return
  }

  for _, val := range res {
     fmt.Println(val.Response.Data)
  }
}

// select and insert user for every batch
func processingBatch(
    batchResult []paginationaggregator.HttpInteraction
) error {

   var newUsers []User

   for _, val := range batchResult {

      resp := Response{}

      if err := json.Unmarshal([]byte(val.Response.Data), 
                   &resp); err != nil {
	     continue
      }

      for _, user := range resp.User {

	   userTmp := User{}
	   
           // check existing user by email
           if err := Mysql.Where("email = ?", 
                user.Email).First(&userTmp).Error; err != nil {

		      if errors.Is(err, gorm.ErrRecordNotFound) {
                       // users not exists so append it
		          newUsers = append(newUsers, user)
		      }

		     continue

	      }
	   }
    }

    if err := Mysql.Create(&newUsers).Error; err != nil {
         return err
    }

    return nil
}

func connectMysql() *gorm.DB {

    dsn := "root:yourpassword@tcp(127.0.0.1:3306)/peoples"

    DB, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
		Logger: logger.Default.LogMode(logger.Silent),
    })

    if err != nil {
	panic(err)
    }

    return DB
}
```
## Reference

- https://github.com/Mhakimamransyah/go-pagination-aggregate