Consume Large JSON Response in Spring
Tags: java, api, performance
Published: 2022-10-08 21:01:02

## Background
In the previous post **[Streaming Large JSON Response in Spring.](https://dev.to/ratulsharker/streaming-large-json-response-in-spring-2pho)** we generated large json as response. In this article, we will consuming that response, without heap memory overflow.

## Goal
In this post, the sole target is to consume that large json response and as an exercise do some basic calculation as a proof of concept about receiving the json response.

As a reference we receive following json contents from the server application

```json
{
    "employees" : [
        {
            "empNo":10001,
            "birthDate":"02 September 1953",
            "firstName":"Georgi",
            "lastName":"Facello",
            "gender":"M",
            "hireDate":"26 June 1986"
        }
        ... ~300K employee entries
    ]
}
```

We will be counting following

- Number of female employee.
- Number of male employee.
- Total number of employee.

NB: The statistics done with the employee list is for demonstration purpose and no way near to real world scenario. Do not take these examples here literally, like counting the number of total, male and female employees from employee list is not the ideal way. Better calculate at database end using aggregate functions.

We will be creating a simple "Spring Web Application" using 

- Spring Boot (2.7.4)
- Spring Web.
- Thymeleaf.
- Lombok and Spring Boot Devtools.

## Implementation

Start with the enum and pojo declaration

**Gender.java**
```java
public enum Gender {
    M,F;
}
```

**Employee.java**
```java
@Getter
@Setter
@NoArgsConstructor
public class Employee {

    private Long empNo;

    private Date birthDate;

    private String firstName;

    private String lastName;

    private Gender gender;

    private Date hireDate;
}
```

**EmployeeStats.java**
```java
@Getter
@Setter
@NoArgsConstructor
public class EmployeeStats {
	
    private Long totalEmployeeCount;
    
    private Long femaleEmployeeCount = 0L;

    private Long maleEmployeeCount = 0L;

    private Long reportGenerationDelayInSeconds;

    public void incrementFemaleEmployeeCount() {
        femaleEmployeeCount++;
    }

    public void incrementMaleEmployeeCount() {
        maleEmployeeCount++;
    }
}
```

Nothing fancy here, just a stats pojo, which will be holding all the statistics with some convenient method for doing the counting.

We will be using `RestTemplate` for the HTTP communication. So lets configure a bean for `RestTemplate`

**WebConfig.java**
```java
@Configuration
public class WebConfig {
	
    @Bean
    public RestTemplate restTemplate() {
        RestTemplate restTemplate = new RestTemplate();
        for(HttpMessageConverter<?> converter : restTemplate.getMessageConverters()) {
            if(converter instanceof MappingJackson2HttpMessageConverter) {
                MappingJackson2HttpMessageConverter jacksonConverter = (MappingJackson2HttpMessageConverter) converter;
                ObjectMapper objectMapper = jacksonConverter.getObjectMapper();
                objectMapper.setDateFormat(new SimpleDateFormat("dd MMMM YYYY"));
			}
        }
        return restTemplate;
    }

    @Bean
    public ObjectMapper restTemplateObjectMapper(RestTemplate restTemplate) {
        for(HttpMessageConverter<?> converter : restTemplate.getMessageConverters()) {
            if(converter instanceof MappingJackson2HttpMessageConverter) {
                return ((MappingJackson2HttpMessageConverter) converter).getObjectMapper();
            }
        }

        return null;
    }
}
```

Important thing to notice here, we customised the date format for converter `MappingJackson2HttpMessageConverter` to align with the json response date format.

An extra bean of `ObjectMapper` is also exposed from the `RestTemplate`'s converter. We will be needing that next, while deserialising the json response.


### Receiving response chunk by chunk

Now the most lucrative part of this whole writing, doing the HTTP request and receive the response. From the previous article **[Streaming Large JSON Response in Spring](https://dev.to/ratulsharker/streaming-large-json-response-in-spring-2pho)** it is obvious that, loading the whole response in a pojo will exhaust the heap memory. Here we will not receive the whole response at once, we will connect a `JsonParser` into the response's input stream and parse received json as per our need. Thus we will be moving along with the whole response without actually loading it fully in the memory.

**EmployeeReportService.java**
```java
@Service
@RequiredArgConstruct
public class EmployeeReportService {
	
    private final RestTemplate restTemplate;
    private final ObjectMapper objectMapper;

    public Long fetchAllEmployee(Consumer<Employee> employeeConsumer) {

        String url = "http://localhost:8080/employees?stream=true";

        return restTemplate.execute(url, HttpMethod.GET, null, (response) -> {

            Long employeeCount = 0L;
			
            JsonParser jsonParser = objectMapper.getFactory().createParser(response.getBody());

            if(jsonParser.nextToken() == JsonToken.START_OBJECT) {
                if(jsonParser.nextFieldName() == "employees") {
                    if(jsonParser.nextToken() == JsonToken.START_ARRAY) {
                        while(jsonParser.nextToken() != JsonToken.END_ARRAY) {
                            Employee employee = jsonParser.readValueAs(Employee.class);
                            employeeConsumer.accept(employee);
                            employeeCount++;
                        }
                    }
                }
            }

            jsonParser.close();

            return employeeCount;
        });

    }
}
```

Actually what is happening here, 

- We initiate the HTTP request using the `RestTemplate` and pass a `ResponseExtractor` implementation.
- The `ResponseExtractor` implementation starts with getting the `inputStream` from the response body.
- Prepared a `JsonParser` using the `objectMapper` configured before and `inputStream` from the response body.
- Then some trivial checking (starting of object `{` token check, starting the field `"employees"` token, starting of the array `[` token), then we start reading the `employee` object one by one, until we find the end of the array token `]`. 
- We pass the parsed `employee` object to the consumer, to process the employee as per the callee need and doing some trivial counting to return later.
- At the end `close()` the `jsonParser`.

### Using the response

Now time to calculate the statistics based on the received employee one by one

**EmployeeReportService.java**
```java
@Service
public class EmployeeReportService {

    .... Previous Code
    
    public EmployeeStats prepareEmployeeStats() {
        final Long startTime = System.currentTimeMillis();
        final EmployeeStats employeeStats = new EmployeeStats();

        final Long totalEmployeeCount = fetchAllEmployee((employee) -> {
            switch (employee.getGender()) {
                case F:
                    employeeStats.incrementFemaleEmployeeCount();
                    break;
                case M:
                    employeeStats.incrementMaleEmployeeCount();
                    break;
            }
        });

        final Long endTime = System.currentTimeMillis();

        employeeStats.setReportGenerationDelayInSeconds((endTime - startTime) / 1000);
        employeeStats.setTotalEmployeeCount(totalEmployeeCount);

        return employeeStats;
    }
}
```

Above method is pretty much self explained. It's using our previously declared `fetchAllEmployee` method to fetch all the employee and counting based on declared rule. To keep track of the required time to prepare the statistics, we also calculated the number of seconds required to generate the statistics.

Now the controller for rendering the statistics into the browser

**EmployeeController.java**
```java
@Controller
@RequiredArgsConstructor
public class EmployeeController {
    
    private final EmployeeReportService employeeReportService;

    @GetMapping("/employee-stats")
    public String getEmployeeStats(Model model) {
        EmployeeStats employeeStats = employeeReportService.prepareEmployeeStats();
        model.addAttribute("stat", employeeStats);
        return "employee-stats";
    }
}
```

and the template 

**employee-stats.html**
```html

... Html

<table border="6">
    <thead>
        <th>Reference</th>
        <th>Value</th>
    </thead>
    <tbody>
        <tr>
            <!-- Because ladies first :) -->
            <td>Female Employee Count</td>
            <td class="value" th:text="${stat.femaleEmployeeCount}"></td>
        </tr>
        <tr>
            <td>Male Employee Count</td>
            <td class="value" th:text="${stat.maleEmployeeCount}"></td>
        </tr>
        <tr>
            <td>Total Employee Count</td>
            <td class="value" th:text="${stat.totalEmployeeCount}"></td>
        </tr>
        <tr>
            <td>Preparation Delay</td>
            <td class="value" th:text="${stat.reportGenerationDelayInSeconds + ' seconds'}"></td>
        </tr>
    </tbody>
</table>

... More Html

```

And the last thing is

```properties
server.port=8081
```

So our consumer application is running on port `8081`.


## Output

Both Server & Consumer application is running with jvm arg `-Xmx32m` to demonstrate the heap size limitation.

![Employee Statistics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/55i6tox4iarqroo7c63j.png)

Clearly the response took longer than the expected because of the way we are counting these values. This example is just for demonstration purpose and to keep the scope of this writing small.

All the code above can be found into my [github repository](https://github.com/ratulSharker/stream-json-response/tree/main/consumer).

[CSV Generation From Large JSON Response in Spring](https://dev.to/ratulsharker/csv-generation-from-large-json-response-in-spring-2bnn) this post coveres more appropriate usage of this streaming based response consuming.