Handling JSON strings and large files in Integration Server
Tags: json, integration, data, tutorial
Published: 2021-08-13 09:32:26

---
title: Handling JSON strings and large files in Integration Server
published: true
date: 2021-03-23 09:23:14 UTC
tags: #json, #integration, #data, #tutorial
canonical_url: https://tech.forums.softwareag.com/t/handling-json-strings-and-large-files-in-integration-server/239719/1
---

## Description

Previous versions of Integration Server provided support for handling JSON content using built-in services for converting JSON strings and files to IData format and back. With Integration Server, version 10.7, this capability is extended using new built-in services for iterating over and extracting data or data fragments from large JSON files.

In this tutorial, we will look at how to:

- Create a simple flow service that converts JSON content to an IData document, transforms the IData document, and finally converts the transformed data back to JSON format.

- Create a flow service that parses a large JSON file containing multiple arrays, retrieves specific array elements, and finally converts them to an IData document.

> **Note** : For information about the built-in JSON services, see the “ **JSON Folder** ” chapter in _webMethods_ _Integration Server Built-In Services Reference._

### Downloadable Resources

[Custom\_Package.zip](/uploads/short-url/63w8GWU6X2a9lULifvWF5vzZsix.zip) (13.5 KB)

The package developed in this tutorial is attached for reference. To explore the services and connections between them further, download the zip, install the included **Custom\_Package** in your Integration Server instance, and open the package in Designer. For information on installing a package published by another server, see the **Managing Packages** chapter in _webMethods Integration Server Administrator’s Guide_.

![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/f/f65026faf4269f22174677376c2ae20e1e160e98.png)

## Transforming JSON content using built-in services

First, let’s consider a simple use case where some JSON content sent to Integration Server needs to be transformed and returned as a JSON string. Let’s assume the JSON content is an array of employees’ first name, last name, and email address which needs to be transformed so that the first name and last name of each employee is combined into a single name.

For this use case, we will create a flow service that:

- Converts the incoming JSON content to IData document using the `pub.json:jsonStringToDocument` service.
- Transforms the IData document to concatenate the first name and last name using the MAP flow step and the `pub.string:concat` service.
- Converts the transformed IData document to a JSON using the `pub.json:documentToJSONString` service.

> **Note** : The flow service that you create in this procedure is available as the **Tranform\_JSON\_Content** service in downloadable **Custom\_Package**.

1. Create an empty flow service. For example, `Transform_JSON_Content`.  
 ![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/1/13ede4ea4f04e66f6c55e6895e642b7f3edf98a6.jpeg)
2. Add the input and output fields that correspond to the incoming JSON string and the returned JSON string respectively.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/7/75b2e0724f9c2ff5e9ffdd6b718b46b3c17f5e4f.png)
3. Add the `pub.json:jsonStringToDocument` service and perform the following steps:
  a. In the Pipeline view of the service, add the following sample incoming JSON content in the **jsonString** field.
```
{
"employees": [{
  	"fname": "Warren",
  	"lname": "Moody",
  	"email": "Warren.Moody@abc.com"
  },
  {
  	"fname": "Sam",
  	"lname": "Brown",
  	"email": "Sam.Brown@abc.com"
  },
  {
  	"fname": "Tina",
  	"lname": "Francis",
  	"email": "Tina.Francis@abc.com"
  },
  {
  	"fname": "Daisy",
  	"lname": "Hall",
  	"email": "Daisy.Hall@abc.com"
  },
  {
  	"fname": "Erik",
  	"lname": "Simon",
  	"email": "Erik.Simon@abc.com"
  }
]
}
```
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/f/f1c4700dcfda2c43f001fdbf487a18ae7b3eb6a9.jpeg)
This service gets the JSON content and converts it to a document object (IData format).
  b. Connect the output of this service to **inEmps** document and drop the other parameters in Pipeline Out.

> **Note** : Drop unnecessary output parameters throughout the procedure to keep the pipelines clean.

4.Add a **MAP** flow step, select the source and target array variables, and create a **ForEach** mapping between the two variables.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/9/93d41fde635bbf391e1eb5923a150e222d219b06.png)

> **Note** : For information about using MAPs in flow services, see the “ **Mapping Data in Flow Services** ” chapter in _webMethods Service Development Help_.

5.In the **ForEach** mapping, add the `pub.string:concat` service. The ForEach loop allows you to consider one array element at a time for transformation.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/e/ec0fcdfed3d601a7c639f0a901acbbfc329e8438.png)
6.In the Pipeline view of the **MAP** , perform the following steps:
  a. Connect the **fname** input parameter to a string field in the `pub.string:concat` service and add a space in the value of the other string field. This step adds a space after the first name of the employee.
  b. Connect the output of the `pub.string:concat` service to the **fname** output parameter in Pipeline Out.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/a/a8262d456c50a6f5da3ab50d16f00524013dd1e1.png)
7.Add a second **MAP** and perform the following steps in the Pipeline view:
  a. Connect the **fname** and **lname** fields to the string parameters in the `pub.string:concat` service and map the output of the service to the name parameter. This step concatenates the first name and the last name of the employee into a single name.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/7/75e403d1863518be061a031046471c4211416120.png)
  b. Connect the email parameters in the **MAP**.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/7/74d6039d9730ab860c986a78d93e6f5ba2220a86.png)
8.Add the `pub.json:documentToJSONString` service to convert the transformed content in the IData format back to JSON format.
  a. Connect **outEmps** to **document**.
  b. Set **prettyPrint** value to `true` to format the JSON content for human readability.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/e/e5a5e24529dbba0978be4ed1bba823bc0a9d745c.jpeg)
9.Save and run the `Transform_JSON_Content` flow service. The transformed JSON content in this scenario will look like this:
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/e/e50ddb8ce79d7e9366cff47bb25e15d44ea26401.png)

**Notes:**

- Similarly, you can use the `pub.json:jsonStreamToDocument` service to convert small JSON files into IData format. The only difference is that the `pub.json:jsonStreamToDocument` service accepts JSON input as stream data.

## Parsing large JSON files

The `pub.json:jsonStringToDocument` and `pub.json:jsonStreamToDocument` services that we used in the above use case are not suitable for parsing or converting large JSON files as they load the entire JSON content into memory at once, which can cause Integration Server to run out of memory.

> **Important:** A JSON file is considered large or small based on the memory available to the JSON services. The memory available to JSON services depends on the memory allocated to an Integration Server instance and the memory consumed by other services running on that instance. For example, if the maximum memory allocated to Integration Server is 512 MB, even a 100MB JSON file might cause Integration Server to run out of memory if multiple other services are contending for memory at the same time.

Integration Server enables you to overcome this bottleneck by using the `pub.client.getArrayIterator` and `pub.json:getNextBatch` services.

For this use case, we will create a flow service that:

- Streams a large JSON file using the `pub.file:getFile` service.
- Creates an iterator object using the `pub.json:getArrayIterator` service which enables the service to retrieve the required data in batches.
- Loops and retrieves the required elements in batches by invoking the `pub.json:getNextBatch` service within a REPEAT flow step.
- Closes the iteration using the `pub:json:closeArrayIterator` service after all the required array elements are retrieved.

> **Note** : A small JSON file is used in this use case only for demonstration purposes. However, the same procedure is applicable to larger JSON files.

### Procedure

1. Create an empty flow service. For example, `Large_JSON_iterator`.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/c/c318fd140aef931d9055471f0f53a9699bb405c7.png)
2. Create a sample JSON file, `sample_input1.json` with the following content:

```
 {
  "degree": [
      {
        "branch": [
        "science",
        "arts",
        "commerce"
       ]
     },
    {
    "science": [
      {
        "combination": [
        "physics",
        "chemistry",
        "biology"
      ]
     },
    {
      "combination": [
        "physics",
        "chemistry",
        "botany"
     ]
    },
    {
      "combination": [
        "physics",
        "chemistry",
        "zoology"
        ]
      }
     ]
   }
 ]
}

```
3.Add the `pub.file:getFile` service and perform the following steps:
  a. Enter the path of the JSON file in the **filename** input parameter.
  b. Set the value of the **loadAs** input parameter as `stream`.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/7/7c90c94415b58f8064bd1d87fe1ffabe29088ea2.png)
This service gets the JSON file and passes it as input stream to the  
`pub.json:getArrayIterator` service.
4.Add the `pub.json:getArrayIterator` service. This service creates an iterator object required to parse the arrays in the JSON input stream.
  a. Link the stream output parameter of the `pub.file:getfile` service to the **jsonSteam** input parameter of the `pub.json:getArrayIterator` service.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/5/567ae667d476814a02063b70507195dae76532b0.png)
  b. Configure the **arrayPaths** input parameter based on the array elements that you want to get from the JSON file. For example, to get the **branch** array and the first **combination** array element from the **science** array, add two **arrayPaths** as shown in the following image.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/f/fdc3445844c86a52c1ba43ce66e2936f302abdd8.png)

> **Note** : The array paths must follow the [JSON Pointer Notation RFC6901](https://tools.ietf.org/html/rfc6901#:~:text=RFC%206901%20JSON%20Pointer%20April%202013%20The%20ABNF,not%20conform%20to%20this%20syntax%20(see%20Section%207).).

5.Add the `pub.json:getNextBatch` service.

> **Note** : The iterator parameter of the `getArrrayIterator` service is an obvious input to the `getNextBatch` service and Designer automatically connects it to the `iterator` parameter of the `getNextBatch` service.

  a. Set a value for **batchSize**. In this example, the **batchSize** is `2`.
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/1/1a26bda2fe8322cb04b13610e26a2add2a6c6fc5.png)

Since the specified array paths may contain numerous array elements, run this service repeatedly until there are no more array elements.

> **Note** : For information about using flow steps such as REPEAT, BRANCH, LOOP, and EXIT while building flow services, see the **Building Flow Services** chapter in _webMethods Service Development Help._

6.Add a **REPEAT** step and set the `Repeat on` property to `SUCCESS`. Indent the `pub.json:getNextBatch` service under **REPEAT**.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/1/1fd0b17acf787ffa35b4b23503a9469e652bb192.png)
7.Add a **BRANCH** step and set the `Switch` property to `iterationStatus/hasNext` and the `Evaluate labels` property to `false`. This step ensures that the `pub.json:getNextBatch` service iterates over array paths until all the array elements are retrieved.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/b/b6cde330292a4ce14145252fd7726c3ed31edd04.png)
8.Add an **EXIT** step and indent it under the **BRANCH** step. Set the `Label` property to `false` and the `Exit from` property to `$loop`. This step terminates the `pub.json:getNextBatch` service when `iterationStatus/hasNext` is `false`.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/8/846f5b179da8b383f8e30d3cdd04457031cb3137.jpeg)
9.Add the `pub.json:closeArrayIterator` service. This service closes the iterator used by the `pub.json:getNextBatch` service.  
![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/9/9e87cb518be08d8adc4352aff74ee8d0b98c52e7.png)
10.Save and run the flow service.

The Debugging feature allows you to see the stepwise results of a flow service. For more information, see the **Debugging Flow Services** chapter in _webMethods Service Development Help_ .

### Flow service results

**Iteration 1** : The `pub.json:getNextBatch` service parses the `/degree/0/branch` array and returns the first 2 elements as the batch size is 2. Since the last element remains, the `hasNext` property is `true`.  
 ![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/2/20ec88743ef6d6450e4e87c7d615c41c0e4bc551.png)

**Iteration 2** : Returns the last element in the branch array. The `hasNext` property remains true as the service is yet to parse `/degree/1/science/0/combination`.  
 ![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/4/49cb5ae5c8e1a6b3d6c1c8b5c615d549c048c732.png)

**Iteration 3** : Returns the first 2 elements in `/degree/1/science/0/combination`.  
 ![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/f/f09fe4c1857322a17a7f9944e78f71cce0414500.png)

**Iteration 4** : Returns the last element in `/degree/1/science/0/combination`, and then the value of `hasNext` becomes `false`. The service execution stops.  
 ![image](https://aws1.discourse-cdn.com/techcommunity/original/2X/3/3fd77eb3cf9443b7de11ba320695901dcbbdde6f.png)

### Additional information

In addition to the services covered in this tutorial, the JSON folder also provides the `pub.json.schema:validate` service that can be used to validate JSON content against a JSON document type.

### References

[webMethods Integration Server Built-In Services Reference, version 10.7](https://documentation.softwareag.com/webmethods/microservices_container/msc10-7/10-7_MSC_PIE_webhelp/index.html#page/integration-server-integrated-webhelp%2Fto-json_folder.html%23)  
[webMethods Service Development, version 10.7](https://documentation.softwareag.com/webmethods/microservices_container/msc10-7/10-7_MSC_PIE_webhelp/index.html#page/integration-server-integrated-webhelp%2Fesb.flow.html%23)

[Read full topic](https://tech.forums.softwareag.com/t/handling-json-strings-and-large-files-in-integration-server/239719/1)