oq - A portable/performant jq wrapper Part 2
Tags: jq, json, python, go
Published: 2021-02-27 15:35:03

---
title: oq - A portable/performant jq wrapper Part 2
published: true
description: Revisiting oq, yq (python), yq (go), and jq performance benchmarks
tags: jq, json, python, go
---

# OQ Revisited

Over a year ago I published a [blog post](https://dev.to/blacksmoke16/oq-a-portable-performant-jq-wrapper-18ka) introducing my [jq](https://github.com/stedolan/jq) wrapper [oq](https://github.com/blacksmoke16/oq).  Given it has been quite a while since then I decided to rerun the benchmarks I did in the original post with the goal of seeing if things changed, both in regards to `oq` itself and the greater ecosystem.

## Methodology

I tested the latest versions of the packages in the first post.  I also added the Go version of `yq`.

The tested packages, and versions, for the benchmarks include:

* [jq](https://github.com/stedolan/jq) (`1.6`) - Installed via `pacman -S jq`
* [yq (go)](https://mikefarah.gitbook.io/yq/) (`4.6.1`) - Downloaded latest binary from the latest [GH Release](https://github.com/mikefarah/yq/releases)
* [yq (python)](https://kislyuk.github.io/yq/) (`2.12.0`) - Installed via `pip3 install yq`
* [oq](https://blacksmoke16.github.io/oq/) (`1.2.0`) - Built locally with Crystal `0.36.1` via `shards build --production --release`

Each package was tested against the same 3 benchmarks in my original post, plus an additional benchmark for `XML` => `JSON`, mainly since `yq (python)` has a dedicated command for handling `XML` input/output that I didn't account for in the first post.  The data is gathered via the [GNU Time](https://www.gnu.org/software/time/) command.  E.g. `/usr/bin/time -v`.

### Setup

OS: `5.10.15-1-MANJARO x86_64`
CPU: `Intel i7-7700k`
Memory: `32GB @ 3,000 MHz`
SSD: `Samsung 970 EVO Plus - 1TB`

## Results

This section summarizes the results, with key metrics in table form.  See the appendix for the raw benchmark data.

### Simple

Simply doing a pass through `JSON` => `JSON` then counting the lines on a pretty small input file:

```json
{
  "guests": [
    {
      "name": "Jim",
      "age": 17,
      "numbers": [
        1,
        2,
        3
      ]
    },
    {
      "name": "Bob",
      "age": 51,
      "numbers": [
        4,
        5,
        6
      ]
    },
    {
      "name": "Susan",
      "age": 85,
      "numbers": [
        7,
        8,
        9
      ]
    }
  ]
}
```

| Name        | Max Memory (MB) | Wall Clock Time (seconds) |
| ----------- | --------------- | ------------------------- |
| jq          | 3.04            | 0.02                      |
| yq (go)     | 6.02            | 0.01                      |
| yq (python) | 14.728          | 0.07                      |
| oq          | 6.536           | 0.07                      |

All packages performed relatively equally, nothing noteworthy.

### Jeopardy.json

Getting the `length` of a 53MB `JSON` file.

The file used: [jeopardy.json](https://drive.google.com/file/d/0BwT5wj_P7BKXb2hfM3d2RHU1ckE/).

| Name        | Max Memory (MB) | Wall Clock Time (seconds) |
| ----------- | --------------- | ------------------------- |
| jq          | 230.32          | 0.66                      |
| yq (go)     | 705.292         | 2.54                      |
| yq (python) | 2,922.996       | 108.89                    |
| oq          | 230.304         | 0.74                      |

This test resulted in some more useful data, causing the true performance of each package to be more obvious.  `jq` and `oq` fared the best, being pretty comparable in both memory usage and execution time.  `yq (go)` was the next best, using 3x the memory and took ~3.5x longer than `oq`.  `yq (python)` was by far the worse; using 12.7x the memory and taking 147x longer than `oq`.

### YAML => XML/JSON

Consuming a ~57MB `YAML` file, and outputting `XML` and `JSON`.

The file used: `sde/bsd/invItems.yaml` from the [EVE Online SDE Export](https://cdn1.eveonline.com/data/sde/tranquility/sde-20190625-TRANQUILITY.zip).

| Name                   | Max Memory (MB) | Wall Clock Time (seconds) |
| ---------------------- | --------------- | ------------------------- |
| jq                     | N/A             | N/A                       |
| yq (go) - JSON         | 5,202.032       | 32.77                     |
| yq (python) - JSON     | 5,742.736       | 206.2                     |
| yq (python) - XML      | 5,742.704       | 217.17                    |
| oq - JSON              | 575.6           | 14.01                     |
| oq - JSON - SimpleYAML | 334.632         | 14.16                     |
| oq - XML               | 649.984         | 15.77                     |
| oq - XML - SimpleYAML  | 334.504         | 15.83                     |

The final test showed similar metrics as the previous one.  `jq` doesn't support non `JSON` formats so it was excluded.  `oq` did the best with the execution time being fairly stable, while the memory usage was halved when using the new `SimpleYAML` format for the input.  Interestingly `yq (go)` fared better execution wise in this test, being only 2.34x slower, but using ~15.5x more memory.  `yq (python)` once again was the worse, with memory usage slightly worse than `yq (go)`, but with an execution time of 15.5x slower than `oq`.

### XML => JSON

Consume the output of the last benchmark, and convert it back to `JSON`.  This test is mainly to not be biased as `yq (python)` includes a dedicated sub-package `xq` for handling `XML` input/output.

| Name        | Max Memory (MB) | Wall Clock Time (seconds) |
| ----------- | --------------- | ------------------------- |
| jq          | N/A             | N/A                       |
| yq (go)     | N/A             | N/A                       |
| yq (python) | 795.06          | 15.59                     |
| oq          | 2600.964        | 20.28                     |

Neither `jq` or `yq (go)` support `XML` input, so they were excluded.  The dedicated sub-package paid off for `yq (python)`, beating `oq` by a fair margin in both memory consumption and execution time.  It looks like it's able to stream the `XML` input, while `XML` input for `oq` is the only input format that _cannot_ be streamed (yet).

## Conclusion

It seems the performance of `yq (python)` has gotten a bit better since last time.  Shaving off over 2GB of memory and 2 minutes of execution time in the `jeopardy.json` benchmark.  However, it still is by far the slowest, and most memory hungry package (when not using `XML` as the input format).  `yq (go)` is in a better spot, but still has pretty high memory usage depending on the exact use case.  `oq` seems to still be the most efficient overall, with both memory and execution time being on par with `jq`.  Thanks to its focus on streaming of data when possible (all but `XML` input at the moment); `oq` is able to use much less memory comparatively.

## Appendix

### Simple

#### jq

```sh
Command being timed: "jq . data.json | wc -l"
User time (seconds): 0.02
System time (seconds): 0.00
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3040
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 288
Voluntary context switches: 1
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
```

#### yq (go)

```sh
Command being timed: "./go-yq -Pj e . data.json | wc -l"
User time (seconds): 0.01
System time (seconds): 0.00
Percent of CPU this job got: 105%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6020
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 617
Voluntary context switches: 76
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
```

Using the `-j` and `-P` options to replicate the default behavior of the other binaries.

#### yq (python)

```sh
Command being timed: "yq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.00
Percent of CPU this job got: 87%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 14728
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 3637
Voluntary context switches: 43
Involuntary context switches: 1
Swaps: 0
File system inputs: 1328
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
```

#### oq

```sh
Command being timed: "oq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.01
Percent of CPU this job got: 104%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6536
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 808
Voluntary context switches: 58
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
```

### Jeopardy.json

#### jq

```sh
216930
Command being timed: "jq length jeopardy.json"
User time (seconds): 0.59
System time (seconds): 0.06
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.66
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230320
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59454
Voluntary context switches: 1
Involuntary context switches: 26
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

#### yq (go)

```sh
216930
Command being timed: "./go-yq e length jeopardy.json"
User time (seconds): 3.41
System time (seconds): 0.22
Percent of CPU this job got: 143%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.54
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 705292
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 176999
Voluntary context switches: 787
Involuntary context switches: 163
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

#### yq (python)

```sh
216930
Command being timed: "yq length jeopardy.json"
User time (seconds): 108.62
System time (seconds): 0.86
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:48.89
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2922996
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 816903
Voluntary context switches: 13512
Involuntary context switches: 586
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

#### oq

```sh
216930
Command being timed: "oq length jeopardy.json"
User time (seconds): 0.70
System time (seconds): 0.12
Percent of CPU this job got: 110%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.74
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230304
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59966
Voluntary context switches: 13569
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

### YAML => XML/JSON

#### jq

N/A

#### yq (go)

##### JSON

```sh
Command being timed: "./go-yq -Pj e . invItems.yaml > invItems.go-yq.json"
User time (seconds): 47.66
System time (seconds): 1.46
Percent of CPU this job got: 149%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5202032
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1395040
Voluntary context switches: 37244
Involuntary context switches: 1922
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

#### yq (python)

##### JSON

```sh
Command being timed: "yq . invItems.yaml > invItems.python-yq.json"
User time (seconds): 205.35
System time (seconds): 1.82
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:26.20
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742736
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1587436
Voluntary context switches: 13384
Involuntary context switches: 1836
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

##### XML

```sh
Command being timed: "yq -s -x --xml-root items --xml-dtd {"item": .[] | .} invItems.yaml > invItems.python-yq.xml"
User time (seconds): 215.17
System time (seconds): 2.03
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:37.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742704
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1683522
Voluntary context switches: 32842
Involuntary context switches: 1031
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

#### oq

##### JSON

```sh
Command being timed: "./oq -i yaml . invItems.yaml > invItems.oq.json"
User time (seconds): 12.90
System time (seconds): 12.24
Percent of CPU this job got: 179%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 575600
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 231391
Voluntary context switches: 289799
Involuntary context switches: 166
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

```sh
Command being timed: "./oq -i simpleyaml . invItems.yaml > invItems.oq.simple.json"
User time (seconds): 12.27
System time (seconds): 12.18
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.16
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334632
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89229
Voluntary context switches: 2981745
Involuntary context switches: 246
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

##### XML

```sh
Command being timed: "./oq -i yaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 16.00
System time (seconds): 12.04
Percent of CPU this job got: 177%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 649984
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 249933
Voluntary context switches: 381174
Involuntary context switches: 266
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

Example output:

```xml
<?xml version="1.0" encoding="utf-8"?>
<items>
  <item>
    <flagID>0</flagID>
    <itemID>0</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  <item>
    <flagID>0</flagID>
    <itemID>1</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  ...
</items>
```

```sh
Command being timed: "oq -i simpleyaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 15.27
System time (seconds): 11.99
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.83
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334504
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89233
Voluntary context switches: 3029485
Involuntary context switches: 298
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

### XML => JSON

#### jq

N/A

#### yq (go)

N/A

#### yq (python)

```sh
Command being timed: "xq .items invItems.python-yq.xml > invItems.python-yq.xml.json"
User time (seconds): 16.37
System time (seconds): 0.39
Percent of CPU this job got: 107%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.59
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 795060
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 341577
Voluntary context switches: 14927
Involuntary context switches: 87
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```

#### oq

```sh
Command being timed: "./oq -i xml .items invItems.oq.xml > invItems.oq.xml.json"
User time (seconds): 20.25
System time (seconds): 16.87
Percent of CPU this job got: 183%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:20.28
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2600964
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 766666
Voluntary context switches: 2117666
Involuntary context switches: 1249
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```