Linking R to iRODS’ new HTTP API 🥳
Tags: openscience, data, rstats, irods
Published: 2024-03-18 07:14:02


> **_NOTE:_** 
> This article is about the release of `rirods` 0.2.0 (see the [documentation](http://rirods.irods4r.org)). iRODS C++ HTTP 0.2.0 is a requirement for this R package.

As of November 2023 [iRODS](https://irods.org) has released the [iRODS C++ HTTP API](https://irods.org/2023/11/initial-release-of-the-irods-http-api/), as a replacement of the old [iRODS C++ REST API](https://github.com/irods/irods_client_rest_cpp). This new interface to the iRODS server comes with a host of improvements including flexibility of management for all types of operations on users, collections and data objects, as well as supporting authentication with [OpenID Connect](https://openid.net/developers/how-connect-works/). The HTTP API is steadily maturing, so it was about time to leverage this functionality with `rirods`.

Besides a lot of potential for future rirods development in terms of iRODS operations, this new release (rirods 0.2.0) ensures some obvious improvements on the same core functionality as in previous releases of rirods. In this blog we will showcase one of those improvements—the speed of uploading and downloading of files to iRODS. The new HTTP API
can now be configured to allow receiving of much larger payloads. An example of how the HTTP API can be configured with a JSON file can be seen [here](https://github.com/irods/irods_demo/blob/main/irods_client_http_api/config.json). On top of that, write requests can now be made in parallel, which can as well be configured for the number of allowed threads.

In order to benchmark this improvement we show upload and download time as well as memory usage for `iget()` and `iput()`, respectively. We use the Docker based [iRODS demo](https://github.com/irods/irods_demo) as our server. We can easily stand this server up, as follows:

``` r
library(rirods)
use_irods_demo()
```


    Do the following to connect with the iRODS demo server: 
    create_irods("http://localhost:9001/irods-http-api/0.2.0") 
    iauth("rods", "rods")

For more information on `use_irods_demo()` see the `demo` vignette (`vignette("demo", package = "rirods")`) or this [page](https://rirods.irods4r.org/articles/demo.html). Now hat the demo server is running we can log in to the system by following the instructions as printed above.

``` r
create_irods("http://localhost:9001/irods-http-api/0.2.0") 
iauth("rods", "rods")
```

This server does obviously not contain any collections or data objects, as can be seen by using `ils()`.

``` r
ils()
```

    This collection does not contain any objects or collections.

This demo server has been configured to allow receiving request bodies of up to 8.4 Mb and we can use a maximum of 3 threads simultaneously during a write operation.

The files transferred to the irods demo server are created with the [GitHub API](https://docs.github.com/en/rest?apiVersion=2022-11-28) and the R package `gh`. The number of repositories and commits for the GitHub Groups; [git](https://github.com/git), [irods](https://github.com/irods), [nodejs](https://github.com/nodejs), and [tidyverse](https://github.com/tidyverse) have been downloaded (7<sup>th</sup> March 2024). The downloaded results are then saved as csv files. The files generated in this way range in size from 2 Kb to 10 Mb ([Figure 1](#fig-github)). The script and files can be found [here](https://github.com/FAIReLABS/iRODS4R/tree/main/posts/rirods-0-2-0).

![Figure 1: Number of repositories and commits on GitHub for the groups git, irods, nodejs, and tidyverse.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uaui3ov6mvpu6txkv63y.png)

These files are then transferred to the demo server with `iput()` and subsequently retrieved with `iget()`, while comparing rirods 0.1.2 and 0.2.0. This is repeated 100 times and performance was monitored with the R package `bench`. We can see that the newer version shows a drastic improvement for file transfers in both directions ([Figure 2](#fig-time)). This was to be expected as the older `rirods` package version needed to chop up the file before transferring it chunk-by-chunk to the server at relatively small file sizes (Kb range). As this all took place in R, this was necessarily slow.

![Figure 2: Median time of file transfer for iget() and iput() based on bench.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bggx5hdt9tujpjric2a7.png)

The same can be said for the memory allocated to up- and download files to iRODS, which improves in the newer version of `rirods` ([Figure 3](#fig-mem)). We see that above the threshold for file transfers configured server-side imposes a little more strain on the system above 8 Mb, as we also need to chop the file in pieces in these instances.

![Figure 3: Memory allocated for file transfer for iget() and iput() based on bench.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n5n150mm2nr3k228lxg7.png)

Increased write and read operations are an obvious improvement when using rirods 0.2.0 and the iRODS HTTP API has a lot of potential to add even more functionality to rirods in the future. Read more about the changes in some of the functions interfaces in the [changelog](https://rirods.irods4r.org/news/index.html). Contributions to the package are warmly welcomed. Please report issues and make pull requests in the [GitHub repository of rirods](https://github.com/irods/irods_client_library_rirods).

The following table list all of the benchmark results.

|                                              source | fun name |         file name | file size (Mb) | memory allocated (Mb) | median time (sec) |
|----------------------------------------------------:|---------:|------------------:|---------------:|----------------------:|------------------:|
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |         git_repos |           0.00 |                  0.13 |              0.11 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |   tidyverse_repos |           0.01 |                  1.20 |              0.31 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |       irods_repos |           0.05 |                  5.01 |              1.58 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |      nodejs_repos |           0.05 |                  5.88 |              1.67 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |      tidy_commits |           2.88 |                307.80 |             80.21 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |       git_commits |           4.92 |                526.89 |            131.62 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |     irods_commits |           6.52 |                695.48 |            180.09 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iget |    nodejs_commits |           9.69 |               1010.00 |            268.12 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |         git_repos |           0.00 |                  0.12 |              0.16 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |   tidyverse_repos |           0.01 |                  1.18 |              0.45 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |       irods_repos |           0.05 |                  5.00 |              1.96 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |      nodejs_repos |           0.05 |                  5.87 |              2.13 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |      tidy_commits |           2.88 |                308.06 |            107.14 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |       git_commits |           4.92 |                526.34 |            176.25 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |     irods_commits |           6.52 |                696.10 |            244.76 |
| https://cran.r-project.org/src/contrib/rirods_0.1.2 |     iput |    nodejs_commits |           9.69 |               1010.00 |            364.56 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |         git_repos |           0.00 |                  1.10 |              0.05 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |   tidyverse_repos |           0.01 |                  1.10 |              0.11 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |       irods_repos |           0.05 |                  1.10 |              0.12 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |      nodejs_repos |           0.05 |                  1.10 |              0.11 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget | tidyverse_commits |           2.88 |                  1.10 |              0.17 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |       git_commits |           4.92 |                  1.12 |              0.24 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |     irods_commits |           6.52 |                  1.10 |              0.29 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iget |    nodejs_commits |           9.69 |                  1.10 |              0.42 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |         git_repos |           0.00 |                  2.17 |              0.09 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |   tidyverse_repos |           0.01 |                  2.18 |              0.10 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |       irods_repos |           0.05 |                  2.21 |              0.11 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |      nodejs_repos |           0.05 |                  2.22 |              0.09 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput | tidyverse_commits |           2.88 |                  4.92 |              0.16 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |       git_commits |           4.92 |                  7.22 |              0.17 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |     irods_commits |           6.52 |                  8.38 |              0.23 |
| https://cran.r-project.org/src/contrib/rirods_0.2.0 |     iput |    nodejs_commits |           9.69 |                752.86 |              2.92 |

Benchmarks results for `iget` and `iput`.