Add `encoding` and `errors` params to JSON loader
Opened by mariosasko (closed)

"Requested" in https://discuss.huggingface.co/t/utf-16-for-datasets/43828/3.

`pd.read_json` also has these parameters, so it makes sense to be consistent.

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006770 / 0.011353 (-0.004583) | 0.004143 / 0.011008 (-0.006865) | 0.098928 / 0.038508 (0.060420) | 0.044893 / 0.023109 (0.021783) | 0.302630 / 0.275898 (0.026732) | 0.368173 / 0.323480 (0.044693) | 0.005631 / 0.007986 (-0.002354) | 0.003397 / 0.004328 (-0.000931) | 0.075748 / 0.004250 (0.071497) | 0.062582 / 0.037052 (0.025530) | 0.329586 / 0.258489 (0.071097) | 0.362625 / 0.293841 (0.068784) | 0.033250 / 0.128546 (-0.095296) | 0.008880 / 0.075646 (-0.066766) | 0.329683 / 0.419271 (-0.089588) | 0.054426 / 0.043533 (0.010893) | 0.297940 / 0.255139 (0.042801) | 0.319796 / 0.283200 (0.036597) | 0.023296 / 0.141683 (-0.118387) | 1.462142 / 1.452155 (0.009987) | 1.495796 / 1.492716 (0.003079) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201771 / 0.018006 (0.183765) | 0.454514 / 0.000490 (0.454024) | 0.003333 / 0.000200 (0.003133) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028084 / 0.037411 (-0.009327) | 0.109452 / 0.014526 (0.094926) | 0.119200 / 0.176557 (-0.057357) | 0.180302 / 0.737135 (-0.556834) | 0.125653 / 0.296338 (-0.170686) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409819 / 0.215209 (0.194610) | 4.055117 / 2.077655 (1.977462) | 1.855279 / 1.504120 (0.351159) | 1.655281 / 1.541195 (0.114086) | 1.687938 / 1.468490 (0.219448) | 0.528352 / 4.584777 (-4.056425) | 3.750250 / 3.745712 (0.004538) | 3.386741 / 5.269862 (-1.883121) | 1.572036 / 4.565676 (-2.993640) | 0.065125 / 0.424275 (-0.359150) | 0.011259 / 0.007607 (0.003652) | 0.513449 / 0.226044 (0.287405) | 5.139421 / 2.268929 (2.870492) | 2.316973 / 55.444624 (-53.127651) | 1.984109 / 6.876477 (-4.892368) | 2.127915 / 2.142072 (-0.014158) | 0.653238 / 4.805227 (-4.151989) | 0.142686 / 6.500664 (-6.357978) | 0.063666 / 0.075469 (-0.011803) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.185174 / 1.841788 (-0.656614) | 14.790282 / 8.074308 (6.715974) | 13.089222 / 10.191392 (2.897830) | 0.146055 / 0.680424 (-0.534369) | 0.017835 / 0.534201 (-0.516366) | 0.399598 / 0.579283 (-0.179685) | 0.425296 / 0.434364 (-0.009068) | 0.478552 / 0.540337 (-0.061786) | 0.579702 / 1.386936 (-0.807234) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006750 / 0.011353 (-0.004603) | 0.004156 / 0.011008 (-0.006853) | 0.074948 / 0.038508 (0.036440) | 0.043368 / 0.023109 (0.020259) | 0.355389 / 0.275898 (0.079491) | 0.429167 / 0.323480 (0.105687) | 0.003911 / 0.007986 (-0.004075) | 0.004340 / 0.004328 (0.000012) | 0.075940 / 0.004250 (0.071689) | 0.054293 / 0.037052 (0.017241) | 0.400317 / 0.258489 (0.141827) | 0.432001 / 0.293841 (0.138160) | 0.032340 / 0.128546 (-0.096206) | 0.008876 / 0.075646 (-0.066770) | 0.082284 / 0.419271 (-0.336987) | 0.050819 / 0.043533 (0.007286) | 0.351994 / 0.255139 (0.096855) | 0.375917 / 0.283200 (0.092717) | 0.022466 / 0.141683 (-0.119217) | 1.538824 / 1.452155 (0.086669) | 1.563995 / 1.492716 (0.071279) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227330 / 0.018006 (0.209323) | 0.446380 / 0.000490 (0.445890) | 0.000408 / 0.000200 (0.000208) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028534 / 0.037411 (-0.008878) | 0.113467 / 0.014526 (0.098941) | 0.123590 / 0.176557 (-0.052966) | 0.174309 / 0.737135 (-0.562827) | 0.130631 / 0.296338 (-0.165707) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441020 / 0.215209 (0.225811) | 4.386564 / 2.077655 (2.308909) | 2.100704 / 1.504120 (0.596584) | 1.901484 / 1.541195 (0.360289) | 1.963494 / 1.468490 (0.495004) | 0.536838 / 4.584777 (-4.047939) | 3.739071 / 3.745712 (-0.006642) | 3.278981 / 5.269862 (-1.990881) | 1.515476 / 4.565676 (-3.050201) | 0.066388 / 0.424275 (-0.357887) | 0.011857 / 0.007607 (0.004250) | 0.545507 / 0.226044 (0.319463) | 5.441479 / 2.268929 (3.172550) | 2.602144 / 55.444624 (-52.842480) | 2.235583 / 6.876477 (-4.640894) | 2.293458 / 2.142072 (0.151385) | 0.658535 / 4.805227 (-4.146692) | 0.141327 / 6.500664 (-6.359337) | 0.063726 / 0.075469 (-0.011743) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.247819 / 1.841788 (-0.593968) | 15.234524 / 8.074308 (7.160216) | 14.592700 / 10.191392 (4.401308) | 0.141952 / 0.680424 (-0.538472) | 0.017747 / 0.534201 (-0.516454) | 0.396819 / 0.579283 (-0.182465) | 0.415902 / 0.434364 (-0.018462) | 0.464619 / 0.540337 (-0.075718) | 0.560866 / 1.386936 (-0.826070) |

</details>
</details>

![](https://cml.dev/watermark.png#4b7f6c59deb868e21f295917548fa2df10dd0158 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008278 / 0.011353 (-0.003075) | 0.005044 / 0.011008 (-0.005964) | 0.123382 / 0.038508 (0.084874) | 0.054039 / 0.023109 (0.030929) | 0.382338 / 0.275898 (0.106440) | 0.453287 / 0.323480 (0.129807) | 0.006342 / 0.007986 (-0.001644) | 0.003930 / 0.004328 (-0.000398) | 0.094039 / 0.004250 (0.089789) | 0.076525 / 0.037052 (0.039472) | 0.394066 / 0.258489 (0.135577) | 0.445600 / 0.293841 (0.151759) | 0.039348 / 0.128546 (-0.089199) | 0.010485 / 0.075646 (-0.065161) | 0.433730 / 0.419271 (0.014459) | 0.082671 / 0.043533 (0.039138) | 0.375250 / 0.255139 (0.120111) | 0.416269 / 0.283200 (0.133070) | 0.038397 / 0.141683 (-0.103286) | 1.864834 / 1.452155 (0.412680) | 2.010453 / 1.492716 (0.517737) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240008 / 0.018006 (0.222002) | 0.470975 / 0.000490 (0.470485) | 0.004001 / 0.000200 (0.003801) | 0.000097 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031107 / 0.037411 (-0.006304) | 0.129371 / 0.014526 (0.114846) | 0.141559 / 0.176557 (-0.034997) | 0.205571 / 0.737135 (-0.531564) | 0.144611 / 0.296338 (-0.151728) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.506972 / 0.215209 (0.291763) | 5.055951 / 2.077655 (2.978296) | 2.397438 / 1.504120 (0.893318) | 2.170435 / 1.541195 (0.629240) | 2.240296 / 1.468490 (0.771806) | 0.641559 / 4.584777 (-3.943218) | 4.644772 / 3.745712 (0.899060) | 4.064200 / 5.269862 (-1.205662) | 1.946991 / 4.565676 (-2.618685) | 0.086413 / 0.424275 (-0.337862) | 0.015082 / 0.007607 (0.007475) | 0.670413 / 0.226044 (0.444369) | 6.331346 / 2.268929 (4.062418) | 2.965813 / 55.444624 (-52.478812) | 2.547952 / 6.876477 (-4.328524) | 2.718390 / 2.142072 (0.576318) | 0.796657 / 4.805227 (-4.008571) | 0.173229 / 6.500664 (-6.327435) | 0.079606 / 0.075469 (0.004137) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.568761 / 1.841788 (-0.273026) | 18.485432 / 8.074308 (10.411124) | 15.758513 / 10.191392 (5.567121) | 0.170427 / 0.680424 (-0.509997) | 0.021421 / 0.534201 (-0.512780) | 0.518623 / 0.579283 (-0.060660) | 0.525887 / 0.434364 (0.091523) | 0.640331 / 0.540337 (0.099993) | 0.766748 / 1.386936 (-0.620188) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007680 / 0.011353 (-0.003673) | 0.005289 / 0.011008 (-0.005719) | 0.093773 / 0.038508 (0.055265) | 0.054997 / 0.023109 (0.031888) | 0.456277 / 0.275898 (0.180379) | 0.500642 / 0.323480 (0.177162) | 0.005935 / 0.007986 (-0.002050) | 0.004375 / 0.004328 (0.000047) | 0.094131 / 0.004250 (0.089881) | 0.063399 / 0.037052 (0.026347) | 0.470546 / 0.258489 (0.212057) | 0.504989 / 0.293841 (0.211148) | 0.038541 / 0.128546 (-0.090006) | 0.010403 / 0.075646 (-0.065244) | 0.102469 / 0.419271 (-0.316802) | 0.063105 / 0.043533 (0.019572) | 0.466005 / 0.255139 (0.210866) | 0.458677 / 0.283200 (0.175477) | 0.028407 / 0.141683 (-0.113276) | 1.893829 / 1.452155 (0.441675) | 1.917954 / 1.492716 (0.425238) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.272760 / 0.018006 (0.254754) | 0.476159 / 0.000490 (0.475669) | 0.008467 / 0.000200 (0.008267) | 0.000146 / 0.000054 (0.000091) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035755 / 0.037411 (-0.001656) | 0.145038 / 0.014526 (0.130512) | 0.148322 / 0.176557 (-0.028235) | 0.210193 / 0.737135 (-0.526943) | 0.156547 / 0.296338 (-0.139792) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.541204 / 0.215209 (0.325995) | 5.382746 / 2.077655 (3.305091) | 2.704229 / 1.504120 (1.200109) | 2.468422 / 1.541195 (0.927227) | 2.522672 / 1.468490 (1.054182) | 0.644899 / 4.584777 (-3.939878) | 4.654401 / 3.745712 (0.908689) | 2.159223 / 5.269862 (-3.110638) | 1.280098 / 4.565676 (-3.285578) | 0.080053 / 0.424275 (-0.344222) | 0.014383 / 0.007607 (0.006776) | 0.662770 / 0.226044 (0.436725) | 6.617651 / 2.268929 (4.348722) | 3.234347 / 55.444624 (-52.210277) | 2.861417 / 6.876477 (-4.015059) | 2.888928 / 2.142072 (0.746856) | 0.792854 / 4.805227 (-4.012374) | 0.172553 / 6.500664 (-6.328111) | 0.078402 / 0.075469 (0.002933) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.565351 / 1.841788 (-0.276436) | 18.681916 / 8.074308 (10.607608) | 17.264473 / 10.191392 (7.073081) | 0.168461 / 0.680424 (-0.511963) | 0.021353 / 0.534201 (-0.512848) | 0.517843 / 0.579283 (-0.061440) | 0.519907 / 0.434364 (0.085543) | 0.623687 / 0.540337 (0.083350) | 0.761796 / 1.386936 (-0.625140) |

</details>
</details>

![](https://cml.dev/watermark.png#bbf58747f734a46e75937bdbcbc05b06ade0224a "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006750 / 0.011353 (-0.004603) | 0.004268 / 0.011008 (-0.006741) | 0.098644 / 0.038508 (0.060136) | 0.044643 / 0.023109 (0.021534) | 0.309420 / 0.275898 (0.033522) | 0.379294 / 0.323480 (0.055815) | 0.005729 / 0.007986 (-0.002256) | 0.003615 / 0.004328 (-0.000714) | 0.076086 / 0.004250 (0.071835) | 0.068994 / 0.037052 (0.031942) | 0.325653 / 0.258489 (0.067164) | 0.375187 / 0.293841 (0.081347) | 0.032546 / 0.128546 (-0.096000) | 0.009089 / 0.075646 (-0.066557) | 0.329905 / 0.419271 (-0.089366) | 0.066832 / 0.043533 (0.023300) | 0.299247 / 0.255139 (0.044108) | 0.323460 / 0.283200 (0.040260) | 0.034226 / 0.141683 (-0.107457) | 1.475659 / 1.452155 (0.023505) | 1.556234 / 1.492716 (0.063518) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292305 / 0.018006 (0.274299) | 0.542584 / 0.000490 (0.542094) | 0.003047 / 0.000200 (0.002847) | 0.000082 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030096 / 0.037411 (-0.007315) | 0.112341 / 0.014526 (0.097815) | 0.124965 / 0.176557 (-0.051591) | 0.183159 / 0.737135 (-0.553976) | 0.131885 / 0.296338 (-0.164453) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426437 / 0.215209 (0.211228) | 4.260984 / 2.077655 (2.183330) | 2.078358 / 1.504120 (0.574238) | 1.877644 / 1.541195 (0.336449) | 2.044036 / 1.468490 (0.575546) | 0.532980 / 4.584777 (-4.051797) | 3.749573 / 3.745712 (0.003860) | 1.944155 / 5.269862 (-3.325706) | 1.090307 / 4.565676 (-3.475370) | 0.065445 / 0.424275 (-0.358830) | 0.011237 / 0.007607 (0.003630) | 0.521448 / 0.226044 (0.295403) | 5.213118 / 2.268929 (2.944189) | 2.507829 / 55.444624 (-52.936795) | 2.177179 / 6.876477 (-4.699297) | 2.351161 / 2.142072 (0.209088) | 0.656775 / 4.805227 (-4.148452) | 0.141207 / 6.500664 (-6.359457) | 0.063286 / 0.075469 (-0.012183) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.190281 / 1.841788 (-0.651506) | 15.327424 / 8.074308 (7.253116) | 13.300695 / 10.191392 (3.109303) | 0.190484 / 0.680424 (-0.489939) | 0.017984 / 0.534201 (-0.516217) | 0.405714 / 0.579283 (-0.173569) | 0.435915 / 0.434364 (0.001551) | 0.494083 / 0.540337 (-0.046254) | 0.600616 / 1.386936 (-0.786320) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006740 / 0.011353 (-0.004613) | 0.004289 / 0.011008 (-0.006719) | 0.076532 / 0.038508 (0.038024) | 0.043305 / 0.023109 (0.020196) | 0.356111 / 0.275898 (0.080213) | 0.434121 / 0.323480 (0.110641) | 0.005599 / 0.007986 (-0.002387) | 0.003461 / 0.004328 (-0.000868) | 0.077097 / 0.004250 (0.072847) | 0.055369 / 0.037052 (0.018317) | 0.367093 / 0.258489 (0.108604) | 0.418801 / 0.293841 (0.124960) | 0.032057 / 0.128546 (-0.096489) | 0.009048 / 0.075646 (-0.066599) | 0.082897 / 0.419271 (-0.336374) | 0.050287 / 0.043533 (0.006754) | 0.352060 / 0.255139 (0.096921) | 0.376278 / 0.283200 (0.093078) | 0.023924 / 0.141683 (-0.117759) | 1.522780 / 1.452155 (0.070626) | 1.578938 / 1.492716 (0.086222) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.287317 / 0.018006 (0.269311) | 0.508490 / 0.000490 (0.508000) | 0.000431 / 0.000200 (0.000231) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031139 / 0.037411 (-0.006272) | 0.113927 / 0.014526 (0.099401) | 0.128147 / 0.176557 (-0.048409) | 0.179712 / 0.737135 (-0.557424) | 0.134364 / 0.296338 (-0.161975) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.452834 / 0.215209 (0.237625) | 4.507944 / 2.077655 (2.430289) | 2.287758 / 1.504120 (0.783638) | 2.091145 / 1.541195 (0.549951) | 2.196228 / 1.468490 (0.727738) | 0.539306 / 4.584777 (-4.045471) | 3.838941 / 3.745712 (0.093228) | 1.908801 / 5.269862 (-3.361060) | 1.139235 / 4.565676 (-3.426442) | 0.066677 / 0.424275 (-0.357599) | 0.011422 / 0.007607 (0.003815) | 0.562966 / 0.226044 (0.336921) | 5.633712 / 2.268929 (3.364784) | 2.788622 / 55.444624 (-52.656002) | 2.438465 / 6.876477 (-4.438012) | 2.523479 / 2.142072 (0.381407) | 0.668730 / 4.805227 (-4.136498) | 0.143977 / 6.500664 (-6.356687) | 0.064661 / 0.075469 (-0.010808) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291708 / 1.841788 (-0.550080) | 15.573316 / 8.074308 (7.499008) | 14.435099 / 10.191392 (4.243707) | 0.147745 / 0.680424 (-0.532679) | 0.017602 / 0.534201 (-0.516599) | 0.401560 / 0.579283 (-0.177723) | 0.429861 / 0.434364 (-0.004502) | 0.469800 / 0.540337 (-0.070538) | 0.567515 / 1.386936 (-0.819421) |

</details>
</details>

![](https://cml.dev/watermark.png#79c340f5dcfd06340f180f6c6ea2d5ef81f49d98 "CML watermark")