Title: Use new hffs
Author: lhoestq (MEMBER)
State: closed
Created: 2023-07-13T15:41:44Z

Thanks to @janineguo 's work in https://github.com/huggingface/datasets/pull/5919 which was needed to support HfFileSystem.

Switching to `HfFileSystem` will help implementing optimization in data files resolution

## Implementation details

I replaced all the from_hf_repo and from_local_or_remote in data_files.py to only use a new `from_patterns` which works for any fsspec path, including hf:// paths, https:// URLs and local paths. This simplifies the codebase since there is no logic duplication anymore when it comes to data files resolution.

I added `_prepare_path_and_storage_options` which returns the right storage_options to use given a path and a `DownloadConfig`. This is the only place where the logic depends on the filesystem type that must be used.

I also removed the `get_metadata_data_files_list ` and `get_patterns_and_data_files` functions added recently, since data files resolution is now handled using a common interface.

## New features

hf:// paths are now supported in data_files

## Breaking changes

DataFilesList and DataFilesDict:
- use `str` paths instead of `Union[Path, Url]`
- require posix paths for windows paths

close https://github.com/huggingface/datasets/issues/6017

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006665 / 0.011353 (-0.004688) | 0.004376 / 0.011008 (-0.006633) | 0.085529 / 0.038508 (0.047021) | 0.076372 / 0.023109 (0.053263) | 0.310019 / 0.275898 (0.034121) | 0.341404 / 0.323480 (0.017924) | 0.005666 / 0.007986 (-0.002320) | 0.003763 / 0.004328 (-0.000566) | 0.064678 / 0.004250 (0.060427) | 0.059283 / 0.037052 (0.022231) | 0.316194 / 0.258489 (0.057704) | 0.349397 / 0.293841 (0.055557) | 0.031199 / 0.128546 (-0.097347) | 0.008724 / 0.075646 (-0.066923) | 0.300236 / 0.419271 (-0.119035) | 0.068872 / 0.043533 (0.025339) | 0.308521 / 0.255139 (0.053382) | 0.331292 / 0.283200 (0.048092) | 0.028236 / 0.141683 (-0.113447) | 1.501365 / 1.452155 (0.049211) | 1.554334 / 1.492716 (0.061618) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238291 / 0.018006 (0.220285) | 0.565069 / 0.000490 (0.564580) | 0.001626 / 0.000200 (0.001426) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029777 / 0.037411 (-0.007634) | 0.082873 / 0.014526 (0.068347) | 0.099619 / 0.176557 (-0.076937) | 0.156572 / 0.737135 (-0.580563) | 0.099887 / 0.296338 (-0.196452) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401017 / 0.215209 (0.185808) | 3.827192 / 2.077655 (1.749537) | 1.861554 / 1.504120 (0.357434) | 1.699869 / 1.541195 (0.158674) | 1.720043 / 1.468490 (0.251553) | 0.486757 / 4.584777 (-4.098020) | 3.638125 / 3.745712 (-0.107587) | 5.844959 / 5.269862 (0.575097) | 3.454901 / 4.565676 (-1.110775) | 0.057650 / 0.424275 (-0.366625) | 0.007341 / 0.007607 (-0.000266) | 0.462698 / 0.226044 (0.236654) | 4.633472 / 2.268929 (2.364544) | 2.287607 / 55.444624 (-53.157017) | 2.057318 / 6.876477 (-4.819159) | 2.203657 / 2.142072 (0.061584) | 0.598136 / 4.805227 (-4.207091) | 0.134012 / 6.500664 (-6.366653) | 0.060824 / 0.075469 (-0.014645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.277752 / 1.841788 (-0.564036) | 20.013398 / 8.074308 (11.939089) | 14.372993 / 10.191392 (4.181601) | 0.169991 / 0.680424 (-0.510433) | 0.018344 / 0.534201 (-0.515857) | 0.396985 / 0.579283 (-0.182299) | 0.416289 / 0.434364 (-0.018075) | 0.458658 / 0.540337 (-0.081680) | 0.692980 / 1.386936 (-0.693956) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006689 / 0.011353 (-0.004664) | 0.004393 / 0.011008 (-0.006615) | 0.064069 / 0.038508 (0.025561) | 0.080717 / 0.023109 (0.057607) | 0.370090 / 0.275898 (0.094191) | 0.400432 / 0.323480 (0.076952) | 0.005613 / 0.007986 (-0.002372) | 0.003641 / 0.004328 (-0.000687) | 0.064771 / 0.004250 (0.060520) | 0.057555 / 0.037052 (0.020502) | 0.392156 / 0.258489 (0.133667) | 0.409842 / 0.293841 (0.116001) | 0.031500 / 0.128546 (-0.097047) | 0.008786 / 0.075646 (-0.066860) | 0.070342 / 0.419271 (-0.348929) | 0.048646 / 0.043533 (0.005113) | 0.360914 / 0.255139 (0.105775) | 0.387626 / 0.283200 (0.104426) | 0.022787 / 0.141683 (-0.118896) | 1.508915 / 1.452155 (0.056761) | 1.539719 / 1.492716 (0.047002) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.257985 / 0.018006 (0.239979) | 0.550990 / 0.000490 (0.550501) | 0.000407 / 0.000200 (0.000207) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030183 / 0.037411 (-0.007228) | 0.086882 / 0.014526 (0.072356) | 0.102382 / 0.176557 (-0.074175) | 0.154745 / 0.737135 (-0.582390) | 0.104008 / 0.296338 (-0.192331) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426284 / 0.215209 (0.211075) | 4.240812 / 2.077655 (2.163158) | 2.261240 / 1.504120 (0.757120) | 2.085905 / 1.541195 (0.544710) | 2.160374 / 1.468490 (0.691883) | 0.481126 / 4.584777 (-4.103651) | 3.516234 / 3.745712 (-0.229478) | 3.325322 / 5.269862 (-1.944539) | 2.043307 / 4.565676 (-2.522369) | 0.056663 / 0.424275 (-0.367612) | 0.007786 / 0.007607 (0.000179) | 0.497614 / 0.226044 (0.271570) | 4.974529 / 2.268929 (2.705600) | 2.700018 / 55.444624 (-52.744606) | 2.393778 / 6.876477 (-4.482699) | 2.628202 / 2.142072 (0.486130) | 0.594316 / 4.805227 (-4.210911) | 0.147092 / 6.500664 (-6.353572) | 0.062207 / 0.075469 (-0.013262) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.315676 / 1.841788 (-0.526112) | 20.749251 / 8.074308 (12.674943) | 14.371553 / 10.191392 (4.180160) | 0.170249 / 0.680424 (-0.510175) | 0.018478 / 0.534201 (-0.515722) | 0.395710 / 0.579283 (-0.183573) | 0.409706 / 0.434364 (-0.024658) | 0.463454 / 0.540337 (-0.076884) | 0.615657 / 1.386936 (-0.771279) |

</details>
</details>

![](https://cml.dev/watermark.png#c5a752d8e8ca0a6ed118b024ba03c1b4a2881177 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007224 / 0.011353 (-0.004129) | 0.004506 / 0.011008 (-0.006503) | 0.096729 / 0.038508 (0.058221) | 0.082394 / 0.023109 (0.059284) | 0.390954 / 0.275898 (0.115056) | 0.416647 / 0.323480 (0.093167) | 0.005894 / 0.007986 (-0.002092) | 0.003756 / 0.004328 (-0.000572) | 0.075800 / 0.004250 (0.071549) | 0.062683 / 0.037052 (0.025631) | 0.398959 / 0.258489 (0.140470) | 0.436624 / 0.293841 (0.142783) | 0.034650 / 0.128546 (-0.093896) | 0.009655 / 0.075646 (-0.065991) | 0.315761 / 0.419271 (-0.103511) | 0.060957 / 0.043533 (0.017424) | 0.385649 / 0.255139 (0.130510) | 0.394022 / 0.283200 (0.110822) | 0.024601 / 0.141683 (-0.117082) | 1.729586 / 1.452155 (0.277431) | 1.724153 / 1.492716 (0.231437) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207070 / 0.018006 (0.189063) | 0.466502 / 0.000490 (0.466012) | 0.010739 / 0.000200 (0.010540) | 0.000214 / 0.000054 (0.000160) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031633 / 0.037411 (-0.005779) | 0.095345 / 0.014526 (0.080819) | 0.105399 / 0.176557 (-0.071157) | 0.174173 / 0.737135 (-0.562962) | 0.104207 / 0.296338 (-0.192132) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435312 / 0.215209 (0.220103) | 4.265600 / 2.077655 (2.187946) | 2.056500 / 1.504120 (0.552380) | 1.848023 / 1.541195 (0.306828) | 1.946156 / 1.468490 (0.477666) | 0.557788 / 4.584777 (-4.026989) | 4.070289 / 3.745712 (0.324577) | 3.608027 / 5.269862 (-1.661835) | 2.214556 / 4.565676 (-2.351121) | 0.062623 / 0.424275 (-0.361652) | 0.008083 / 0.007607 (0.000476) | 0.491782 / 0.226044 (0.265738) | 4.989963 / 2.268929 (2.721035) | 2.575867 / 55.444624 (-52.868757) | 2.208045 / 6.876477 (-4.668431) | 2.364184 / 2.142072 (0.222112) | 0.633925 / 4.805227 (-4.171302) | 0.144323 / 6.500664 (-6.356341) | 0.067505 / 0.075469 (-0.007965) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.467219 / 1.841788 (-0.374569) | 22.334967 / 8.074308 (14.260659) | 15.715747 / 10.191392 (5.524355) | 0.175443 / 0.680424 (-0.504980) | 0.026165 / 0.534201 (-0.508036) | 0.490675 / 0.579283 (-0.088608) | 0.509211 / 0.434364 (0.074847) | 0.586303 / 0.540337 (0.045965) | 0.785052 / 1.386936 (-0.601884) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007893 / 0.011353 (-0.003460) | 0.004577 / 0.011008 (-0.006431) | 0.075781 / 0.038508 (0.037273) | 0.095492 / 0.023109 (0.072382) | 0.433259 / 0.275898 (0.157361) | 0.469386 / 0.323480 (0.145906) | 0.006317 / 0.007986 (-0.001669) | 0.003708 / 0.004328 (-0.000621) | 0.074417 / 0.004250 (0.070167) | 0.068605 / 0.037052 (0.031552) | 0.448701 / 0.258489 (0.190212) | 0.469131 / 0.293841 (0.175290) | 0.036647 / 0.128546 (-0.091899) | 0.010077 / 0.075646 (-0.065570) | 0.082457 / 0.419271 (-0.336815) | 0.063255 / 0.043533 (0.019722) | 0.428144 / 0.255139 (0.173005) | 0.451872 / 0.283200 (0.168672) | 0.033953 / 0.141683 (-0.107730) | 1.781752 / 1.452155 (0.329597) | 1.869014 / 1.492716 (0.376297) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223596 / 0.018006 (0.205590) | 0.470307 / 0.000490 (0.469818) | 0.005059 / 0.000200 (0.004859) | 0.000104 / 0.000054 (0.000049) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038804 / 0.037411 (0.001393) | 0.117879 / 0.014526 (0.103353) | 0.140701 / 0.176557 (-0.035855) | 0.194672 / 0.737135 (-0.542463) | 0.132806 / 0.296338 (-0.163533) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.510109 / 0.215209 (0.294900) | 4.729457 / 2.077655 (2.651803) | 2.512113 / 1.504120 (1.007993) | 2.302553 / 1.541195 (0.761358) | 2.420462 / 1.468490 (0.951972) | 0.531682 / 4.584777 (-4.053095) | 4.061208 / 3.745712 (0.315496) | 3.588542 / 5.269862 (-1.681320) | 2.203187 / 4.565676 (-2.362489) | 0.065791 / 0.424275 (-0.358484) | 0.008839 / 0.007607 (0.001232) | 0.562041 / 0.226044 (0.335997) | 5.702340 / 2.268929 (3.433412) | 3.127609 / 55.444624 (-52.317015) | 2.823060 / 6.876477 (-4.053417) | 2.898675 / 2.142072 (0.756603) | 0.659589 / 4.805227 (-4.145638) | 0.148798 / 6.500664 (-6.351866) | 0.070787 / 0.075469 (-0.004682) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.478317 / 1.841788 (-0.363471) | 21.995400 / 8.074308 (13.921092) | 16.770729 / 10.191392 (6.579337) | 0.226333 / 0.680424 (-0.454091) | 0.021835 / 0.534201 (-0.512366) | 0.460373 / 0.579283 (-0.118910) | 0.479494 / 0.434364 (0.045130) | 0.529470 / 0.540337 (-0.010868) | 0.718066 / 1.386936 (-0.668870) |

</details>
</details>

![](https://cml.dev/watermark.png#9a717b8eb80b0e50b25818127f79a35e0866fb14 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007824 / 0.011353 (-0.003529) | 0.004601 / 0.011008 (-0.006407) | 0.100025 / 0.038508 (0.061517) | 0.096046 / 0.023109 (0.072936) | 0.376226 / 0.275898 (0.100328) | 0.410905 / 0.323480 (0.087425) | 0.006048 / 0.007986 (-0.001938) | 0.003817 / 0.004328 (-0.000511) | 0.076624 / 0.004250 (0.072374) | 0.066390 / 0.037052 (0.029338) | 0.380098 / 0.258489 (0.121609) | 0.413603 / 0.293841 (0.119762) | 0.036546 / 0.128546 (-0.092001) | 0.009881 / 0.075646 (-0.065765) | 0.344338 / 0.419271 (-0.074934) | 0.061882 / 0.043533 (0.018350) | 0.368568 / 0.255139 (0.113429) | 0.397133 / 0.283200 (0.113934) | 0.027255 / 0.141683 (-0.114428) | 1.795099 / 1.452155 (0.342945) | 1.852443 / 1.492716 (0.359727) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.247436 / 0.018006 (0.229430) | 0.494119 / 0.000490 (0.493629) | 0.004359 / 0.000200 (0.004159) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034765 / 0.037411 (-0.002647) | 0.104541 / 0.014526 (0.090015) | 0.113898 / 0.176557 (-0.062659) | 0.183634 / 0.737135 (-0.553501) | 0.116423 / 0.296338 (-0.179916) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458747 / 0.215209 (0.243538) | 4.555740 / 2.077655 (2.478085) | 2.217240 / 1.504120 (0.713121) | 2.039879 / 1.541195 (0.498684) | 2.088581 / 1.468490 (0.620091) | 0.588063 / 4.584777 (-3.996714) | 4.238226 / 3.745712 (0.492514) | 4.768060 / 5.269862 (-0.501802) | 2.857117 / 4.565676 (-1.708560) | 0.068742 / 0.424275 (-0.355533) | 0.008667 / 0.007607 (0.001059) | 0.549294 / 0.226044 (0.323249) | 5.464635 / 2.268929 (3.195706) | 2.744435 / 55.444624 (-52.700189) | 2.347660 / 6.876477 (-4.528816) | 2.616816 / 2.142072 (0.474743) | 0.703701 / 4.805227 (-4.101526) | 0.159749 / 6.500664 (-6.340915) | 0.071990 / 0.075469 (-0.003479) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.486599 / 1.841788 (-0.355188) | 22.745438 / 8.074308 (14.671130) | 16.822332 / 10.191392 (6.630940) | 0.184730 / 0.680424 (-0.495694) | 0.021267 / 0.534201 (-0.512934) | 0.467108 / 0.579283 (-0.112176) | 0.472674 / 0.434364 (0.038311) | 0.548094 / 0.540337 (0.007756) | 0.735885 / 1.386936 (-0.651051) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007746 / 0.011353 (-0.003607) | 0.004585 / 0.011008 (-0.006423) | 0.076943 / 0.038508 (0.038435) | 0.087473 / 0.023109 (0.064363) | 0.480099 / 0.275898 (0.204201) | 0.495271 / 0.323480 (0.171791) | 0.006348 / 0.007986 (-0.001638) | 0.003902 / 0.004328 (-0.000426) | 0.077586 / 0.004250 (0.073335) | 0.066467 / 0.037052 (0.029415) | 0.468741 / 0.258489 (0.210252) | 0.506778 / 0.293841 (0.212937) | 0.036877 / 0.128546 (-0.091669) | 0.010102 / 0.075646 (-0.065545) | 0.084419 / 0.419271 (-0.334852) | 0.058721 / 0.043533 (0.015188) | 0.453633 / 0.255139 (0.198494) | 0.481171 / 0.283200 (0.197971) | 0.028716 / 0.141683 (-0.112967) | 1.853048 / 1.452155 (0.400893) | 1.885847 / 1.492716 (0.393130) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192136 / 0.018006 (0.174130) | 0.484481 / 0.000490 (0.483991) | 0.002951 / 0.000200 (0.002751) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037949 / 0.037411 (0.000538) | 0.108364 / 0.014526 (0.093838) | 0.119542 / 0.176557 (-0.057014) | 0.188542 / 0.737135 (-0.548593) | 0.122011 / 0.296338 (-0.174327) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.483135 / 0.215209 (0.267926) | 4.849715 / 2.077655 (2.772060) | 2.497736 / 1.504120 (0.993616) | 2.314243 / 1.541195 (0.773048) | 2.412739 / 1.468490 (0.944249) | 0.564137 / 4.584777 (-4.020639) | 4.242273 / 3.745712 (0.496561) | 6.337843 / 5.269862 (1.067982) | 3.923250 / 4.565676 (-0.642426) | 0.066464 / 0.424275 (-0.357811) | 0.009217 / 0.007607 (0.001610) | 0.575667 / 0.226044 (0.349623) | 5.746187 / 2.268929 (3.477258) | 3.069655 / 55.444624 (-52.374969) | 2.674798 / 6.876477 (-4.201679) | 2.956535 / 2.142072 (0.814463) | 0.701043 / 4.805227 (-4.104185) | 0.157241 / 6.500664 (-6.343423) | 0.073175 / 0.075469 (-0.002294) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.609943 / 1.841788 (-0.231844) | 23.478594 / 8.074308 (15.404286) | 17.454437 / 10.191392 (7.263045) | 0.186422 / 0.680424 (-0.494002) | 0.021703 / 0.534201 (-0.512498) | 0.471704 / 0.579283 (-0.107579) | 0.480553 / 0.434364 (0.046189) | 0.552881 / 0.540337 (0.012544) | 0.722515 / 1.386936 (-0.664421) |

</details>
</details>

![](https://cml.dev/watermark.png#84645f80049cd00d9e0d4908faf3c3203fdcf21d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007542 / 0.011353 (-0.003811) | 0.004692 / 0.011008 (-0.006316) | 0.099155 / 0.038508 (0.060647) | 0.089365 / 0.023109 (0.066256) | 0.370870 / 0.275898 (0.094972) | 0.422152 / 0.323480 (0.098673) | 0.006223 / 0.007986 (-0.001763) | 0.003852 / 0.004328 (-0.000476) | 0.075438 / 0.004250 (0.071188) | 0.065973 / 0.037052 (0.028921) | 0.381513 / 0.258489 (0.123024) | 0.416196 / 0.293841 (0.122355) | 0.035483 / 0.128546 (-0.093063) | 0.009884 / 0.075646 (-0.065762) | 0.341290 / 0.419271 (-0.077982) | 0.060546 / 0.043533 (0.017014) | 0.365101 / 0.255139 (0.109962) | 0.391058 / 0.283200 (0.107859) | 0.026325 / 0.141683 (-0.115358) | 1.815168 / 1.452155 (0.363013) | 1.834711 / 1.492716 (0.341994) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222177 / 0.018006 (0.204171) | 0.501151 / 0.000490 (0.500662) | 0.010202 / 0.000200 (0.010002) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034043 / 0.037411 (-0.003368) | 0.097884 / 0.014526 (0.083358) | 0.114022 / 0.176557 (-0.062534) | 0.186200 / 0.737135 (-0.550935) | 0.115555 / 0.296338 (-0.180783) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.485857 / 0.215209 (0.270648) | 4.959263 / 2.077655 (2.881608) | 2.501085 / 1.504120 (0.996965) | 2.234660 / 1.541195 (0.693465) | 2.238585 / 1.468490 (0.770095) | 0.645431 / 4.584777 (-3.939345) | 4.434311 / 3.745712 (0.688599) | 4.771491 / 5.269862 (-0.498371) | 2.778963 / 4.565676 (-1.786714) | 0.075615 / 0.424275 (-0.348660) | 0.009502 / 0.007607 (0.001895) | 0.546539 / 0.226044 (0.320495) | 5.464242 / 2.268929 (3.195314) | 2.894101 / 55.444624 (-52.550524) | 2.513761 / 6.876477 (-4.362715) | 2.719843 / 2.142072 (0.577770) | 0.678828 / 4.805227 (-4.126399) | 0.157839 / 6.500664 (-6.342825) | 0.071305 / 0.075469 (-0.004164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.496879 / 1.841788 (-0.344909) | 22.214452 / 8.074308 (14.140144) | 17.707541 / 10.191392 (7.516149) | 0.197008 / 0.680424 (-0.483416) | 0.024883 / 0.534201 (-0.509318) | 0.493611 / 0.579283 (-0.085672) | 0.500677 / 0.434364 (0.066313) | 0.569381 / 0.540337 (0.029044) | 0.773950 / 1.386936 (-0.612986) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007337 / 0.011353 (-0.004015) | 0.004572 / 0.011008 (-0.006436) | 0.091123 / 0.038508 (0.052615) | 0.079762 / 0.023109 (0.056652) | 0.450527 / 0.275898 (0.174629) | 0.525097 / 0.323480 (0.201617) | 0.005873 / 0.007986 (-0.002112) | 0.003797 / 0.004328 (-0.000532) | 0.076259 / 0.004250 (0.072009) | 0.062745 / 0.037052 (0.025692) | 0.465553 / 0.258489 (0.207064) | 0.546026 / 0.293841 (0.252186) | 0.035638 / 0.128546 (-0.092909) | 0.010086 / 0.075646 (-0.065560) | 0.109269 / 0.419271 (-0.310002) | 0.056765 / 0.043533 (0.013233) | 0.440887 / 0.255139 (0.185748) | 0.513325 / 0.283200 (0.230125) | 0.027206 / 0.141683 (-0.114476) | 1.863564 / 1.452155 (0.411409) | 1.918206 / 1.492716 (0.425490) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266479 / 0.018006 (0.248473) | 0.487971 / 0.000490 (0.487481) | 0.012246 / 0.000200 (0.012046) | 0.000119 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035281 / 0.037411 (-0.002130) | 0.102991 / 0.014526 (0.088465) | 0.114638 / 0.176557 (-0.061919) | 0.184117 / 0.737135 (-0.553018) | 0.117943 / 0.296338 (-0.178396) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.497897 / 0.215209 (0.282688) | 4.973806 / 2.077655 (2.896151) | 2.596146 / 1.504120 (1.092026) | 2.419694 / 1.541195 (0.878499) | 2.525784 / 1.468490 (1.057294) | 0.568021 / 4.584777 (-4.016756) | 4.296431 / 3.745712 (0.550719) | 3.690682 / 5.269862 (-1.579179) | 2.345965 / 4.565676 (-2.219712) | 0.066859 / 0.424275 (-0.357416) | 0.009093 / 0.007607 (0.001486) | 0.582616 / 0.226044 (0.356571) | 5.826528 / 2.268929 (3.557600) | 3.253222 / 55.444624 (-52.191403) | 2.798447 / 6.876477 (-4.078030) | 3.054609 / 2.142072 (0.912537) | 0.678816 / 4.805227 (-4.126411) | 0.157966 / 6.500664 (-6.342698) | 0.073797 / 0.075469 (-0.001672) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.599480 / 1.841788 (-0.242308) | 23.249738 / 8.074308 (15.175430) | 16.965406 / 10.191392 (6.774014) | 0.171390 / 0.680424 (-0.509034) | 0.021810 / 0.534201 (-0.512391) | 0.483339 / 0.579283 (-0.095944) | 0.496615 / 0.434364 (0.062251) | 0.583786 / 0.540337 (0.043448) | 0.741699 / 1.386936 (-0.645237) |

</details>
</details>

![](https://cml.dev/watermark.png#7935cd2e564f5d1c66ed1acf731703724ba7a287 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006054 / 0.011353 (-0.005299) | 0.003706 / 0.011008 (-0.007302) | 0.080060 / 0.038508 (0.041552) | 0.061479 / 0.023109 (0.038370) | 0.327981 / 0.275898 (0.052083) | 0.356930 / 0.323480 (0.033450) | 0.004671 / 0.007986 (-0.003315) | 0.002901 / 0.004328 (-0.001428) | 0.062425 / 0.004250 (0.058174) | 0.046310 / 0.037052 (0.009258) | 0.323657 / 0.258489 (0.065168) | 0.370130 / 0.293841 (0.076289) | 0.027151 / 0.128546 (-0.101395) | 0.007850 / 0.075646 (-0.067797) | 0.262300 / 0.419271 (-0.156971) | 0.045456 / 0.043533 (0.001923) | 0.325569 / 0.255139 (0.070430) | 0.352962 / 0.283200 (0.069762) | 0.020156 / 0.141683 (-0.121527) | 1.429404 / 1.452155 (-0.022750) | 1.615032 / 1.492716 (0.122316) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187309 / 0.018006 (0.169303) | 0.428848 / 0.000490 (0.428358) | 0.003599 / 0.000200 (0.003399) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023260 / 0.037411 (-0.014151) | 0.072467 / 0.014526 (0.057941) | 0.082398 / 0.176557 (-0.094159) | 0.142573 / 0.737135 (-0.594562) | 0.082570 / 0.296338 (-0.213768) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426503 / 0.215209 (0.211294) | 4.267875 / 2.077655 (2.190220) | 2.189762 / 1.504120 (0.685642) | 2.027992 / 1.541195 (0.486798) | 2.053211 / 1.468490 (0.584721) | 0.503850 / 4.584777 (-4.080927) | 3.086444 / 3.745712 (-0.659268) | 3.319492 / 5.269862 (-1.950370) | 2.070714 / 4.565676 (-2.494962) | 0.057591 / 0.424275 (-0.366684) | 0.006407 / 0.007607 (-0.001200) | 0.501145 / 0.226044 (0.275100) | 5.017753 / 2.268929 (2.748825) | 2.643145 / 55.444624 (-52.801479) | 2.327440 / 6.876477 (-4.549037) | 2.460250 / 2.142072 (0.318178) | 0.589397 / 4.805227 (-4.215830) | 0.124948 / 6.500664 (-6.375716) | 0.060450 / 0.075469 (-0.015020) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.279870 / 1.841788 (-0.561918) | 18.115908 / 8.074308 (10.041600) | 13.570032 / 10.191392 (3.378640) | 0.132981 / 0.680424 (-0.547442) | 0.016942 / 0.534201 (-0.517259) | 0.333591 / 0.579283 (-0.245692) | 0.358844 / 0.434364 (-0.075520) | 0.395748 / 0.540337 (-0.144590) | 0.546213 / 1.386936 (-0.840723) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006062 / 0.011353 (-0.005291) | 0.003673 / 0.011008 (-0.007336) | 0.064726 / 0.038508 (0.026218) | 0.061854 / 0.023109 (0.038745) | 0.385343 / 0.275898 (0.109445) | 0.441284 / 0.323480 (0.117805) | 0.004830 / 0.007986 (-0.003156) | 0.002909 / 0.004328 (-0.001420) | 0.063874 / 0.004250 (0.059624) | 0.049331 / 0.037052 (0.012278) | 0.418484 / 0.258489 (0.159995) | 0.451397 / 0.293841 (0.157556) | 0.027665 / 0.128546 (-0.100881) | 0.008088 / 0.075646 (-0.067558) | 0.069625 / 0.419271 (-0.349646) | 0.043437 / 0.043533 (-0.000095) | 0.359789 / 0.255139 (0.104650) | 0.430206 / 0.283200 (0.147007) | 0.022308 / 0.141683 (-0.119375) | 1.461030 / 1.452155 (0.008875) | 1.513683 / 1.492716 (0.020966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230958 / 0.018006 (0.212952) | 0.417553 / 0.000490 (0.417063) | 0.000802 / 0.000200 (0.000602) | 0.000066 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025421 / 0.037411 (-0.011991) | 0.077156 / 0.014526 (0.062630) | 0.087533 / 0.176557 (-0.089024) | 0.138048 / 0.737135 (-0.599087) | 0.089358 / 0.296338 (-0.206981) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439172 / 0.215209 (0.223963) | 4.409509 / 2.077655 (2.331854) | 2.491270 / 1.504120 (0.987150) | 2.308446 / 1.541195 (0.767252) | 2.378440 / 1.468490 (0.909950) | 0.499834 / 4.584777 (-4.084943) | 3.083168 / 3.745712 (-0.662544) | 2.867543 / 5.269862 (-2.402318) | 1.876354 / 4.565676 (-2.689323) | 0.057092 / 0.424275 (-0.367183) | 0.006955 / 0.007607 (-0.000653) | 0.513799 / 0.226044 (0.287754) | 5.126660 / 2.268929 (2.857731) | 2.917348 / 55.444624 (-52.527277) | 2.508035 / 6.876477 (-4.368441) | 2.698089 / 2.142072 (0.556016) | 0.586828 / 4.805227 (-4.218399) | 0.124740 / 6.500664 (-6.375924) | 0.062276 / 0.075469 (-0.013193) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291624 / 1.841788 (-0.550164) | 18.199968 / 8.074308 (10.125660) | 13.888139 / 10.191392 (3.696747) | 0.162955 / 0.680424 (-0.517469) | 0.017343 / 0.534201 (-0.516858) | 0.334683 / 0.579283 (-0.244600) | 0.352708 / 0.434364 (-0.081656) | 0.400629 / 0.540337 (-0.139708) | 0.539497 / 1.386936 (-0.847439) |

</details>
</details>

![](https://cml.dev/watermark.png#e7976db7fe22c6b93a869488d07b8137ea6a0db4 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007500 / 0.011353 (-0.003853) | 0.004498 / 0.011008 (-0.006510) | 0.100239 / 0.038508 (0.061731) | 0.083424 / 0.023109 (0.060315) | 0.366664 / 0.275898 (0.090766) | 0.406641 / 0.323480 (0.083161) | 0.004577 / 0.007986 (-0.003409) | 0.004809 / 0.004328 (0.000480) | 0.076898 / 0.004250 (0.072647) | 0.064021 / 0.037052 (0.026969) | 0.375836 / 0.258489 (0.117347) | 0.413008 / 0.293841 (0.119167) | 0.036010 / 0.128546 (-0.092537) | 0.009655 / 0.075646 (-0.065991) | 0.342595 / 0.419271 (-0.076677) | 0.061846 / 0.043533 (0.018313) | 0.376543 / 0.255139 (0.121404) | 0.395858 / 0.283200 (0.112659) | 0.026792 / 0.141683 (-0.114891) | 1.775569 / 1.452155 (0.323414) | 1.865077 / 1.492716 (0.372360) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221521 / 0.018006 (0.203514) | 0.474604 / 0.000490 (0.474114) | 0.004354 / 0.000200 (0.004154) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032947 / 0.037411 (-0.004464) | 0.100454 / 0.014526 (0.085928) | 0.111955 / 0.176557 (-0.064602) | 0.179752 / 0.737135 (-0.557383) | 0.114282 / 0.296338 (-0.182056) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458261 / 0.215209 (0.243052) | 4.563536 / 2.077655 (2.485881) | 2.231928 / 1.504120 (0.727808) | 2.036751 / 1.541195 (0.495556) | 2.170413 / 1.468490 (0.701923) | 0.570825 / 4.584777 (-4.013952) | 4.505762 / 3.745712 (0.760050) | 5.033461 / 5.269862 (-0.236401) | 2.704989 / 4.565676 (-1.860687) | 0.067011 / 0.424275 (-0.357264) | 0.008568 / 0.007607 (0.000961) | 0.545151 / 0.226044 (0.319106) | 5.438984 / 2.268929 (3.170055) | 2.771818 / 55.444624 (-52.672806) | 2.393082 / 6.876477 (-4.483395) | 2.467173 / 2.142072 (0.325101) | 0.678849 / 4.805227 (-4.126379) | 0.160480 / 6.500664 (-6.340184) | 0.073681 / 0.075469 (-0.001788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.532272 / 1.841788 (-0.309516) | 22.548741 / 8.074308 (14.474433) | 17.091044 / 10.191392 (6.899652) | 0.172100 / 0.680424 (-0.508324) | 0.022220 / 0.534201 (-0.511981) | 0.467871 / 0.579283 (-0.111412) | 0.491135 / 0.434364 (0.056771) | 0.548433 / 0.540337 (0.008096) | 0.733340 / 1.386936 (-0.653596) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007593 / 0.011353 (-0.003760) | 0.004656 / 0.011008 (-0.006352) | 0.076940 / 0.038508 (0.038431) | 0.085183 / 0.023109 (0.062073) | 0.447178 / 0.275898 (0.171280) | 0.469545 / 0.323480 (0.146065) | 0.006023 / 0.007986 (-0.001962) | 0.003808 / 0.004328 (-0.000520) | 0.076767 / 0.004250 (0.072517) | 0.065713 / 0.037052 (0.028661) | 0.445573 / 0.258489 (0.187084) | 0.481689 / 0.293841 (0.187848) | 0.036893 / 0.128546 (-0.091654) | 0.009976 / 0.075646 (-0.065670) | 0.084443 / 0.419271 (-0.334829) | 0.058829 / 0.043533 (0.015297) | 0.429291 / 0.255139 (0.174152) | 0.454016 / 0.283200 (0.170816) | 0.027289 / 0.141683 (-0.114394) | 1.806786 / 1.452155 (0.354632) | 1.887680 / 1.492716 (0.394964) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.241012 / 0.018006 (0.223006) | 0.470629 / 0.000490 (0.470139) | 0.003213 / 0.000200 (0.003013) | 0.000107 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036896 / 0.037411 (-0.000515) | 0.106932 / 0.014526 (0.092406) | 0.120333 / 0.176557 (-0.056223) | 0.186271 / 0.737135 (-0.550865) | 0.121581 / 0.296338 (-0.174758) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.507782 / 0.215209 (0.292573) | 5.062932 / 2.077655 (2.985278) | 2.689539 / 1.504120 (1.185419) | 2.482978 / 1.541195 (0.941784) | 2.561320 / 1.468490 (1.092830) | 0.570664 / 4.584777 (-4.014113) | 4.346051 / 3.745712 (0.600339) | 6.479374 / 5.269862 (1.209513) | 4.096483 / 4.565676 (-0.469194) | 0.067564 / 0.424275 (-0.356711) | 0.009147 / 0.007607 (0.001540) | 0.596059 / 0.226044 (0.370015) | 5.963223 / 2.268929 (3.694295) | 3.201039 / 55.444624 (-52.243585) | 2.816581 / 6.876477 (-4.059896) | 3.047821 / 2.142072 (0.905748) | 0.687749 / 4.805227 (-4.117478) | 0.158174 / 6.500664 (-6.342490) | 0.073329 / 0.075469 (-0.002140) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.601346 / 1.841788 (-0.240441) | 23.712210 / 8.074308 (15.637902) | 16.567272 / 10.191392 (6.375880) | 0.224745 / 0.680424 (-0.455679) | 0.021662 / 0.534201 (-0.512539) | 0.471427 / 0.579283 (-0.107856) | 0.498751 / 0.434364 (0.064387) | 0.572047 / 0.540337 (0.031710) | 0.821868 / 1.386936 (-0.565068) |

</details>
</details>

![](https://cml.dev/watermark.png#34d0c9027c750adc89f3d04a6bf2e9cb95915da4 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006371 / 0.011353 (-0.004981) | 0.003749 / 0.011008 (-0.007259) | 0.084155 / 0.038508 (0.045647) | 0.072450 / 0.023109 (0.049340) | 0.308002 / 0.275898 (0.032104) | 0.340471 / 0.323480 (0.016991) | 0.005054 / 0.007986 (-0.002931) | 0.003176 / 0.004328 (-0.001152) | 0.064867 / 0.004250 (0.060616) | 0.054305 / 0.037052 (0.017252) | 0.321047 / 0.258489 (0.062558) | 0.345999 / 0.293841 (0.052158) | 0.030507 / 0.128546 (-0.098039) | 0.008299 / 0.075646 (-0.067347) | 0.287682 / 0.419271 (-0.131590) | 0.052048 / 0.043533 (0.008515) | 0.308322 / 0.255139 (0.053183) | 0.333220 / 0.283200 (0.050020) | 0.022698 / 0.141683 (-0.118985) | 1.474033 / 1.452155 (0.021879) | 1.544790 / 1.492716 (0.052074) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200612 / 0.018006 (0.182606) | 0.450934 / 0.000490 (0.450445) | 0.005383 / 0.000200 (0.005183) | 0.000200 / 0.000054 (0.000145) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027759 / 0.037411 (-0.009652) | 0.080935 / 0.014526 (0.066409) | 0.093041 / 0.176557 (-0.083516) | 0.148643 / 0.737135 (-0.588492) | 0.093463 / 0.296338 (-0.202876) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.381653 / 0.215209 (0.166444) | 3.810699 / 2.077655 (1.733044) | 1.866858 / 1.504120 (0.362738) | 1.716985 / 1.541195 (0.175790) | 1.788071 / 1.468490 (0.319581) | 0.481130 / 4.584777 (-4.103647) | 3.529798 / 3.745712 (-0.215914) | 3.982037 / 5.269862 (-1.287824) | 2.324866 / 4.565676 (-2.240811) | 0.056767 / 0.424275 (-0.367508) | 0.007306 / 0.007607 (-0.000301) | 0.459472 / 0.226044 (0.233428) | 4.602808 / 2.268929 (2.333879) | 2.332014 / 55.444624 (-53.112610) | 2.044858 / 6.876477 (-4.831619) | 2.204165 / 2.142072 (0.062093) | 0.577946 / 4.805227 (-4.227281) | 0.130900 / 6.500664 (-6.369764) | 0.059054 / 0.075469 (-0.016415) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.245211 / 1.841788 (-0.596576) | 19.176397 / 8.074308 (11.102089) | 13.995280 / 10.191392 (3.803888) | 0.171743 / 0.680424 (-0.508681) | 0.018038 / 0.534201 (-0.516163) | 0.392338 / 0.579283 (-0.186945) | 0.419370 / 0.434364 (-0.014994) | 0.477829 / 0.540337 (-0.062508) | 0.677409 / 1.386936 (-0.709527) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006513 / 0.011353 (-0.004840) | 0.003984 / 0.011008 (-0.007024) | 0.064516 / 0.038508 (0.026008) | 0.070504 / 0.023109 (0.047395) | 0.384509 / 0.275898 (0.108611) | 0.410564 / 0.323480 (0.087084) | 0.005310 / 0.007986 (-0.002675) | 0.003268 / 0.004328 (-0.001061) | 0.064684 / 0.004250 (0.060433) | 0.055367 / 0.037052 (0.018315) | 0.399108 / 0.258489 (0.140619) | 0.422740 / 0.293841 (0.128900) | 0.031624 / 0.128546 (-0.096922) | 0.008617 / 0.075646 (-0.067030) | 0.070929 / 0.419271 (-0.348342) | 0.049146 / 0.043533 (0.005613) | 0.385492 / 0.255139 (0.130353) | 0.407434 / 0.283200 (0.124234) | 0.021972 / 0.141683 (-0.119711) | 1.496135 / 1.452155 (0.043980) | 1.533739 / 1.492716 (0.041023) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226218 / 0.018006 (0.208211) | 0.443176 / 0.000490 (0.442686) | 0.000376 / 0.000200 (0.000176) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030315 / 0.037411 (-0.007097) | 0.086416 / 0.014526 (0.071890) | 0.097725 / 0.176557 (-0.078831) | 0.150407 / 0.737135 (-0.586728) | 0.099914 / 0.296338 (-0.196424) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409807 / 0.215209 (0.194598) | 4.099086 / 2.077655 (2.021431) | 2.103160 / 1.504120 (0.599040) | 1.927927 / 1.541195 (0.386733) | 1.977751 / 1.468490 (0.509261) | 0.476995 / 4.584777 (-4.107781) | 3.521835 / 3.745712 (-0.223877) | 3.237695 / 5.269862 (-2.032167) | 1.995953 / 4.565676 (-2.569724) | 0.056208 / 0.424275 (-0.368068) | 0.007660 / 0.007607 (0.000053) | 0.483537 / 0.226044 (0.257492) | 4.833974 / 2.268929 (2.565046) | 2.589115 / 55.444624 (-52.855510) | 2.228076 / 6.876477 (-4.648401) | 2.395271 / 2.142072 (0.253198) | 0.577534 / 4.805227 (-4.227694) | 0.131432 / 6.500664 (-6.369232) | 0.060999 / 0.075469 (-0.014471) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356043 / 1.841788 (-0.485745) | 19.470401 / 8.074308 (11.396093) | 14.091266 / 10.191392 (3.899874) | 0.166809 / 0.680424 (-0.513615) | 0.018782 / 0.534201 (-0.515419) | 0.394916 / 0.579283 (-0.184367) | 0.411378 / 0.434364 (-0.022986) | 0.466886 / 0.540337 (-0.073451) | 0.617369 / 1.386936 (-0.769567) |

</details>
</details>

![](https://cml.dev/watermark.png#601ae6c7baff33a600fd10b12940966024fd2221 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007590 / 0.011353 (-0.003762) | 0.004068 / 0.011008 (-0.006941) | 0.105479 / 0.038508 (0.066971) | 0.085614 / 0.023109 (0.062505) | 0.384325 / 0.275898 (0.108427) | 0.467867 / 0.323480 (0.144387) | 0.004652 / 0.007986 (-0.003333) | 0.005445 / 0.004328 (0.001117) | 0.079604 / 0.004250 (0.075353) | 0.066031 / 0.037052 (0.028978) | 0.426184 / 0.258489 (0.167695) | 0.480712 / 0.293841 (0.186871) | 0.037837 / 0.128546 (-0.090709) | 0.009765 / 0.075646 (-0.065882) | 0.351316 / 0.419271 (-0.067955) | 0.063634 / 0.043533 (0.020101) | 0.420297 / 0.255139 (0.165158) | 0.449169 / 0.283200 (0.165969) | 0.030947 / 0.141683 (-0.110736) | 1.840184 / 1.452155 (0.388029) | 1.934074 / 1.492716 (0.441357) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223483 / 0.018006 (0.205477) | 0.521086 / 0.000490 (0.520596) | 0.000379 / 0.000200 (0.000179) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032011 / 0.037411 (-0.005400) | 0.101474 / 0.014526 (0.086948) | 0.108652 / 0.176557 (-0.067904) | 0.173340 / 0.737135 (-0.563796) | 0.114186 / 0.296338 (-0.182153) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.478020 / 0.215209 (0.262811) | 4.645400 / 2.077655 (2.567746) | 2.590763 / 1.504120 (1.086643) | 2.383002 / 1.541195 (0.841807) | 2.482550 / 1.468490 (1.014060) | 0.572417 / 4.584777 (-4.012360) | 4.233436 / 3.745712 (0.487724) | 4.858823 / 5.269862 (-0.411038) | 2.838913 / 4.565676 (-1.726764) | 0.070010 / 0.424275 (-0.354265) | 0.009602 / 0.007607 (0.001995) | 0.538735 / 0.226044 (0.312691) | 5.534340 / 2.268929 (3.265411) | 2.915006 / 55.444624 (-52.529619) | 2.625132 / 6.876477 (-4.251345) | 2.537838 / 2.142072 (0.395766) | 0.667870 / 4.805227 (-4.137357) | 0.146330 / 6.500664 (-6.354334) | 0.071631 / 0.075469 (-0.003838) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.594686 / 1.841788 (-0.247101) | 22.311113 / 8.074308 (14.236804) | 17.603983 / 10.191392 (7.412591) | 0.195995 / 0.680424 (-0.484428) | 0.022254 / 0.534201 (-0.511947) | 0.479661 / 0.579283 (-0.099622) | 0.463626 / 0.434364 (0.029262) | 0.483465 / 0.540337 (-0.056873) | 0.676141 / 1.386936 (-0.710795) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006146 / 0.011353 (-0.005207) | 0.004856 / 0.011008 (-0.006152) | 0.067506 / 0.038508 (0.028998) | 0.073968 / 0.023109 (0.050859) | 0.470013 / 0.275898 (0.194115) | 0.479022 / 0.323480 (0.155542) | 0.005972 / 0.007986 (-0.002014) | 0.003846 / 0.004328 (-0.000483) | 0.075141 / 0.004250 (0.070890) | 0.058597 / 0.037052 (0.021544) | 0.481454 / 0.258489 (0.222965) | 0.515634 / 0.293841 (0.221793) | 0.034979 / 0.128546 (-0.093567) | 0.010385 / 0.075646 (-0.065261) | 0.072649 / 0.419271 (-0.346622) | 0.058183 / 0.043533 (0.014650) | 0.462138 / 0.255139 (0.206999) | 0.476093 / 0.283200 (0.192893) | 0.032918 / 0.141683 (-0.108765) | 1.820530 / 1.452155 (0.368375) | 1.626360 / 1.492716 (0.133644) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208970 / 0.018006 (0.190964) | 0.492478 / 0.000490 (0.491988) | 0.005487 / 0.000200 (0.005287) | 0.000140 / 0.000054 (0.000086) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037896 / 0.037411 (0.000484) | 0.089752 / 0.014526 (0.075227) | 0.107445 / 0.176557 (-0.069111) | 0.181260 / 0.737135 (-0.555876) | 0.105700 / 0.296338 (-0.190639) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.495031 / 0.215209 (0.279821) | 4.806939 / 2.077655 (2.729284) | 2.227928 / 1.504120 (0.723808) | 2.067117 / 1.541195 (0.525922) | 2.348982 / 1.468490 (0.880492) | 0.567201 / 4.584777 (-4.017576) | 4.166592 / 3.745712 (0.420880) | 3.654329 / 5.269862 (-1.615533) | 2.331092 / 4.565676 (-2.234584) | 0.062212 / 0.424275 (-0.362063) | 0.008775 / 0.007607 (0.001168) | 0.515413 / 0.226044 (0.289369) | 5.449300 / 2.268929 (3.180371) | 3.206574 / 55.444624 (-52.238050) | 2.600455 / 6.876477 (-4.276022) | 3.041162 / 2.142072 (0.899089) | 0.681899 / 4.805227 (-4.123328) | 0.155400 / 6.500664 (-6.345265) | 0.073933 / 0.075469 (-0.001537) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.572329 / 1.841788 (-0.269459) | 23.638519 / 8.074308 (15.564211) | 17.145663 / 10.191392 (6.954271) | 0.232690 / 0.680424 (-0.447734) | 0.028620 / 0.534201 (-0.505581) | 0.488105 / 0.579283 (-0.091178) | 0.490365 / 0.434364 (0.056001) | 0.599501 / 0.540337 (0.059164) | 0.708101 / 1.386936 (-0.678835) |

</details>
</details>

![](https://cml.dev/watermark.png#4a761315900880a25b347ad19b78bd567cfce1f0 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005947 / 0.011353 (-0.005406) | 0.003577 / 0.011008 (-0.007431) | 0.081631 / 0.038508 (0.043122) | 0.058651 / 0.023109 (0.035541) | 0.342742 / 0.275898 (0.066843) | 0.384130 / 0.323480 (0.060650) | 0.004620 / 0.007986 (-0.003366) | 0.002885 / 0.004328 (-0.001444) | 0.063698 / 0.004250 (0.059448) | 0.048953 / 0.037052 (0.011901) | 0.367880 / 0.258489 (0.109391) | 0.407050 / 0.293841 (0.113209) | 0.027242 / 0.128546 (-0.101305) | 0.007914 / 0.075646 (-0.067733) | 0.262156 / 0.419271 (-0.157116) | 0.044750 / 0.043533 (0.001218) | 0.351613 / 0.255139 (0.096474) | 0.380284 / 0.283200 (0.097084) | 0.020080 / 0.141683 (-0.121603) | 1.498101 / 1.452155 (0.045946) | 1.543608 / 1.492716 (0.050892) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180014 / 0.018006 (0.162008) | 0.436172 / 0.000490 (0.435682) | 0.003694 / 0.000200 (0.003494) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024389 / 0.037411 (-0.013022) | 0.072874 / 0.014526 (0.058348) | 0.083469 / 0.176557 (-0.093088) | 0.144600 / 0.737135 (-0.592536) | 0.084229 / 0.296338 (-0.212110) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.391636 / 0.215209 (0.176427) | 3.906941 / 2.077655 (1.829286) | 1.901944 / 1.504120 (0.397825) | 1.762702 / 1.541195 (0.221507) | 1.817970 / 1.468490 (0.349480) | 0.500345 / 4.584777 (-4.084432) | 3.011351 / 3.745712 (-0.734361) | 4.417763 / 5.269862 (-0.852098) | 2.689744 / 4.565676 (-1.875933) | 0.057765 / 0.424275 (-0.366511) | 0.006412 / 0.007607 (-0.001195) | 0.468156 / 0.226044 (0.242112) | 4.664975 / 2.268929 (2.396047) | 2.323355 / 55.444624 (-53.121270) | 1.984280 / 6.876477 (-4.892197) | 2.165215 / 2.142072 (0.023142) | 0.586950 / 4.805227 (-4.218278) | 0.124363 / 6.500664 (-6.376301) | 0.060702 / 0.075469 (-0.014767) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.238870 / 1.841788 (-0.602917) | 18.587360 / 8.074308 (10.513052) | 13.831674 / 10.191392 (3.640282) | 0.143542 / 0.680424 (-0.536882) | 0.016913 / 0.534201 (-0.517288) | 0.332314 / 0.579283 (-0.246969) | 0.345419 / 0.434364 (-0.088945) | 0.381257 / 0.540337 (-0.159081) | 0.537844 / 1.386936 (-0.849092) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006294 / 0.011353 (-0.005059) | 0.003714 / 0.011008 (-0.007294) | 0.062684 / 0.038508 (0.024176) | 0.063520 / 0.023109 (0.040411) | 0.389591 / 0.275898 (0.113693) | 0.444278 / 0.323480 (0.120798) | 0.004825 / 0.007986 (-0.003160) | 0.003010 / 0.004328 (-0.001318) | 0.062767 / 0.004250 (0.058517) | 0.051739 / 0.037052 (0.014686) | 0.434299 / 0.258489 (0.175810) | 0.452003 / 0.293841 (0.158162) | 0.027375 / 0.128546 (-0.101171) | 0.008135 / 0.075646 (-0.067511) | 0.067401 / 0.419271 (-0.351871) | 0.042752 / 0.043533 (-0.000780) | 0.367633 / 0.255139 (0.112494) | 0.433039 / 0.283200 (0.149840) | 0.021086 / 0.141683 (-0.120597) | 1.488024 / 1.452155 (0.035870) | 1.507767 / 1.492716 (0.015050) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230046 / 0.018006 (0.212040) | 0.428085 / 0.000490 (0.427595) | 0.002188 / 0.000200 (0.001988) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026705 / 0.037411 (-0.010706) | 0.082466 / 0.014526 (0.067940) | 0.089378 / 0.176557 (-0.087179) | 0.147287 / 0.737135 (-0.589849) | 0.090426 / 0.296338 (-0.205913) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430882 / 0.215209 (0.215672) | 4.296224 / 2.077655 (2.218569) | 2.229982 / 1.504120 (0.725862) | 2.048506 / 1.541195 (0.507311) | 2.129514 / 1.468490 (0.661024) | 0.502964 / 4.584777 (-4.081813) | 3.048125 / 3.745712 (-0.697587) | 4.208636 / 5.269862 (-1.061226) | 2.594015 / 4.565676 (-1.971661) | 0.057967 / 0.424275 (-0.366308) | 0.006875 / 0.007607 (-0.000732) | 0.513872 / 0.226044 (0.287828) | 5.126435 / 2.268929 (2.857506) | 2.691278 / 55.444624 (-52.753346) | 2.361723 / 6.876477 (-4.514754) | 2.511213 / 2.142072 (0.369141) | 0.593558 / 4.805227 (-4.211670) | 0.129332 / 6.500664 (-6.371332) | 0.064051 / 0.075469 (-0.011418) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.289049 / 1.841788 (-0.552739) | 18.912363 / 8.074308 (10.838055) | 14.226500 / 10.191392 (4.035108) | 0.131392 / 0.680424 (-0.549032) | 0.016750 / 0.534201 (-0.517451) | 0.330078 / 0.579283 (-0.249205) | 0.347588 / 0.434364 (-0.086776) | 0.383234 / 0.540337 (-0.157103) | 0.510967 / 1.386936 (-0.875969) |

</details>
</details>

![](https://cml.dev/watermark.png#d7892beb30bab0633b84398c5ea43d7e69fe38cc "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005974 / 0.011353 (-0.005379) | 0.003691 / 0.011008 (-0.007317) | 0.079410 / 0.038508 (0.040902) | 0.061769 / 0.023109 (0.038660) | 0.323310 / 0.275898 (0.047412) | 0.354325 / 0.323480 (0.030845) | 0.004794 / 0.007986 (-0.003191) | 0.002899 / 0.004328 (-0.001430) | 0.062104 / 0.004250 (0.057854) | 0.048973 / 0.037052 (0.011921) | 0.326497 / 0.258489 (0.068008) | 0.361347 / 0.293841 (0.067506) | 0.026741 / 0.128546 (-0.101805) | 0.007936 / 0.075646 (-0.067710) | 0.259168 / 0.419271 (-0.160104) | 0.044859 / 0.043533 (0.001327) | 0.319342 / 0.255139 (0.064203) | 0.343711 / 0.283200 (0.060511) | 0.022298 / 0.141683 (-0.119384) | 1.451595 / 1.452155 (-0.000560) | 1.573730 / 1.492716 (0.081014) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.173086 / 0.018006 (0.155080) | 0.432400 / 0.000490 (0.431910) | 0.003739 / 0.000200 (0.003539) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024477 / 0.037411 (-0.012934) | 0.073463 / 0.014526 (0.058937) | 0.083410 / 0.176557 (-0.093146) | 0.144760 / 0.737135 (-0.592376) | 0.084199 / 0.296338 (-0.212140) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.388251 / 0.215209 (0.173042) | 3.875375 / 2.077655 (1.797720) | 1.875515 / 1.504120 (0.371395) | 1.729282 / 1.541195 (0.188087) | 1.784732 / 1.468490 (0.316242) | 0.496985 / 4.584777 (-4.087792) | 3.030276 / 3.745712 (-0.715436) | 2.813192 / 5.269862 (-2.456669) | 1.868647 / 4.565676 (-2.697030) | 0.057376 / 0.424275 (-0.366899) | 0.006463 / 0.007607 (-0.001144) | 0.462153 / 0.226044 (0.236108) | 4.586583 / 2.268929 (2.317654) | 2.287730 / 55.444624 (-53.156894) | 1.972177 / 6.876477 (-4.904299) | 2.151592 / 2.142072 (0.009520) | 0.587169 / 4.805227 (-4.218058) | 0.127063 / 6.500664 (-6.373601) | 0.060297 / 0.075469 (-0.015172) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.267651 / 1.841788 (-0.574136) | 18.426011 / 8.074308 (10.351703) | 14.050470 / 10.191392 (3.859078) | 0.148063 / 0.680424 (-0.532361) | 0.017112 / 0.534201 (-0.517089) | 0.330051 / 0.579283 (-0.249232) | 0.358730 / 0.434364 (-0.075634) | 0.392365 / 0.540337 (-0.147972) | 0.534650 / 1.386936 (-0.852286) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005936 / 0.011353 (-0.005417) | 0.003652 / 0.011008 (-0.007356) | 0.063066 / 0.038508 (0.024558) | 0.060617 / 0.023109 (0.037507) | 0.388293 / 0.275898 (0.112395) | 0.411422 / 0.323480 (0.087942) | 0.004691 / 0.007986 (-0.003295) | 0.002857 / 0.004328 (-0.001472) | 0.064198 / 0.004250 (0.059947) | 0.049124 / 0.037052 (0.012071) | 0.403601 / 0.258489 (0.145112) | 0.413619 / 0.293841 (0.119778) | 0.027279 / 0.128546 (-0.101267) | 0.008072 / 0.075646 (-0.067575) | 0.067890 / 0.419271 (-0.351381) | 0.041866 / 0.043533 (-0.001667) | 0.393438 / 0.255139 (0.138299) | 0.402865 / 0.283200 (0.119666) | 0.023381 / 0.141683 (-0.118302) | 1.496324 / 1.452155 (0.044170) | 1.538080 / 1.492716 (0.045364) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212065 / 0.018006 (0.194059) | 0.410511 / 0.000490 (0.410021) | 0.001236 / 0.000200 (0.001036) | 0.000067 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026012 / 0.037411 (-0.011399) | 0.076592 / 0.014526 (0.062066) | 0.085963 / 0.176557 (-0.090594) | 0.137803 / 0.737135 (-0.599332) | 0.087594 / 0.296338 (-0.208745) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434283 / 0.215209 (0.219074) | 4.345478 / 2.077655 (2.267824) | 2.400954 / 1.504120 (0.896834) | 2.282024 / 1.541195 (0.740829) | 2.414247 / 1.468490 (0.945757) | 0.501855 / 4.584777 (-4.082922) | 3.059433 / 3.745712 (-0.686279) | 2.811288 / 5.269862 (-2.458574) | 1.856839 / 4.565676 (-2.708838) | 0.058017 / 0.424275 (-0.366258) | 0.006844 / 0.007607 (-0.000763) | 0.515376 / 0.226044 (0.289332) | 5.148775 / 2.268929 (2.879847) | 2.930807 / 55.444624 (-52.513817) | 2.520532 / 6.876477 (-4.355944) | 2.746299 / 2.142072 (0.604227) | 0.590102 / 4.805227 (-4.215125) | 0.125747 / 6.500664 (-6.374917) | 0.061873 / 0.075469 (-0.013597) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.306247 / 1.841788 (-0.535541) | 18.366048 / 8.074308 (10.291740) | 13.855617 / 10.191392 (3.664225) | 0.150124 / 0.680424 (-0.530300) | 0.017189 / 0.534201 (-0.517012) | 0.336285 / 0.579283 (-0.242998) | 0.344985 / 0.434364 (-0.089379) | 0.397973 / 0.540337 (-0.142364) | 0.536142 / 1.386936 (-0.850794) |

</details>
</details>

![](https://cml.dev/watermark.png#1ae24cf12054b4a512f198979b1ca7707bb99d56 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006401 / 0.011353 (-0.004952) | 0.003789 / 0.011008 (-0.007219) | 0.079516 / 0.038508 (0.041008) | 0.068279 / 0.023109 (0.045170) | 0.295691 / 0.275898 (0.019793) | 0.327208 / 0.323480 (0.003728) | 0.005070 / 0.007986 (-0.002915) | 0.003044 / 0.004328 (-0.001285) | 0.061411 / 0.004250 (0.057161) | 0.053227 / 0.037052 (0.016175) | 0.297368 / 0.258489 (0.038879) | 0.334740 / 0.293841 (0.040899) | 0.029459 / 0.128546 (-0.099087) | 0.008080 / 0.075646 (-0.067566) | 0.267344 / 0.419271 (-0.151927) | 0.049877 / 0.043533 (0.006344) | 0.293853 / 0.255139 (0.038714) | 0.319819 / 0.283200 (0.036620) | 0.022593 / 0.141683 (-0.119089) | 1.459054 / 1.452155 (0.006900) | 1.471250 / 1.492716 (-0.021466) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.194326 / 0.018006 (0.176320) | 0.443565 / 0.000490 (0.443075) | 0.003745 / 0.000200 (0.003545) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026640 / 0.037411 (-0.010772) | 0.077630 / 0.014526 (0.063104) | 0.089364 / 0.176557 (-0.087192) | 0.147327 / 0.737135 (-0.589809) | 0.089603 / 0.296338 (-0.206735) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.373758 / 0.215209 (0.158549) | 3.746778 / 2.077655 (1.669123) | 1.814991 / 1.504120 (0.310871) | 1.645650 / 1.541195 (0.104455) | 1.690752 / 1.468490 (0.222262) | 0.472117 / 4.584777 (-4.112660) | 3.457346 / 3.745712 (-0.288367) | 3.138869 / 5.269862 (-2.130993) | 1.934924 / 4.565676 (-2.630753) | 0.055709 / 0.424275 (-0.368566) | 0.006680 / 0.007607 (-0.000927) | 0.446874 / 0.226044 (0.220829) | 4.458409 / 2.268929 (2.189480) | 2.253932 / 55.444624 (-53.190693) | 2.007240 / 6.876477 (-4.869237) | 2.081687 / 2.142072 (-0.060386) | 0.563379 / 4.805227 (-4.241848) | 0.128694 / 6.500664 (-6.371970) | 0.057409 / 0.075469 (-0.018060) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.212231 / 1.841788 (-0.629556) | 18.519121 / 8.074308 (10.444813) | 13.582243 / 10.191392 (3.390851) | 0.142488 / 0.680424 (-0.537936) | 0.017421 / 0.534201 (-0.516780) | 0.366864 / 0.579283 (-0.212419) | 0.401467 / 0.434364 (-0.032897) | 0.443659 / 0.540337 (-0.096679) | 0.618854 / 1.386936 (-0.768082) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006121 / 0.011353 (-0.005232) | 0.003690 / 0.011008 (-0.007318) | 0.060340 / 0.038508 (0.021832) | 0.067215 / 0.023109 (0.044106) | 0.382846 / 0.275898 (0.106948) | 0.415774 / 0.323480 (0.092294) | 0.004868 / 0.007986 (-0.003118) | 0.003108 / 0.004328 (-0.001221) | 0.060572 / 0.004250 (0.056321) | 0.050453 / 0.037052 (0.013401) | 0.400494 / 0.258489 (0.142005) | 0.424368 / 0.293841 (0.130527) | 0.030279 / 0.128546 (-0.098267) | 0.008151 / 0.075646 (-0.067495) | 0.066707 / 0.419271 (-0.352564) | 0.046118 / 0.043533 (0.002585) | 0.386697 / 0.255139 (0.131558) | 0.410156 / 0.283200 (0.126957) | 0.020688 / 0.141683 (-0.120995) | 1.418162 / 1.452155 (-0.033993) | 1.463057 / 1.492716 (-0.029659) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216081 / 0.018006 (0.198075) | 0.440541 / 0.000490 (0.440051) | 0.000371 / 0.000200 (0.000171) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027763 / 0.037411 (-0.009648) | 0.082316 / 0.014526 (0.067791) | 0.094086 / 0.176557 (-0.082471) | 0.144738 / 0.737135 (-0.592398) | 0.094837 / 0.296338 (-0.201501) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396277 / 0.215209 (0.181068) | 3.958791 / 2.077655 (1.881136) | 2.021367 / 1.504120 (0.517247) | 1.860112 / 1.541195 (0.318917) | 1.886032 / 1.468490 (0.417541) | 0.468536 / 4.584777 (-4.116241) | 3.417950 / 3.745712 (-0.327762) | 4.849991 / 5.269862 (-0.419871) | 2.773935 / 4.565676 (-1.791742) | 0.055813 / 0.424275 (-0.368462) | 0.007053 / 0.007607 (-0.000554) | 0.470167 / 0.226044 (0.244122) | 4.702969 / 2.268929 (2.434041) | 2.474161 / 55.444624 (-52.970464) | 2.171256 / 6.876477 (-4.705220) | 2.315373 / 2.142072 (0.173301) | 0.589195 / 4.805227 (-4.216032) | 0.128237 / 6.500664 (-6.372427) | 0.058641 / 0.075469 (-0.016828) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.292947 / 1.841788 (-0.548841) | 18.851300 / 8.074308 (10.776992) | 14.089764 / 10.191392 (3.898372) | 0.164853 / 0.680424 (-0.515571) | 0.017281 / 0.534201 (-0.516920) | 0.359112 / 0.579283 (-0.220171) | 0.386696 / 0.434364 (-0.047668) | 0.428222 / 0.540337 (-0.112115) | 0.568659 / 1.386936 (-0.818277) |

</details>
</details>

![](https://cml.dev/watermark.png#563864ded894b468e2ba3f677ef79c5ab3fe65df "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006051 / 0.011353 (-0.005301) | 0.003654 / 0.011008 (-0.007355) | 0.080081 / 0.038508 (0.041572) | 0.062925 / 0.023109 (0.039815) | 0.358097 / 0.275898 (0.082199) | 0.405728 / 0.323480 (0.082248) | 0.005359 / 0.007986 (-0.002627) | 0.002820 / 0.004328 (-0.001508) | 0.063108 / 0.004250 (0.058858) | 0.049627 / 0.037052 (0.012575) | 0.397870 / 0.258489 (0.139381) | 0.437157 / 0.293841 (0.143316) | 0.027707 / 0.128546 (-0.100839) | 0.007911 / 0.075646 (-0.067735) | 0.260991 / 0.419271 (-0.158280) | 0.044771 / 0.043533 (0.001238) | 0.340230 / 0.255139 (0.085091) | 0.384925 / 0.283200 (0.101725) | 0.021369 / 0.141683 (-0.120314) | 1.431439 / 1.452155 (-0.020715) | 1.478794 / 1.492716 (-0.013922) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.182626 / 0.018006 (0.164620) | 0.435551 / 0.000490 (0.435061) | 0.003015 / 0.000200 (0.002815) | 0.000064 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024703 / 0.037411 (-0.012708) | 0.073640 / 0.014526 (0.059114) | 0.084598 / 0.176557 (-0.091959) | 0.145810 / 0.737135 (-0.591325) | 0.085125 / 0.296338 (-0.211213) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.394539 / 0.215209 (0.179330) | 3.945882 / 2.077655 (1.868227) | 1.947166 / 1.504120 (0.443046) | 1.763305 / 1.541195 (0.222111) | 1.816208 / 1.468490 (0.347718) | 0.498880 / 4.584777 (-4.085897) | 3.098283 / 3.745712 (-0.647429) | 2.823474 / 5.269862 (-2.446388) | 1.873993 / 4.565676 (-2.691684) | 0.058097 / 0.424275 (-0.366179) | 0.006488 / 0.007607 (-0.001119) | 0.466711 / 0.226044 (0.240667) | 4.671520 / 2.268929 (2.402592) | 2.363381 / 55.444624 (-53.081243) | 2.052092 / 6.876477 (-4.824385) | 2.209212 / 2.142072 (0.067140) | 0.594650 / 4.805227 (-4.210577) | 0.125604 / 6.500664 (-6.375060) | 0.061511 / 0.075469 (-0.013958) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.226564 / 1.841788 (-0.615224) | 18.583605 / 8.074308 (10.509297) | 13.993091 / 10.191392 (3.801699) | 0.146185 / 0.680424 (-0.534239) | 0.016839 / 0.534201 (-0.517362) | 0.334116 / 0.579283 (-0.245167) | 0.360780 / 0.434364 (-0.073584) | 0.386008 / 0.540337 (-0.154329) | 0.643278 / 1.386936 (-0.743658) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006174 / 0.011353 (-0.005179) | 0.003658 / 0.011008 (-0.007350) | 0.063250 / 0.038508 (0.024742) | 0.063542 / 0.023109 (0.040433) | 0.366845 / 0.275898 (0.090947) | 0.409794 / 0.323480 (0.086314) | 0.005678 / 0.007986 (-0.002308) | 0.003061 / 0.004328 (-0.001268) | 0.063561 / 0.004250 (0.059311) | 0.052648 / 0.037052 (0.015596) | 0.378096 / 0.258489 (0.119607) | 0.410706 / 0.293841 (0.116865) | 0.027668 / 0.128546 (-0.100878) | 0.008045 / 0.075646 (-0.067601) | 0.068290 / 0.419271 (-0.350981) | 0.042602 / 0.043533 (-0.000930) | 0.364976 / 0.255139 (0.109837) | 0.395599 / 0.283200 (0.112400) | 0.022733 / 0.141683 (-0.118950) | 1.522473 / 1.452155 (0.070319) | 1.515891 / 1.492716 (0.023175) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232554 / 0.018006 (0.214547) | 0.420702 / 0.000490 (0.420213) | 0.002161 / 0.000200 (0.001961) | 0.000064 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026276 / 0.037411 (-0.011135) | 0.078504 / 0.014526 (0.063978) | 0.088989 / 0.176557 (-0.087567) | 0.144044 / 0.737135 (-0.593091) | 0.091074 / 0.296338 (-0.205265) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420189 / 0.215209 (0.204980) | 4.189596 / 2.077655 (2.111941) | 2.316425 / 1.504120 (0.812305) | 2.186877 / 1.541195 (0.645682) | 2.259065 / 1.468490 (0.790575) | 0.502827 / 4.584777 (-4.081950) | 3.135266 / 3.745712 (-0.610446) | 2.838808 / 5.269862 (-2.431053) | 1.876519 / 4.565676 (-2.689158) | 0.057802 / 0.424275 (-0.366473) | 0.006824 / 0.007607 (-0.000784) | 0.500213 / 0.226044 (0.274168) | 4.999798 / 2.268929 (2.730869) | 2.627713 / 55.444624 (-52.816911) | 2.344263 / 6.876477 (-4.532214) | 2.415449 / 2.142072 (0.273376) | 0.593082 / 4.805227 (-4.212145) | 0.125787 / 6.500664 (-6.374877) | 0.062699 / 0.075469 (-0.012770) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.308219 / 1.841788 (-0.533569) | 18.703099 / 8.074308 (10.628791) | 13.976234 / 10.191392 (3.784842) | 0.144037 / 0.680424 (-0.536387) | 0.016592 / 0.534201 (-0.517609) | 0.333078 / 0.579283 (-0.246206) | 0.342317 / 0.434364 (-0.092047) | 0.396837 / 0.540337 (-0.143500) | 0.532641 / 1.386936 (-0.854295) |

</details>
</details>

![](https://cml.dev/watermark.png#14f6edd9222e577dccb962ed5338b79b73502fa5 "CML watermark")