Title: fix tqdm lock deletion
Author: lhoestq (MEMBER)
State: closed
Created: 2023-07-25T11:17:25Z

related to https://github.com/huggingface/datasets/issues/6066

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006573 / 0.011353 (-0.004780) | 0.004014 / 0.011008 (-0.006994) | 0.084999 / 0.038508 (0.046491) | 0.074965 / 0.023109 (0.051855) | 0.313294 / 0.275898 (0.037396) | 0.349678 / 0.323480 (0.026198) | 0.005401 / 0.007986 (-0.002585) | 0.003401 / 0.004328 (-0.000927) | 0.065363 / 0.004250 (0.061112) | 0.057159 / 0.037052 (0.020107) | 0.313260 / 0.258489 (0.054771) | 0.354654 / 0.293841 (0.060813) | 0.030895 / 0.128546 (-0.097651) | 0.008605 / 0.075646 (-0.067042) | 0.289190 / 0.419271 (-0.130081) | 0.052474 / 0.043533 (0.008942) | 0.316193 / 0.255139 (0.061054) | 0.339966 / 0.283200 (0.056767) | 0.024112 / 0.141683 (-0.117571) | 1.515606 / 1.452155 (0.063452) | 1.571428 / 1.492716 (0.078711) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203284 / 0.018006 (0.185278) | 0.452720 / 0.000490 (0.452230) | 0.003891 / 0.000200 (0.003691) | 0.000094 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028992 / 0.037411 (-0.008419) | 0.083170 / 0.014526 (0.068644) | 0.097739 / 0.176557 (-0.078817) | 0.153401 / 0.737135 (-0.583734) | 0.098628 / 0.296338 (-0.197711) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.390190 / 0.215209 (0.174981) | 3.901272 / 2.077655 (1.823617) | 1.887194 / 1.504120 (0.383074) | 1.723696 / 1.541195 (0.182501) | 1.800537 / 1.468490 (0.332047) | 0.481758 / 4.584777 (-4.103019) | 3.605098 / 3.745712 (-0.140614) | 3.304482 / 5.269862 (-1.965380) | 2.053515 / 4.565676 (-2.512161) | 0.056997 / 0.424275 (-0.367278) | 0.007347 / 0.007607 (-0.000260) | 0.461367 / 0.226044 (0.235323) | 4.606152 / 2.268929 (2.337223) | 2.470048 / 55.444624 (-52.974576) | 2.060019 / 6.876477 (-4.816458) | 2.320507 / 2.142072 (0.178435) | 0.575050 / 4.805227 (-4.230178) | 0.133030 / 6.500664 (-6.367634) | 0.061508 / 0.075469 (-0.013962) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.275430 / 1.841788 (-0.566357) | 19.725453 / 8.074308 (11.651145) | 14.396360 / 10.191392 (4.204968) | 0.157980 / 0.680424 (-0.522443) | 0.018516 / 0.534201 (-0.515685) | 0.394717 / 0.579283 (-0.184566) | 0.404948 / 0.434364 (-0.029415) | 0.474001 / 0.540337 (-0.066336) | 0.668463 / 1.386936 (-0.718474) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006697 / 0.011353 (-0.004656) | 0.004206 / 0.011008 (-0.006802) | 0.065458 / 0.038508 (0.026950) | 0.075845 / 0.023109 (0.052735) | 0.365051 / 0.275898 (0.089153) | 0.400919 / 0.323480 (0.077439) | 0.005347 / 0.007986 (-0.002638) | 0.003386 / 0.004328 (-0.000943) | 0.065398 / 0.004250 (0.061148) | 0.057320 / 0.037052 (0.020268) | 0.379161 / 0.258489 (0.120672) | 0.406892 / 0.293841 (0.113051) | 0.031986 / 0.128546 (-0.096560) | 0.008674 / 0.075646 (-0.066972) | 0.071723 / 0.419271 (-0.347549) | 0.049897 / 0.043533 (0.006364) | 0.372034 / 0.255139 (0.116895) | 0.394293 / 0.283200 (0.111094) | 0.023681 / 0.141683 (-0.118002) | 1.479793 / 1.452155 (0.027639) | 1.553105 / 1.492716 (0.060389) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233660 / 0.018006 (0.215654) | 0.454412 / 0.000490 (0.453923) | 0.004473 / 0.000200 (0.004273) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031115 / 0.037411 (-0.006296) | 0.090541 / 0.014526 (0.076015) | 0.104363 / 0.176557 (-0.072193) | 0.161022 / 0.737135 (-0.576114) | 0.105114 / 0.296338 (-0.191225) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427668 / 0.215209 (0.212459) | 4.263145 / 2.077655 (2.185490) | 2.247043 / 1.504120 (0.742923) | 2.082554 / 1.541195 (0.541360) | 2.170505 / 1.468490 (0.702015) | 0.491802 / 4.584777 (-4.092975) | 3.587295 / 3.745712 (-0.158417) | 3.344697 / 5.269862 (-1.925165) | 2.060529 / 4.565676 (-2.505148) | 0.057829 / 0.424275 (-0.366446) | 0.007780 / 0.007607 (0.000173) | 0.503374 / 0.226044 (0.277330) | 5.034742 / 2.268929 (2.765814) | 2.701957 / 55.444624 (-52.742667) | 2.479002 / 6.876477 (-4.397474) | 2.622055 / 2.142072 (0.479982) | 0.591363 / 4.805227 (-4.213864) | 0.133834 / 6.500664 (-6.366830) | 0.062276 / 0.075469 (-0.013193) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.338788 / 1.841788 (-0.503000) | 20.333599 / 8.074308 (12.259291) | 14.783196 / 10.191392 (4.591804) | 0.168695 / 0.680424 (-0.511729) | 0.018478 / 0.534201 (-0.515723) | 0.397398 / 0.579283 (-0.181885) | 0.409900 / 0.434364 (-0.024464) | 0.475315 / 0.540337 (-0.065023) | 0.644267 / 1.386936 (-0.742669) |

</details>
</details>

![](https://cml.dev/watermark.png#cb0b324e0bae4c93bb5509b2f0731bc346adb21b "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007315 / 0.011353 (-0.004038) | 0.004294 / 0.011008 (-0.006714) | 0.100300 / 0.038508 (0.061792) | 0.077780 / 0.023109 (0.054670) | 0.353728 / 0.275898 (0.077830) | 0.400538 / 0.323480 (0.077058) | 0.005807 / 0.007986 (-0.002178) | 0.003649 / 0.004328 (-0.000680) | 0.077548 / 0.004250 (0.073297) | 0.058834 / 0.037052 (0.021781) | 0.352064 / 0.258489 (0.093574) | 0.399951 / 0.293841 (0.106110) | 0.036472 / 0.128546 (-0.092074) | 0.008653 / 0.075646 (-0.066994) | 0.323089 / 0.419271 (-0.096182) | 0.075127 / 0.043533 (0.031594) | 0.334412 / 0.255139 (0.079273) | 0.375718 / 0.283200 (0.092519) | 0.027915 / 0.141683 (-0.113768) | 1.698795 / 1.452155 (0.246640) | 1.781447 / 1.492716 (0.288730) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216111 / 0.018006 (0.198104) | 0.507706 / 0.000490 (0.507216) | 0.000851 / 0.000200 (0.000651) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030451 / 0.037411 (-0.006960) | 0.087488 / 0.014526 (0.072962) | 0.105094 / 0.176557 (-0.071462) | 0.168130 / 0.737135 (-0.569006) | 0.106791 / 0.296338 (-0.189547) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426291 / 0.215209 (0.211082) | 4.281046 / 2.077655 (2.203391) | 2.162268 / 1.504120 (0.658148) | 1.909503 / 1.541195 (0.368309) | 1.943165 / 1.468490 (0.474675) | 0.516667 / 4.584777 (-4.068110) | 4.113218 / 3.745712 (0.367506) | 5.931372 / 5.269862 (0.661510) | 3.563521 / 4.565676 (-1.002155) | 0.062415 / 0.424275 (-0.361860) | 0.007577 / 0.007607 (-0.000030) | 0.534588 / 0.226044 (0.308543) | 5.183490 / 2.268929 (2.914561) | 2.790662 / 55.444624 (-52.653962) | 2.258630 / 6.876477 (-4.617846) | 2.499930 / 2.142072 (0.357857) | 0.606154 / 4.805227 (-4.199073) | 0.136093 / 6.500664 (-6.364571) | 0.061151 / 0.075469 (-0.014318) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.398392 / 1.841788 (-0.443396) | 21.482150 / 8.074308 (13.407842) | 15.477336 / 10.191392 (5.285944) | 0.192878 / 0.680424 (-0.487546) | 0.021764 / 0.534201 (-0.512437) | 0.437149 / 0.579283 (-0.142134) | 0.439976 / 0.434364 (0.005612) | 0.514498 / 0.540337 (-0.025840) | 0.762642 / 1.386936 (-0.624294) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007504 / 0.011353 (-0.003849) | 0.004526 / 0.011008 (-0.006482) | 0.071008 / 0.038508 (0.032500) | 0.078305 / 0.023109 (0.055195) | 0.436160 / 0.275898 (0.160262) | 0.439048 / 0.323480 (0.115568) | 0.006061 / 0.007986 (-0.001925) | 0.003681 / 0.004328 (-0.000648) | 0.069445 / 0.004250 (0.065195) | 0.059258 / 0.037052 (0.022206) | 0.437745 / 0.258489 (0.179256) | 0.464247 / 0.293841 (0.170406) | 0.033286 / 0.128546 (-0.095260) | 0.009846 / 0.075646 (-0.065800) | 0.076330 / 0.419271 (-0.342941) | 0.051919 / 0.043533 (0.008386) | 0.432817 / 0.255139 (0.177678) | 0.426295 / 0.283200 (0.143095) | 0.029818 / 0.141683 (-0.111865) | 1.747640 / 1.452155 (0.295485) | 1.726653 / 1.492716 (0.233937) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.251253 / 0.018006 (0.233247) | 0.483394 / 0.000490 (0.482904) | 0.003992 / 0.000200 (0.003793) | 0.000096 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032180 / 0.037411 (-0.005231) | 0.095425 / 0.014526 (0.080900) | 0.105908 / 0.176557 (-0.070648) | 0.164732 / 0.737135 (-0.572403) | 0.115903 / 0.296338 (-0.180435) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.469467 / 0.215209 (0.254258) | 4.633239 / 2.077655 (2.555584) | 2.517557 / 1.504120 (1.013437) | 2.352726 / 1.541195 (0.811531) | 2.314618 / 1.468490 (0.846128) | 0.548446 / 4.584777 (-4.036331) | 3.908797 / 3.745712 (0.163085) | 3.525941 / 5.269862 (-1.743921) | 2.178858 / 4.565676 (-2.386819) | 0.057614 / 0.424275 (-0.366661) | 0.008604 / 0.007607 (0.000997) | 0.554756 / 0.226044 (0.328711) | 5.325635 / 2.268929 (3.056706) | 3.014266 / 55.444624 (-52.430359) | 2.844165 / 6.876477 (-4.032312) | 2.903019 / 2.142072 (0.760947) | 0.617750 / 4.805227 (-4.187478) | 0.144259 / 6.500664 (-6.356405) | 0.065944 / 0.075469 (-0.009525) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.504625 / 1.841788 (-0.337163) | 22.400787 / 8.074308 (14.326479) | 15.223702 / 10.191392 (5.032310) | 0.213357 / 0.680424 (-0.467067) | 0.019310 / 0.534201 (-0.514891) | 0.456596 / 0.579283 (-0.122687) | 0.473811 / 0.434364 (0.039447) | 0.517800 / 0.540337 (-0.022537) | 0.792468 / 1.386936 (-0.594468) |

</details>
</details>

![](https://cml.dev/watermark.png#03750f4a4c664125c7de910be004710b181dd354 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007420 / 0.011353 (-0.003933) | 0.004502 / 0.011008 (-0.006506) | 0.097882 / 0.038508 (0.059374) | 0.079084 / 0.023109 (0.055975) | 0.361797 / 0.275898 (0.085899) | 0.416563 / 0.323480 (0.093083) | 0.006106 / 0.007986 (-0.001879) | 0.003803 / 0.004328 (-0.000526) | 0.074669 / 0.004250 (0.070418) | 0.062168 / 0.037052 (0.025116) | 0.378844 / 0.258489 (0.120355) | 0.426601 / 0.293841 (0.132760) | 0.035619 / 0.128546 (-0.092927) | 0.009686 / 0.075646 (-0.065960) | 0.336481 / 0.419271 (-0.082790) | 0.065553 / 0.043533 (0.022021) | 0.362501 / 0.255139 (0.107362) | 0.399752 / 0.283200 (0.116552) | 0.028685 / 0.141683 (-0.112998) | 1.683495 / 1.452155 (0.231340) | 1.786105 / 1.492716 (0.293388) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220792 / 0.018006 (0.202786) | 0.501936 / 0.000490 (0.501447) | 0.000389 / 0.000200 (0.000189) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032180 / 0.037411 (-0.005232) | 0.093079 / 0.014526 (0.078553) | 0.107967 / 0.176557 (-0.068589) | 0.171747 / 0.737135 (-0.565389) | 0.107920 / 0.296338 (-0.188418) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444431 / 0.215209 (0.229222) | 4.454934 / 2.077655 (2.377279) | 2.140265 / 1.504120 (0.636145) | 1.960126 / 1.541195 (0.418931) | 2.049649 / 1.468490 (0.581158) | 0.557861 / 4.584777 (-4.026916) | 4.046240 / 3.745712 (0.300528) | 4.513748 / 5.269862 (-0.756114) | 2.593643 / 4.565676 (-1.972034) | 0.066795 / 0.424275 (-0.357480) | 0.008302 / 0.007607 (0.000694) | 0.535643 / 0.226044 (0.309599) | 5.299429 / 2.268929 (3.030500) | 2.656019 / 55.444624 (-52.788606) | 2.281214 / 6.876477 (-4.595263) | 2.302910 / 2.142072 (0.160837) | 0.661696 / 4.805227 (-4.143532) | 0.149787 / 6.500664 (-6.350877) | 0.069609 / 0.075469 (-0.005860) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.509842 / 1.841788 (-0.331946) | 21.717504 / 8.074308 (13.643196) | 15.825102 / 10.191392 (5.633710) | 0.168115 / 0.680424 (-0.512309) | 0.021637 / 0.534201 (-0.512564) | 0.454270 / 0.579283 (-0.125013) | 0.458531 / 0.434364 (0.024167) | 0.523052 / 0.540337 (-0.017285) | 0.711219 / 1.386936 (-0.675717) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007189 / 0.011353 (-0.004164) | 0.004437 / 0.011008 (-0.006571) | 0.075111 / 0.038508 (0.036603) | 0.079245 / 0.023109 (0.056136) | 0.423169 / 0.275898 (0.147270) | 0.455007 / 0.323480 (0.131527) | 0.006076 / 0.007986 (-0.001909) | 0.003819 / 0.004328 (-0.000509) | 0.074976 / 0.004250 (0.070726) | 0.062127 / 0.037052 (0.025075) | 0.456809 / 0.258489 (0.198320) | 0.474707 / 0.293841 (0.180867) | 0.036221 / 0.128546 (-0.092325) | 0.009428 / 0.075646 (-0.066218) | 0.082842 / 0.419271 (-0.336429) | 0.057086 / 0.043533 (0.013553) | 0.436121 / 0.255139 (0.180982) | 0.453934 / 0.283200 (0.170734) | 0.026045 / 0.141683 (-0.115638) | 1.789782 / 1.452155 (0.337627) | 1.820934 / 1.492716 (0.328218) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230790 / 0.018006 (0.212784) | 0.497987 / 0.000490 (0.497497) | 0.002775 / 0.000200 (0.002575) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034418 / 0.037411 (-0.002994) | 0.105567 / 0.014526 (0.091041) | 0.113134 / 0.176557 (-0.063423) | 0.173742 / 0.737135 (-0.563394) | 0.115936 / 0.296338 (-0.180403) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.502259 / 0.215209 (0.287050) | 4.969877 / 2.077655 (2.892222) | 2.684860 / 1.504120 (1.180740) | 2.484386 / 1.541195 (0.943192) | 2.543061 / 1.468490 (1.074571) | 0.545733 / 4.584777 (-4.039044) | 4.029660 / 3.745712 (0.283948) | 5.927883 / 5.269862 (0.658021) | 3.528372 / 4.565676 (-1.037305) | 0.065957 / 0.424275 (-0.358318) | 0.008933 / 0.007607 (0.001326) | 0.601630 / 0.226044 (0.375585) | 5.825872 / 2.268929 (3.556944) | 3.230721 / 55.444624 (-52.213904) | 2.891308 / 6.876477 (-3.985169) | 3.054994 / 2.142072 (0.912922) | 0.665480 / 4.805227 (-4.139747) | 0.154815 / 6.500664 (-6.345849) | 0.072997 / 0.075469 (-0.002472) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.549892 / 1.841788 (-0.291896) | 22.337484 / 8.074308 (14.263176) | 16.308286 / 10.191392 (6.116894) | 0.189594 / 0.680424 (-0.490830) | 0.021844 / 0.534201 (-0.512357) | 0.456958 / 0.579283 (-0.122325) | 0.459957 / 0.434364 (0.025593) | 0.529014 / 0.540337 (-0.011323) | 0.700359 / 1.386936 (-0.686577) |

</details>
</details>

![](https://cml.dev/watermark.png#32e4df86b5fb0bc164433ce615af641ec3ba437e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009050 / 0.011353 (-0.002303) | 0.004968 / 0.011008 (-0.006040) | 0.114315 / 0.038508 (0.075807) | 0.084475 / 0.023109 (0.061366) | 0.426325 / 0.275898 (0.150427) | 0.457870 / 0.323480 (0.134390) | 0.007076 / 0.007986 (-0.000910) | 0.004635 / 0.004328 (0.000307) | 0.082950 / 0.004250 (0.078700) | 0.065414 / 0.037052 (0.028361) | 0.441936 / 0.258489 (0.183447) | 0.476983 / 0.293841 (0.183142) | 0.048575 / 0.128546 (-0.079972) | 0.013929 / 0.075646 (-0.061717) | 0.377498 / 0.419271 (-0.041774) | 0.081503 / 0.043533 (0.037970) | 0.426706 / 0.255139 (0.171567) | 0.460374 / 0.283200 (0.177175) | 0.046052 / 0.141683 (-0.095631) | 1.894896 / 1.452155 (0.442741) | 1.998639 / 1.492716 (0.505923) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.313267 / 0.018006 (0.295261) | 0.607501 / 0.000490 (0.607012) | 0.003369 / 0.000200 (0.003169) | 0.000102 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032266 / 0.037411 (-0.005145) | 0.120138 / 0.014526 (0.105613) | 0.115044 / 0.176557 (-0.061513) | 0.181374 / 0.737135 (-0.555761) | 0.114681 / 0.296338 (-0.181657) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.648039 / 0.215209 (0.432830) | 6.005048 / 2.077655 (3.927394) | 2.674524 / 1.504120 (1.170404) | 2.284831 / 1.541195 (0.743637) | 2.360150 / 1.468490 (0.891660) | 0.888021 / 4.584777 (-3.696756) | 5.419840 / 3.745712 (1.674128) | 4.825816 / 5.269862 (-0.444046) | 3.140876 / 4.565676 (-1.424801) | 0.099511 / 0.424275 (-0.324764) | 0.009176 / 0.007607 (0.001569) | 0.735646 / 0.226044 (0.509602) | 7.224026 / 2.268929 (4.955097) | 3.551146 / 55.444624 (-51.893478) | 2.844374 / 6.876477 (-4.032103) | 3.145307 / 2.142072 (1.003235) | 1.077636 / 4.805227 (-3.727591) | 0.217754 / 6.500664 (-6.282910) | 0.081755 / 0.075469 (0.006286) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.670956 / 1.841788 (-0.170831) | 25.524961 / 8.074308 (17.450653) | 23.061596 / 10.191392 (12.870204) | 0.247524 / 0.680424 (-0.432899) | 0.031712 / 0.534201 (-0.502489) | 0.513049 / 0.579283 (-0.066234) | 0.614568 / 0.434364 (0.180204) | 0.574669 / 0.540337 (0.034331) | 0.816621 / 1.386936 (-0.570315) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009384 / 0.011353 (-0.001969) | 0.004959 / 0.011008 (-0.006049) | 0.084782 / 0.038508 (0.046274) | 0.098086 / 0.023109 (0.074977) | 0.544395 / 0.275898 (0.268497) | 0.585157 / 0.323480 (0.261677) | 0.006507 / 0.007986 (-0.001479) | 0.004151 / 0.004328 (-0.000178) | 0.088596 / 0.004250 (0.084345) | 0.069149 / 0.037052 (0.032097) | 0.533109 / 0.258489 (0.274620) | 0.604117 / 0.293841 (0.310276) | 0.047685 / 0.128546 (-0.080861) | 0.013651 / 0.075646 (-0.061996) | 0.096566 / 0.419271 (-0.322705) | 0.062022 / 0.043533 (0.018489) | 0.561897 / 0.255139 (0.306758) | 0.617636 / 0.283200 (0.334436) | 0.034636 / 0.141683 (-0.107047) | 1.854667 / 1.452155 (0.402512) | 1.908923 / 1.492716 (0.416207) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260633 / 0.018006 (0.242627) | 0.622268 / 0.000490 (0.621778) | 0.002116 / 0.000200 (0.001916) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035161 / 0.037411 (-0.002250) | 0.103707 / 0.014526 (0.089181) | 0.115467 / 0.176557 (-0.061090) | 0.180077 / 0.737135 (-0.557059) | 0.118871 / 0.296338 (-0.177467) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.628481 / 0.215209 (0.413271) | 6.304929 / 2.077655 (4.227275) | 3.027775 / 1.504120 (1.523655) | 2.753880 / 1.541195 (1.212686) | 2.820442 / 1.468490 (1.351952) | 0.851103 / 4.584777 (-3.733674) | 5.427383 / 3.745712 (1.681670) | 7.434310 / 5.269862 (2.164449) | 4.418790 / 4.565676 (-0.146887) | 0.101733 / 0.424275 (-0.322542) | 0.009701 / 0.007607 (0.002094) | 0.763033 / 0.226044 (0.536989) | 7.497927 / 2.268929 (5.228998) | 3.735335 / 55.444624 (-51.709290) | 3.149200 / 6.876477 (-3.727277) | 3.306214 / 2.142072 (1.164141) | 1.085440 / 4.805227 (-3.719787) | 0.207562 / 6.500664 (-6.293102) | 0.078091 / 0.075469 (0.002622) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.820097 / 1.841788 (-0.021691) | 25.525539 / 8.074308 (17.451231) | 21.874219 / 10.191392 (11.682827) | 0.228391 / 0.680424 (-0.452033) | 0.029584 / 0.534201 (-0.504617) | 0.511546 / 0.579283 (-0.067737) | 0.602719 / 0.434364 (0.168355) | 0.581874 / 0.540337 (0.041537) | 0.802861 / 1.386936 (-0.584075) |

</details>
</details>

![](https://cml.dev/watermark.png#6063ea2069c8b5641b983ba2c1d39b60afe7c00a "CML watermark")