Title: Use object detection images from `huggingface/documentation-images`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-08-24T16:16:09Z


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005847 / 0.011353 (-0.005506) | 0.003488 / 0.011008 (-0.007521) | 0.079545 / 0.038508 (0.041037) | 0.055114 / 0.023109 (0.032005) | 0.312694 / 0.275898 (0.036796) | 0.338808 / 0.323480 (0.015329) | 0.004573 / 0.007986 (-0.003413) | 0.002818 / 0.004328 (-0.001510) | 0.062102 / 0.004250 (0.057852) | 0.044072 / 0.037052 (0.007019) | 0.317682 / 0.258489 (0.059192) | 0.354139 / 0.293841 (0.060298) | 0.026905 / 0.128546 (-0.101641) | 0.007990 / 0.075646 (-0.067656) | 0.260071 / 0.419271 (-0.159201) | 0.043658 / 0.043533 (0.000125) | 0.313828 / 0.255139 (0.058689) | 0.339678 / 0.283200 (0.056478) | 0.020076 / 0.141683 (-0.121607) | 1.446321 / 1.452155 (-0.005834) | 1.527046 / 1.492716 (0.034330) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.197801 / 0.018006 (0.179795) | 0.432874 / 0.000490 (0.432385) | 0.004093 / 0.000200 (0.003893) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023505 / 0.037411 (-0.013906) | 0.072377 / 0.014526 (0.057852) | 0.081058 / 0.176557 (-0.095498) | 0.141628 / 0.737135 (-0.595507) | 0.081622 / 0.296338 (-0.214716) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395005 / 0.215209 (0.179795) | 3.949006 / 2.077655 (1.871352) | 1.934028 / 1.504120 (0.429908) | 1.756065 / 1.541195 (0.214871) | 1.778719 / 1.468490 (0.310229) | 0.501279 / 4.584777 (-4.083498) | 3.032120 / 3.745712 (-0.713592) | 2.859751 / 5.269862 (-2.410110) | 1.885924 / 4.565676 (-2.679753) | 0.057236 / 0.424275 (-0.367039) | 0.006704 / 0.007607 (-0.000903) | 0.465794 / 0.226044 (0.239750) | 4.648622 / 2.268929 (2.379694) | 2.345649 / 55.444624 (-53.098975) | 1.981122 / 6.876477 (-4.895355) | 2.148235 / 2.142072 (0.006163) | 0.591466 / 4.805227 (-4.213761) | 0.125262 / 6.500664 (-6.375402) | 0.061305 / 0.075469 (-0.014164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243932 / 1.841788 (-0.597856) | 17.912110 / 8.074308 (9.837802) | 13.662097 / 10.191392 (3.470705) | 0.148051 / 0.680424 (-0.532373) | 0.016778 / 0.534201 (-0.517423) | 0.340342 / 0.579283 (-0.238941) | 0.351720 / 0.434364 (-0.082644) | 0.377837 / 0.540337 (-0.162501) | 0.521163 / 1.386936 (-0.865774) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006011 / 0.011353 (-0.005342) | 0.003549 / 0.011008 (-0.007459) | 0.063579 / 0.038508 (0.025071) | 0.056196 / 0.023109 (0.033087) | 0.448879 / 0.275898 (0.172981) | 0.491542 / 0.323480 (0.168062) | 0.004597 / 0.007986 (-0.003389) | 0.002790 / 0.004328 (-0.001539) | 0.063257 / 0.004250 (0.059006) | 0.045653 / 0.037052 (0.008600) | 0.459714 / 0.258489 (0.201225) | 0.491371 / 0.293841 (0.197530) | 0.028124 / 0.128546 (-0.100422) | 0.008016 / 0.075646 (-0.067630) | 0.069418 / 0.419271 (-0.349853) | 0.040393 / 0.043533 (-0.003140) | 0.450978 / 0.255139 (0.195839) | 0.472075 / 0.283200 (0.188875) | 0.020006 / 0.141683 (-0.121677) | 1.451946 / 1.452155 (-0.000209) | 1.513557 / 1.492716 (0.020840) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225416 / 0.018006 (0.207410) | 0.412287 / 0.000490 (0.411797) | 0.004075 / 0.000200 (0.003875) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025949 / 0.037411 (-0.011463) | 0.080633 / 0.014526 (0.066108) | 0.089960 / 0.176557 (-0.086597) | 0.144530 / 0.737135 (-0.592606) | 0.091427 / 0.296338 (-0.204911) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462311 / 0.215209 (0.247102) | 4.605063 / 2.077655 (2.527408) | 2.541083 / 1.504120 (1.036963) | 2.356341 / 1.541195 (0.815147) | 2.389824 / 1.468490 (0.921334) | 0.507397 / 4.584777 (-4.077380) | 3.079023 / 3.745712 (-0.666689) | 2.792025 / 5.269862 (-2.477837) | 1.846931 / 4.565676 (-2.718746) | 0.058422 / 0.424275 (-0.365853) | 0.006409 / 0.007607 (-0.001199) | 0.530648 / 0.226044 (0.304604) | 5.321030 / 2.268929 (3.052101) | 2.978335 / 55.444624 (-52.466289) | 2.641188 / 6.876477 (-4.235288) | 2.780450 / 2.142072 (0.638378) | 0.593864 / 4.805227 (-4.211363) | 0.125394 / 6.500664 (-6.375270) | 0.061432 / 0.075469 (-0.014037) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.337142 / 1.841788 (-0.504646) | 18.841575 / 8.074308 (10.767267) | 14.678622 / 10.191392 (4.487230) | 0.144491 / 0.680424 (-0.535933) | 0.018145 / 0.534201 (-0.516056) | 0.339376 / 0.579283 (-0.239907) | 0.339129 / 0.434364 (-0.095235) | 0.394842 / 0.540337 (-0.145495) | 0.547924 / 1.386936 (-0.839012) |

</details>
</details>

![](https://cml.dev/watermark.png#57af0ab30796df59d28bf933e756ffbe5f34db1e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006478 / 0.011353 (-0.004875) | 0.003845 / 0.011008 (-0.007163) | 0.084179 / 0.038508 (0.045671) | 0.071327 / 0.023109 (0.048217) | 0.315206 / 0.275898 (0.039308) | 0.353477 / 0.323480 (0.029997) | 0.005267 / 0.007986 (-0.002719) | 0.003282 / 0.004328 (-0.001046) | 0.064062 / 0.004250 (0.059811) | 0.051940 / 0.037052 (0.014888) | 0.332004 / 0.258489 (0.073515) | 0.363199 / 0.293841 (0.069358) | 0.030546 / 0.128546 (-0.098000) | 0.008453 / 0.075646 (-0.067193) | 0.287636 / 0.419271 (-0.131636) | 0.051999 / 0.043533 (0.008466) | 0.325220 / 0.255139 (0.070081) | 0.355324 / 0.283200 (0.072125) | 0.023417 / 0.141683 (-0.118266) | 1.473370 / 1.452155 (0.021215) | 1.596903 / 1.492716 (0.104186) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212645 / 0.018006 (0.194638) | 0.463766 / 0.000490 (0.463276) | 0.002834 / 0.000200 (0.002634) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028424 / 0.037411 (-0.008987) | 0.082188 / 0.014526 (0.067662) | 0.777186 / 0.176557 (0.600629) | 0.218290 / 0.737135 (-0.518845) | 0.099098 / 0.296338 (-0.197240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.387138 / 0.215209 (0.171929) | 3.845655 / 2.077655 (1.768000) | 1.929812 / 1.504120 (0.425692) | 1.718263 / 1.541195 (0.177069) | 1.760933 / 1.468490 (0.292443) | 0.475171 / 4.584777 (-4.109606) | 3.523366 / 3.745712 (-0.222346) | 3.167322 / 5.269862 (-2.102540) | 1.975164 / 4.565676 (-2.590513) | 0.056106 / 0.424275 (-0.368169) | 0.007448 / 0.007607 (-0.000159) | 0.459824 / 0.226044 (0.233779) | 4.590566 / 2.268929 (2.321638) | 2.377968 / 55.444624 (-53.066656) | 2.034052 / 6.876477 (-4.842425) | 2.224976 / 2.142072 (0.082904) | 0.575901 / 4.805227 (-4.229326) | 0.131546 / 6.500664 (-6.369118) | 0.059266 / 0.075469 (-0.016203) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254783 / 1.841788 (-0.587005) | 19.497795 / 8.074308 (11.423487) | 13.937672 / 10.191392 (3.746280) | 0.164092 / 0.680424 (-0.516332) | 0.017915 / 0.534201 (-0.516286) | 0.391430 / 0.579283 (-0.187853) | 0.403681 / 0.434364 (-0.030683) | 0.457711 / 0.540337 (-0.082626) | 0.620395 / 1.386936 (-0.766541) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006793 / 0.011353 (-0.004560) | 0.004101 / 0.011008 (-0.006907) | 0.064780 / 0.038508 (0.026272) | 0.071087 / 0.023109 (0.047977) | 0.401963 / 0.275898 (0.126065) | 0.433085 / 0.323480 (0.109605) | 0.005348 / 0.007986 (-0.002638) | 0.003289 / 0.004328 (-0.001039) | 0.065209 / 0.004250 (0.060958) | 0.054202 / 0.037052 (0.017150) | 0.405629 / 0.258489 (0.147140) | 0.440326 / 0.293841 (0.146485) | 0.032283 / 0.128546 (-0.096263) | 0.008510 / 0.075646 (-0.067137) | 0.071144 / 0.419271 (-0.348127) | 0.047414 / 0.043533 (0.003881) | 0.402065 / 0.255139 (0.146926) | 0.421217 / 0.283200 (0.138017) | 0.021924 / 0.141683 (-0.119759) | 1.490067 / 1.452155 (0.037913) | 1.539134 / 1.492716 (0.046417) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280072 / 0.018006 (0.262066) | 0.456130 / 0.000490 (0.455641) | 0.020926 / 0.000200 (0.020726) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032040 / 0.037411 (-0.005371) | 0.092343 / 0.014526 (0.077817) | 0.104866 / 0.176557 (-0.071690) | 0.156631 / 0.737135 (-0.580505) | 0.107203 / 0.296338 (-0.189136) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426268 / 0.215209 (0.211059) | 4.255539 / 2.077655 (2.177884) | 2.285077 / 1.504120 (0.780957) | 2.114277 / 1.541195 (0.573083) | 2.159242 / 1.468490 (0.690752) | 0.489421 / 4.584777 (-4.095356) | 3.630797 / 3.745712 (-0.114915) | 3.205238 / 5.269862 (-2.064624) | 1.985846 / 4.565676 (-2.579830) | 0.057436 / 0.424275 (-0.366839) | 0.007154 / 0.007607 (-0.000454) | 0.507294 / 0.226044 (0.281250) | 5.050105 / 2.268929 (2.781176) | 2.750474 / 55.444624 (-52.694151) | 2.404116 / 6.876477 (-4.472360) | 2.576483 / 2.142072 (0.434411) | 0.584909 / 4.805227 (-4.220318) | 0.130695 / 6.500664 (-6.369969) | 0.059743 / 0.075469 (-0.015726) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.352702 / 1.841788 (-0.489086) | 19.687944 / 8.074308 (11.613636) | 14.991847 / 10.191392 (4.800455) | 0.185164 / 0.680424 (-0.495260) | 0.020314 / 0.534201 (-0.513887) | 0.395162 / 0.579283 (-0.184121) | 0.408917 / 0.434364 (-0.025447) | 0.467049 / 0.540337 (-0.073288) | 0.649209 / 1.386936 (-0.737727) |

</details>
</details>

![](https://cml.dev/watermark.png#885518608ceab83b7ed8ceba7a0b72bc68096026 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006142 / 0.011353 (-0.005211) | 0.003621 / 0.011008 (-0.007387) | 0.079880 / 0.038508 (0.041372) | 0.059283 / 0.023109 (0.036173) | 0.310971 / 0.275898 (0.035072) | 0.351620 / 0.323480 (0.028140) | 0.003453 / 0.007986 (-0.004532) | 0.003785 / 0.004328 (-0.000543) | 0.062395 / 0.004250 (0.058145) | 0.047614 / 0.037052 (0.010562) | 0.312688 / 0.258489 (0.054199) | 0.363762 / 0.293841 (0.069921) | 0.027051 / 0.128546 (-0.101495) | 0.007920 / 0.075646 (-0.067726) | 0.261080 / 0.419271 (-0.158192) | 0.044476 / 0.043533 (0.000943) | 0.312615 / 0.255139 (0.057476) | 0.343672 / 0.283200 (0.060472) | 0.022723 / 0.141683 (-0.118960) | 1.441449 / 1.452155 (-0.010706) | 1.509253 / 1.492716 (0.016536) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.193171 / 0.018006 (0.175165) | 0.434771 / 0.000490 (0.434281) | 0.003114 / 0.000200 (0.002914) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024209 / 0.037411 (-0.013203) | 0.073891 / 0.014526 (0.059365) | 0.083497 / 0.176557 (-0.093060) | 0.144962 / 0.737135 (-0.592173) | 0.084594 / 0.296338 (-0.211745) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.392512 / 0.215209 (0.177303) | 3.912692 / 2.077655 (1.835037) | 1.914010 / 1.504120 (0.409890) | 1.743827 / 1.541195 (0.202632) | 1.829244 / 1.468490 (0.360753) | 0.497740 / 4.584777 (-4.087037) | 2.979222 / 3.745712 (-0.766490) | 2.849786 / 5.269862 (-2.420076) | 1.874411 / 4.565676 (-2.691265) | 0.057270 / 0.424275 (-0.367005) | 0.006673 / 0.007607 (-0.000934) | 0.460724 / 0.226044 (0.234679) | 4.600617 / 2.268929 (2.331689) | 2.333178 / 55.444624 (-53.111446) | 1.999902 / 6.876477 (-4.876575) | 2.170600 / 2.142072 (0.028528) | 0.587716 / 4.805227 (-4.217511) | 0.126374 / 6.500664 (-6.374290) | 0.061926 / 0.075469 (-0.013543) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.229767 / 1.841788 (-0.612021) | 18.494462 / 8.074308 (10.420154) | 13.799801 / 10.191392 (3.608409) | 0.137952 / 0.680424 (-0.542472) | 0.017037 / 0.534201 (-0.517164) | 0.333252 / 0.579283 (-0.246031) | 0.357276 / 0.434364 (-0.077088) | 0.380069 / 0.540337 (-0.160268) | 0.526968 / 1.386936 (-0.859968) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006185 / 0.011353 (-0.005168) | 0.003595 / 0.011008 (-0.007413) | 0.063371 / 0.038508 (0.024863) | 0.060461 / 0.023109 (0.037351) | 0.455016 / 0.275898 (0.179118) | 0.490505 / 0.323480 (0.167026) | 0.004738 / 0.007986 (-0.003247) | 0.002852 / 0.004328 (-0.001477) | 0.064161 / 0.004250 (0.059910) | 0.047411 / 0.037052 (0.010359) | 0.453815 / 0.258489 (0.195326) | 0.485354 / 0.293841 (0.191513) | 0.028358 / 0.128546 (-0.100188) | 0.008101 / 0.075646 (-0.067545) | 0.068399 / 0.419271 (-0.350873) | 0.040928 / 0.043533 (-0.002605) | 0.462263 / 0.255139 (0.207124) | 0.478773 / 0.283200 (0.195574) | 0.019787 / 0.141683 (-0.121896) | 1.475798 / 1.452155 (0.023643) | 1.563890 / 1.492716 (0.071174) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239701 / 0.018006 (0.221695) | 0.417442 / 0.000490 (0.416953) | 0.005895 / 0.000200 (0.005695) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026155 / 0.037411 (-0.011256) | 0.081264 / 0.014526 (0.066738) | 0.089734 / 0.176557 (-0.086822) | 0.143965 / 0.737135 (-0.593171) | 0.092156 / 0.296338 (-0.204182) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456420 / 0.215209 (0.241211) | 4.545675 / 2.077655 (2.468020) | 2.477141 / 1.504120 (0.973022) | 2.295142 / 1.541195 (0.753947) | 2.349525 / 1.468490 (0.881035) | 0.502485 / 4.584777 (-4.082292) | 3.072347 / 3.745712 (-0.673365) | 2.798565 / 5.269862 (-2.471296) | 1.849030 / 4.565676 (-2.716647) | 0.057789 / 0.424275 (-0.366487) | 0.006436 / 0.007607 (-0.001172) | 0.529648 / 0.226044 (0.303604) | 5.285670 / 2.268929 (3.016741) | 2.954964 / 55.444624 (-52.489660) | 2.593161 / 6.876477 (-4.283316) | 2.735254 / 2.142072 (0.593181) | 0.587635 / 4.805227 (-4.217592) | 0.124732 / 6.500664 (-6.375932) | 0.060999 / 0.075469 (-0.014470) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.354957 / 1.841788 (-0.486831) | 18.803998 / 8.074308 (10.729690) | 14.902712 / 10.191392 (4.711320) | 0.146729 / 0.680424 (-0.533695) | 0.017989 / 0.534201 (-0.516212) | 0.333633 / 0.579283 (-0.245650) | 0.347685 / 0.434364 (-0.086679) | 0.386497 / 0.540337 (-0.153840) | 0.590885 / 1.386936 (-0.796051) |

</details>
</details>

![](https://cml.dev/watermark.png#392d8a46f4da066408785281d9b87760f7273254 "CML watermark")