Title: Improve `Dataset.from_list` docstring
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-07-24T12:36:38Z


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008340 / 0.011353 (-0.003013) | 0.005053 / 0.011008 (-0.005955) | 0.103294 / 0.038508 (0.064786) | 0.069417 / 0.023109 (0.046308) | 0.436922 / 0.275898 (0.161024) | 0.461348 / 0.323480 (0.137868) | 0.006030 / 0.007986 (-0.001955) | 0.003727 / 0.004328 (-0.000601) | 0.076384 / 0.004250 (0.072134) | 0.056742 / 0.037052 (0.019689) | 0.439996 / 0.258489 (0.181507) | 0.469417 / 0.293841 (0.175577) | 0.044343 / 0.128546 (-0.084203) | 0.012634 / 0.075646 (-0.063013) | 0.359746 / 0.419271 (-0.059525) | 0.064842 / 0.043533 (0.021309) | 0.425960 / 0.255139 (0.170821) | 0.458568 / 0.283200 (0.175368) | 0.039802 / 0.141683 (-0.101881) | 1.687320 / 1.452155 (0.235165) | 1.806212 / 1.492716 (0.313496) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.255484 / 0.018006 (0.237478) | 0.563039 / 0.000490 (0.562549) | 0.000445 / 0.000200 (0.000245) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027511 / 0.037411 (-0.009900) | 0.089185 / 0.014526 (0.074659) | 0.098397 / 0.176557 (-0.078160) | 0.163897 / 0.737135 (-0.573238) | 0.099905 / 0.296338 (-0.196434) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.612737 / 0.215209 (0.397528) | 6.209948 / 2.077655 (4.132294) | 2.756060 / 1.504120 (1.251940) | 2.402115 / 1.541195 (0.860920) | 2.422665 / 1.468490 (0.954175) | 0.834799 / 4.584777 (-3.749977) | 5.251699 / 3.745712 (1.505986) | 5.554141 / 5.269862 (0.284280) | 3.254699 / 4.565676 (-1.310977) | 0.095697 / 0.424275 (-0.328578) | 0.009406 / 0.007607 (0.001799) | 0.729025 / 0.226044 (0.502980) | 7.195521 / 2.268929 (4.926593) | 3.360264 / 55.444624 (-52.084361) | 2.696764 / 6.876477 (-4.179713) | 2.702796 / 2.142072 (0.560724) | 0.974420 / 4.805227 (-3.830808) | 0.195215 / 6.500664 (-6.305450) | 0.069754 / 0.075469 (-0.005715) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.553458 / 1.841788 (-0.288330) | 21.972436 / 8.074308 (13.898128) | 20.027392 / 10.191392 (9.836000) | 0.216950 / 0.680424 (-0.463474) | 0.032196 / 0.534201 (-0.502005) | 0.449884 / 0.579283 (-0.129399) | 0.586213 / 0.434364 (0.151849) | 0.537227 / 0.540337 (-0.003111) | 0.751022 / 1.386936 (-0.635914) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007859 / 0.011353 (-0.003493) | 0.004762 / 0.011008 (-0.006246) | 0.086023 / 0.038508 (0.047515) | 0.069218 / 0.023109 (0.046109) | 0.449312 / 0.275898 (0.173414) | 0.481687 / 0.323480 (0.158207) | 0.006318 / 0.007986 (-0.001668) | 0.004063 / 0.004328 (-0.000266) | 0.076917 / 0.004250 (0.072667) | 0.058034 / 0.037052 (0.020981) | 0.474265 / 0.258489 (0.215775) | 0.497736 / 0.293841 (0.203895) | 0.044587 / 0.128546 (-0.083959) | 0.013880 / 0.075646 (-0.061766) | 0.089233 / 0.419271 (-0.330038) | 0.058760 / 0.043533 (0.015227) | 0.439515 / 0.255139 (0.184376) | 0.473246 / 0.283200 (0.190047) | 0.042968 / 0.141683 (-0.098715) | 1.802647 / 1.452155 (0.350493) | 1.778563 / 1.492716 (0.285847) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.343741 / 0.018006 (0.325735) | 0.567409 / 0.000490 (0.566919) | 0.029727 / 0.000200 (0.029527) | 0.000147 / 0.000054 (0.000092) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031021 / 0.037411 (-0.006390) | 0.096659 / 0.014526 (0.082133) | 0.103341 / 0.176557 (-0.073215) | 0.169893 / 0.737135 (-0.567242) | 0.103280 / 0.296338 (-0.193058) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.584724 / 0.215209 (0.369515) | 5.792596 / 2.077655 (3.714941) | 2.683133 / 1.504120 (1.179013) | 2.367837 / 1.541195 (0.826643) | 2.378567 / 1.468490 (0.910076) | 0.803427 / 4.584777 (-3.781350) | 5.179017 / 3.745712 (1.433305) | 4.446323 / 5.269862 (-0.823538) | 2.771731 / 4.565676 (-1.793945) | 0.100943 / 0.424275 (-0.323332) | 0.009875 / 0.007607 (0.002268) | 0.725260 / 0.226044 (0.499216) | 7.149728 / 2.268929 (4.880800) | 3.646438 / 55.444624 (-51.798187) | 2.793858 / 6.876477 (-4.082618) | 2.971966 / 2.142072 (0.829894) | 0.998147 / 4.805227 (-3.807080) | 0.198004 / 6.500664 (-6.302660) | 0.072581 / 0.075469 (-0.002888) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.696737 / 1.841788 (-0.145051) | 22.615193 / 8.074308 (14.540884) | 20.272421 / 10.191392 (10.081029) | 0.237459 / 0.680424 (-0.442965) | 0.034774 / 0.534201 (-0.499427) | 0.484649 / 0.579283 (-0.094634) | 0.590263 / 0.434364 (0.155899) | 0.547833 / 0.540337 (0.007495) | 0.762109 / 1.386936 (-0.624827) |

</details>
</details>

![](https://cml.dev/watermark.png#4bc3628b5a8f71ad7cfc014d8ba5e798f26becb7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011183 / 0.011353 (-0.000170) | 0.005267 / 0.011008 (-0.005741) | 0.108506 / 0.038508 (0.069997) | 0.083541 / 0.023109 (0.060431) | 0.452189 / 0.275898 (0.176291) | 0.496229 / 0.323480 (0.172749) | 0.004951 / 0.007986 (-0.003035) | 0.004452 / 0.004328 (0.000124) | 0.085133 / 0.004250 (0.080883) | 0.061291 / 0.037052 (0.024239) | 0.450453 / 0.258489 (0.191964) | 0.506456 / 0.293841 (0.212616) | 0.049784 / 0.128546 (-0.078762) | 0.014738 / 0.075646 (-0.060908) | 0.372603 / 0.419271 (-0.046669) | 0.065223 / 0.043533 (0.021690) | 0.467872 / 0.255139 (0.212733) | 0.500062 / 0.283200 (0.216862) | 0.040911 / 0.141683 (-0.100772) | 1.852970 / 1.452155 (0.400816) | 2.016996 / 1.492716 (0.524280) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.262620 / 0.018006 (0.244614) | 0.593925 / 0.000490 (0.593435) | 0.000413 / 0.000200 (0.000213) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035713 / 0.037411 (-0.001698) | 0.111403 / 0.014526 (0.096878) | 0.117259 / 0.176557 (-0.059298) | 0.201545 / 0.737135 (-0.535590) | 0.133111 / 0.296338 (-0.163228) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.597318 / 0.215209 (0.382109) | 5.882691 / 2.077655 (3.805036) | 2.572203 / 1.504120 (1.068083) | 2.248016 / 1.541195 (0.706821) | 2.359103 / 1.468490 (0.890613) | 0.852023 / 4.584777 (-3.732754) | 5.270831 / 3.745712 (1.525119) | 4.712915 / 5.269862 (-0.556947) | 3.124295 / 4.565676 (-1.441381) | 0.092045 / 0.424275 (-0.332230) | 0.007834 / 0.007607 (0.000227) | 0.695711 / 0.226044 (0.469666) | 7.011760 / 2.268929 (4.742831) | 3.333300 / 55.444624 (-52.111325) | 2.745889 / 6.876477 (-4.130587) | 3.153458 / 2.142072 (1.011385) | 1.011089 / 4.805227 (-3.794139) | 0.207467 / 6.500664 (-6.293197) | 0.079802 / 0.075469 (0.004333) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.703784 / 1.841788 (-0.138003) | 24.414340 / 8.074308 (16.340032) | 22.534528 / 10.191392 (12.343136) | 0.276129 / 0.680424 (-0.404295) | 0.027954 / 0.534201 (-0.506247) | 0.484261 / 0.579283 (-0.095022) | 0.605316 / 0.434364 (0.170952) | 0.557219 / 0.540337 (0.016882) | 0.802209 / 1.386936 (-0.584727) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009109 / 0.011353 (-0.002244) | 0.005376 / 0.011008 (-0.005632) | 0.085141 / 0.038508 (0.046633) | 0.100560 / 0.023109 (0.077450) | 0.482673 / 0.275898 (0.206775) | 0.551582 / 0.323480 (0.228103) | 0.006756 / 0.007986 (-0.001229) | 0.004171 / 0.004328 (-0.000158) | 0.084184 / 0.004250 (0.079933) | 0.069283 / 0.037052 (0.032230) | 0.517722 / 0.258489 (0.259233) | 0.542641 / 0.293841 (0.248801) | 0.047790 / 0.128546 (-0.080756) | 0.014063 / 0.075646 (-0.061583) | 0.110591 / 0.419271 (-0.308680) | 0.064373 / 0.043533 (0.020840) | 0.496636 / 0.255139 (0.241497) | 0.551906 / 0.283200 (0.268707) | 0.046187 / 0.141683 (-0.095496) | 1.864836 / 1.452155 (0.412681) | 1.923765 / 1.492716 (0.431049) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.286558 / 0.018006 (0.268552) | 0.610353 / 0.000490 (0.609863) | 0.012647 / 0.000200 (0.012447) | 0.000162 / 0.000054 (0.000107) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037099 / 0.037411 (-0.000313) | 0.108608 / 0.014526 (0.094082) | 0.120386 / 0.176557 (-0.056170) | 0.183450 / 0.737135 (-0.553686) | 0.124860 / 0.296338 (-0.171479) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.629006 / 0.215209 (0.413797) | 6.309206 / 2.077655 (4.231551) | 2.878558 / 1.504120 (1.374438) | 2.616093 / 1.541195 (1.074898) | 2.668096 / 1.468490 (1.199606) | 0.865732 / 4.584777 (-3.719045) | 5.312433 / 3.745712 (1.566721) | 4.799352 / 5.269862 (-0.470509) | 3.142207 / 4.565676 (-1.423469) | 0.099591 / 0.424275 (-0.324684) | 0.009159 / 0.007607 (0.001552) | 0.730999 / 0.226044 (0.504954) | 7.486442 / 2.268929 (5.217513) | 3.657699 / 55.444624 (-51.786925) | 3.080094 / 6.876477 (-3.796383) | 3.320976 / 2.142072 (1.178904) | 1.089324 / 4.805227 (-3.715904) | 0.222831 / 6.500664 (-6.277833) | 0.083976 / 0.075469 (0.008507) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.793181 / 1.841788 (-0.048607) | 25.307444 / 8.074308 (17.233136) | 21.321713 / 10.191392 (11.130321) | 0.216326 / 0.680424 (-0.464098) | 0.034298 / 0.534201 (-0.499903) | 0.497173 / 0.579283 (-0.082110) | 0.643550 / 0.434364 (0.209186) | 0.581213 / 0.540337 (0.040876) | 0.830973 / 1.386936 (-0.555963) |

</details>
</details>

![](https://cml.dev/watermark.png#24875bb8494c3a7803182b08c70747b1b1a6bf4d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006886 / 0.011353 (-0.004467) | 0.004267 / 0.011008 (-0.006741) | 0.086182 / 0.038508 (0.047674) | 0.083405 / 0.023109 (0.060296) | 0.313717 / 0.275898 (0.037819) | 0.351476 / 0.323480 (0.027996) | 0.005702 / 0.007986 (-0.002284) | 0.003802 / 0.004328 (-0.000526) | 0.065759 / 0.004250 (0.061508) | 0.060056 / 0.037052 (0.023003) | 0.315871 / 0.258489 (0.057382) | 0.364520 / 0.293841 (0.070679) | 0.032067 / 0.128546 (-0.096479) | 0.008679 / 0.075646 (-0.066967) | 0.294968 / 0.419271 (-0.124303) | 0.054684 / 0.043533 (0.011152) | 0.314124 / 0.255139 (0.058985) | 0.337312 / 0.283200 (0.054113) | 0.025051 / 0.141683 (-0.116632) | 1.505242 / 1.452155 (0.053087) | 1.608263 / 1.492716 (0.115547) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266562 / 0.018006 (0.248556) | 0.579887 / 0.000490 (0.579397) | 0.004161 / 0.000200 (0.003961) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031153 / 0.037411 (-0.006258) | 0.087703 / 0.014526 (0.073177) | 0.103864 / 0.176557 (-0.072693) | 0.159032 / 0.737135 (-0.578104) | 0.102482 / 0.296338 (-0.193857) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405805 / 0.215209 (0.190596) | 4.050669 / 2.077655 (1.973014) | 2.064384 / 1.504120 (0.560264) | 1.892825 / 1.541195 (0.351630) | 2.001083 / 1.468490 (0.532593) | 0.478174 / 4.584777 (-4.106603) | 3.542580 / 3.745712 (-0.203132) | 3.319205 / 5.269862 (-1.950656) | 2.075868 / 4.565676 (-2.489808) | 0.057345 / 0.424275 (-0.366930) | 0.007459 / 0.007607 (-0.000148) | 0.483564 / 0.226044 (0.257520) | 4.827746 / 2.268929 (2.558818) | 2.579541 / 55.444624 (-52.865083) | 2.205125 / 6.876477 (-4.671352) | 2.489206 / 2.142072 (0.347133) | 0.575843 / 4.805227 (-4.229384) | 0.133010 / 6.500664 (-6.367654) | 0.061082 / 0.075469 (-0.014387) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.286059 / 1.841788 (-0.555729) | 20.575173 / 8.074308 (12.500865) | 14.351692 / 10.191392 (4.160300) | 0.150401 / 0.680424 (-0.530022) | 0.018678 / 0.534201 (-0.515523) | 0.397860 / 0.579283 (-0.181423) | 0.419474 / 0.434364 (-0.014890) | 0.474492 / 0.540337 (-0.065846) | 0.659510 / 1.386936 (-0.727426) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006948 / 0.011353 (-0.004405) | 0.004305 / 0.011008 (-0.006703) | 0.064220 / 0.038508 (0.025712) | 0.083251 / 0.023109 (0.060142) | 0.388148 / 0.275898 (0.112250) | 0.417834 / 0.323480 (0.094354) | 0.005762 / 0.007986 (-0.002224) | 0.003803 / 0.004328 (-0.000525) | 0.066365 / 0.004250 (0.062114) | 0.061808 / 0.037052 (0.024756) | 0.390889 / 0.258489 (0.132400) | 0.430619 / 0.293841 (0.136778) | 0.031777 / 0.128546 (-0.096770) | 0.008781 / 0.075646 (-0.066865) | 0.070844 / 0.419271 (-0.348427) | 0.050552 / 0.043533 (0.007019) | 0.378420 / 0.255139 (0.123281) | 0.403273 / 0.283200 (0.120074) | 0.024578 / 0.141683 (-0.117105) | 1.494790 / 1.452155 (0.042636) | 1.549408 / 1.492716 (0.056692) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.302668 / 0.018006 (0.284662) | 0.542235 / 0.000490 (0.541746) | 0.001847 / 0.000200 (0.001647) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031947 / 0.037411 (-0.005465) | 0.092220 / 0.014526 (0.077694) | 0.104525 / 0.176557 (-0.072031) | 0.162000 / 0.737135 (-0.575135) | 0.106795 / 0.296338 (-0.189543) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.412035 / 0.215209 (0.196826) | 4.106527 / 2.077655 (2.028872) | 2.111529 / 1.504120 (0.607409) | 1.953201 / 1.541195 (0.412006) | 2.079258 / 1.468490 (0.610768) | 0.479562 / 4.584777 (-4.105215) | 3.606256 / 3.745712 (-0.139456) | 5.175250 / 5.269862 (-0.094612) | 3.292465 / 4.565676 (-1.273212) | 0.057726 / 0.424275 (-0.366549) | 0.008247 / 0.007607 (0.000640) | 0.486143 / 0.226044 (0.260098) | 4.859051 / 2.268929 (2.590123) | 2.675629 / 55.444624 (-52.768995) | 2.267448 / 6.876477 (-4.609029) | 2.567639 / 2.142072 (0.425567) | 0.580822 / 4.805227 (-4.224406) | 0.134942 / 6.500664 (-6.365722) | 0.063825 / 0.075469 (-0.011644) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.334421 / 1.841788 (-0.507367) | 20.481428 / 8.074308 (12.407120) | 14.227943 / 10.191392 (4.036551) | 0.170711 / 0.680424 (-0.509713) | 0.018212 / 0.534201 (-0.515989) | 0.397212 / 0.579283 (-0.182071) | 0.411934 / 0.434364 (-0.022430) | 0.478019 / 0.540337 (-0.062319) | 0.645434 / 1.386936 (-0.741502) |

</details>
</details>

![](https://cml.dev/watermark.png#ef3d3f10886e23a65cce3bfd939b8ec0d5a5c2c1 "CML watermark")