Title: Set dev version
Author: albertvillanova (MEMBER)
State: closed
Created: 2023-11-16T08:12:55Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6428). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004839 / 0.011353 (-0.006514) | 0.002928 / 0.011008 (-0.008080) | 0.061730 / 0.038508 (0.023221) | 0.030523 / 0.023109 (0.007414) | 0.252679 / 0.275898 (-0.023219) | 0.281597 / 0.323480 (-0.041883) | 0.003025 / 0.007986 (-0.004961) | 0.002374 / 0.004328 (-0.001955) | 0.048134 / 0.004250 (0.043884) | 0.045843 / 0.037052 (0.008791) | 0.256274 / 0.258489 (-0.002215) | 0.288704 / 0.293841 (-0.005137) | 0.023486 / 0.128546 (-0.105060) | 0.007186 / 0.075646 (-0.068461) | 0.202519 / 0.419271 (-0.216753) | 0.058192 / 0.043533 (0.014659) | 0.256448 / 0.255139 (0.001309) | 0.279417 / 0.283200 (-0.003783) | 0.019942 / 0.141683 (-0.121740) | 1.100954 / 1.452155 (-0.351201) | 1.168183 / 1.492716 (-0.324533) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091314 / 0.018006 (0.073308) | 0.298614 / 0.000490 (0.298124) | 0.000232 / 0.000200 (0.000032) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018071 / 0.037411 (-0.019340) | 0.062265 / 0.014526 (0.047740) | 0.073228 / 0.176557 (-0.103328) | 0.119163 / 0.737135 (-0.617972) | 0.074717 / 0.296338 (-0.221622) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.273906 / 0.215209 (0.058697) | 2.683995 / 2.077655 (0.606340) | 1.418773 / 1.504120 (-0.085347) | 1.310473 / 1.541195 (-0.230722) | 1.303152 / 1.468490 (-0.165339) | 0.390846 / 4.584777 (-4.193931) | 2.346407 / 3.745712 (-1.399305) | 2.582945 / 5.269862 (-2.686916) | 1.569549 / 4.565676 (-2.996128) | 0.044893 / 0.424275 (-0.379383) | 0.004754 / 0.007607 (-0.002853) | 0.323491 / 0.226044 (0.097447) | 3.229736 / 2.268929 (0.960808) | 1.783551 / 55.444624 (-53.661074) | 1.499685 / 6.876477 (-5.376792) | 1.515826 / 2.142072 (-0.626246) | 0.475768 / 4.805227 (-4.329460) | 0.099579 / 6.500664 (-6.401085) | 0.042709 / 0.075469 (-0.032760) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.926120 / 1.841788 (-0.915667) | 11.597189 / 8.074308 (3.522881) | 10.327055 / 10.191392 (0.135663) | 0.127479 / 0.680424 (-0.552945) | 0.014844 / 0.534201 (-0.519357) | 0.261181 / 0.579283 (-0.318102) | 0.258407 / 0.434364 (-0.175957) | 0.303192 / 0.540337 (-0.237146) | 0.416665 / 1.386936 (-0.970271) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004759 / 0.011353 (-0.006594) | 0.002780 / 0.011008 (-0.008228) | 0.047991 / 0.038508 (0.009483) | 0.052263 / 0.023109 (0.029153) | 0.261228 / 0.275898 (-0.014670) | 0.287779 / 0.323480 (-0.035701) | 0.003961 / 0.007986 (-0.004024) | 0.002357 / 0.004328 (-0.001971) | 0.047755 / 0.004250 (0.043505) | 0.038066 / 0.037052 (0.001014) | 0.269502 / 0.258489 (0.011013) | 0.298348 / 0.293841 (0.004507) | 0.024398 / 0.128546 (-0.104149) | 0.007189 / 0.075646 (-0.068457) | 0.053356 / 0.419271 (-0.365915) | 0.032459 / 0.043533 (-0.011074) | 0.266389 / 0.255139 (0.011250) | 0.305367 / 0.283200 (0.022168) | 0.017629 / 0.141683 (-0.124054) | 1.145789 / 1.452155 (-0.306366) | 1.204778 / 1.492716 (-0.287938) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091347 / 0.018006 (0.073341) | 0.298671 / 0.000490 (0.298181) | 0.000229 / 0.000200 (0.000029) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021374 / 0.037411 (-0.016037) | 0.068869 / 0.014526 (0.054344) | 0.080443 / 0.176557 (-0.096113) | 0.118759 / 0.737135 (-0.618376) | 0.081646 / 0.296338 (-0.214692) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295274 / 0.215209 (0.080065) | 2.889349 / 2.077655 (0.811695) | 1.561020 / 1.504120 (0.056900) | 1.425025 / 1.541195 (-0.116170) | 1.495446 / 1.468490 (0.026956) | 0.403825 / 4.584777 (-4.180952) | 2.404905 / 3.745712 (-1.340807) | 2.590104 / 5.269862 (-2.679758) | 1.570559 / 4.565676 (-2.995118) | 0.046342 / 0.424275 (-0.377933) | 0.004799 / 0.007607 (-0.002809) | 0.349981 / 0.226044 (0.123937) | 3.437341 / 2.268929 (1.168412) | 1.948155 / 55.444624 (-53.496469) | 1.637765 / 6.876477 (-5.238711) | 1.671521 / 2.142072 (-0.470551) | 0.479500 / 4.805227 (-4.325727) | 0.098305 / 6.500664 (-6.402359) | 0.040864 / 0.075469 (-0.034605) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.979986 / 1.841788 (-0.861801) | 12.169722 / 8.074308 (4.095413) | 11.297345 / 10.191392 (1.105953) | 0.129123 / 0.680424 (-0.551301) | 0.015389 / 0.534201 (-0.518812) | 0.270964 / 0.579283 (-0.308319) | 0.269590 / 0.434364 (-0.164774) | 0.310662 / 0.540337 (-0.229675) | 0.406272 / 1.386936 (-0.980664) |

</details>
</details>

![](https://cml.dev/watermark.png#31873f1e9acbe013e6d396d1ed5492db8cd59dd3 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004620 / 0.011353 (-0.006733) | 0.002971 / 0.011008 (-0.008038) | 0.062864 / 0.038508 (0.024355) | 0.028743 / 0.023109 (0.005634) | 0.246729 / 0.275898 (-0.029169) | 0.271165 / 0.323480 (-0.052315) | 0.003930 / 0.007986 (-0.004056) | 0.002422 / 0.004328 (-0.001906) | 0.047430 / 0.004250 (0.043180) | 0.044895 / 0.037052 (0.007843) | 0.249128 / 0.258489 (-0.009361) | 0.283384 / 0.293841 (-0.010457) | 0.023288 / 0.128546 (-0.105259) | 0.007241 / 0.075646 (-0.068405) | 0.207551 / 0.419271 (-0.211720) | 0.055008 / 0.043533 (0.011475) | 0.252781 / 0.255139 (-0.002358) | 0.296924 / 0.283200 (0.013724) | 0.017860 / 0.141683 (-0.123822) | 1.094597 / 1.452155 (-0.357558) | 1.162314 / 1.492716 (-0.330402) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091423 / 0.018006 (0.073417) | 0.302833 / 0.000490 (0.302343) | 0.000242 / 0.000200 (0.000042) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018143 / 0.037411 (-0.019268) | 0.066371 / 0.014526 (0.051845) | 0.072774 / 0.176557 (-0.103783) | 0.119062 / 0.737135 (-0.618073) | 0.102836 / 0.296338 (-0.193502) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280117 / 0.215209 (0.064908) | 2.757955 / 2.077655 (0.680301) | 1.494994 / 1.504120 (-0.009126) | 1.375325 / 1.541195 (-0.165870) | 1.384179 / 1.468490 (-0.084311) | 0.399824 / 4.584777 (-4.184953) | 2.368575 / 3.745712 (-1.377137) | 2.574035 / 5.269862 (-2.695827) | 1.548738 / 4.565676 (-3.016939) | 0.045841 / 0.424275 (-0.378434) | 0.004799 / 0.007607 (-0.002808) | 0.331522 / 0.226044 (0.105478) | 3.324471 / 2.268929 (1.055543) | 1.838637 / 55.444624 (-53.605987) | 1.562854 / 6.876477 (-5.313623) | 1.581736 / 2.142072 (-0.560336) | 0.468832 / 4.805227 (-4.336396) | 0.099309 / 6.500664 (-6.401355) | 0.042078 / 0.075469 (-0.033391) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.928468 / 1.841788 (-0.913320) | 11.331143 / 8.074308 (3.256835) | 10.296213 / 10.191392 (0.104821) | 0.138912 / 0.680424 (-0.541511) | 0.014044 / 0.534201 (-0.520157) | 0.267293 / 0.579283 (-0.311991) | 0.267267 / 0.434364 (-0.167097) | 0.306560 / 0.540337 (-0.233778) | 0.423926 / 1.386936 (-0.963010) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004842 / 0.011353 (-0.006511) | 0.002917 / 0.011008 (-0.008091) | 0.048263 / 0.038508 (0.009755) | 0.051453 / 0.023109 (0.028344) | 0.278330 / 0.275898 (0.002432) | 0.298569 / 0.323480 (-0.024911) | 0.003936 / 0.007986 (-0.004049) | 0.002479 / 0.004328 (-0.001850) | 0.048281 / 0.004250 (0.044031) | 0.038925 / 0.037052 (0.001872) | 0.285258 / 0.258489 (0.026769) | 0.313701 / 0.293841 (0.019860) | 0.024916 / 0.128546 (-0.103630) | 0.007142 / 0.075646 (-0.068504) | 0.053634 / 0.419271 (-0.365638) | 0.032842 / 0.043533 (-0.010690) | 0.279373 / 0.255139 (0.024234) | 0.295844 / 0.283200 (0.012644) | 0.018142 / 0.141683 (-0.123541) | 1.136960 / 1.452155 (-0.315195) | 1.184438 / 1.492716 (-0.308278) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090271 / 0.018006 (0.072264) | 0.299940 / 0.000490 (0.299450) | 0.000234 / 0.000200 (0.000034) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021175 / 0.037411 (-0.016237) | 0.070924 / 0.014526 (0.056398) | 0.080584 / 0.176557 (-0.095972) | 0.119278 / 0.737135 (-0.617857) | 0.082361 / 0.296338 (-0.213977) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298312 / 0.215209 (0.083103) | 2.895361 / 2.077655 (0.817706) | 1.616120 / 1.504120 (0.112001) | 1.484444 / 1.541195 (-0.056750) | 1.541893 / 1.468490 (0.073403) | 0.409968 / 4.584777 (-4.174809) | 2.423639 / 3.745712 (-1.322073) | 2.585122 / 5.269862 (-2.684740) | 1.540343 / 4.565676 (-3.025333) | 0.046604 / 0.424275 (-0.377671) | 0.004742 / 0.007607 (-0.002865) | 0.341659 / 0.226044 (0.115614) | 3.409259 / 2.268929 (1.140330) | 2.007068 / 55.444624 (-53.437556) | 1.681348 / 6.876477 (-5.195129) | 1.719253 / 2.142072 (-0.422819) | 0.482301 / 4.805227 (-4.322926) | 0.099619 / 6.500664 (-6.401045) | 0.041247 / 0.075469 (-0.034222) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971783 / 1.841788 (-0.870004) | 12.208000 / 8.074308 (4.133692) | 10.948230 / 10.191392 (0.756838) | 0.131824 / 0.680424 (-0.548599) | 0.015696 / 0.534201 (-0.518505) | 0.272265 / 0.579283 (-0.307018) | 0.276093 / 0.434364 (-0.158270) | 0.305897 / 0.540337 (-0.234441) | 0.411632 / 1.386936 (-0.975304) |

</details>
</details>

![](https://cml.dev/watermark.png#2bf75fe522c6fedd16d00b4a928f613dee11f73c "CML watermark")