Title: Pin dill lower version
Author: albertvillanova (MEMBER)
State: closed
Created: 2023-02-01T09:33:42Z

Pin `dill` lower version compatible with `datasets`.

Related to:
- #5487
- #288

Note that the required `dill._dill` module was introduced in dill-2.8.0, however we have heuristically tested that datasets can only be installed with dill>=3.0.0 (otherwise pip hangs indefinitely while preparing metadata for multiprocess-0.70.7)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008798 / 0.011353 (-0.002554) | 0.005313 / 0.011008 (-0.005695) | 0.099234 / 0.038508 (0.060726) | 0.033935 / 0.023109 (0.010826) | 0.306610 / 0.275898 (0.030712) | 0.373151 / 0.323480 (0.049671) | 0.008305 / 0.007986 (0.000320) | 0.004647 / 0.004328 (0.000319) | 0.079984 / 0.004250 (0.075733) | 0.042546 / 0.037052 (0.005493) | 0.355105 / 0.258489 (0.096616) | 0.332769 / 0.293841 (0.038928) | 0.037708 / 0.128546 (-0.090839) | 0.012141 / 0.075646 (-0.063505) | 0.365338 / 0.419271 (-0.053933) | 0.048875 / 0.043533 (0.005343) | 0.301771 / 0.255139 (0.046632) | 0.323301 / 0.283200 (0.040101) | 0.099116 / 0.141683 (-0.042566) | 1.463948 / 1.452155 (0.011793) | 1.563006 / 1.492716 (0.070290) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219799 / 0.018006 (0.201793) | 0.524126 / 0.000490 (0.523636) | 0.003899 / 0.000200 (0.003699) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028361 / 0.037411 (-0.009050) | 0.111386 / 0.014526 (0.096860) | 0.125749 / 0.176557 (-0.050807) | 0.167026 / 0.737135 (-0.570109) | 0.132082 / 0.296338 (-0.164257) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.385046 / 0.215209 (0.169837) | 3.933129 / 2.077655 (1.855475) | 1.823395 / 1.504120 (0.319276) | 1.646468 / 1.541195 (0.105273) | 1.658835 / 1.468490 (0.190344) | 0.708300 / 4.584777 (-3.876477) | 4.001478 / 3.745712 (0.255766) | 2.221773 / 5.269862 (-3.048089) | 1.597925 / 4.565676 (-2.967751) | 0.088699 / 0.424275 (-0.335577) | 0.013575 / 0.007607 (0.005968) | 0.520577 / 0.226044 (0.294533) | 5.044313 / 2.268929 (2.775385) | 2.239862 / 55.444624 (-53.204763) | 2.060394 / 6.876477 (-4.816083) | 2.060684 / 2.142072 (-0.081389) | 0.844862 / 4.805227 (-3.960365) | 0.190321 / 6.500664 (-6.310343) | 0.071595 / 0.075469 (-0.003875) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.400048 / 1.841788 (-0.441740) | 15.684159 / 8.074308 (7.609851) | 14.369298 / 10.191392 (4.177906) | 0.164874 / 0.680424 (-0.515550) | 0.033219 / 0.534201 (-0.500982) | 0.449176 / 0.579283 (-0.130107) | 0.456560 / 0.434364 (0.022196) | 0.517978 / 0.540337 (-0.022359) | 0.635467 / 1.386936 (-0.751469) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007263 / 0.011353 (-0.004089) | 0.005451 / 0.011008 (-0.005558) | 0.078785 / 0.038508 (0.040277) | 0.032656 / 0.023109 (0.009546) | 0.346384 / 0.275898 (0.070486) | 0.390778 / 0.323480 (0.067299) | 0.005848 / 0.007986 (-0.002137) | 0.004565 / 0.004328 (0.000236) | 0.077903 / 0.004250 (0.073652) | 0.048659 / 0.037052 (0.011606) | 0.368629 / 0.258489 (0.110140) | 0.401632 / 0.293841 (0.107791) | 0.038516 / 0.128546 (-0.090030) | 0.011895 / 0.075646 (-0.063752) | 0.089185 / 0.419271 (-0.330086) | 0.049875 / 0.043533 (0.006342) | 0.344771 / 0.255139 (0.089632) | 0.378237 / 0.283200 (0.095038) | 0.099184 / 0.141683 (-0.042498) | 1.505058 / 1.452155 (0.052903) | 1.555330 / 1.492716 (0.062614) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209132 / 0.018006 (0.191126) | 0.479928 / 0.000490 (0.479438) | 0.005923 / 0.000200 (0.005723) | 0.000113 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029187 / 0.037411 (-0.008224) | 0.117026 / 0.014526 (0.102500) | 0.131834 / 0.176557 (-0.044722) | 0.172797 / 0.737135 (-0.564339) | 0.129098 / 0.296338 (-0.167240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.450214 / 0.215209 (0.235005) | 4.323950 / 2.077655 (2.246295) | 2.210100 / 1.504120 (0.705980) | 2.058733 / 1.541195 (0.517538) | 1.968191 / 1.468490 (0.499701) | 0.694918 / 4.584777 (-3.889859) | 4.176559 / 3.745712 (0.430846) | 2.118211 / 5.269862 (-3.151651) | 1.410652 / 4.565676 (-3.155024) | 0.093606 / 0.424275 (-0.330669) | 0.013729 / 0.007607 (0.006122) | 0.528463 / 0.226044 (0.302418) | 5.311766 / 2.268929 (3.042837) | 2.522981 / 55.444624 (-52.921644) | 2.177191 / 6.876477 (-4.699285) | 2.211448 / 2.142072 (0.069375) | 0.824334 / 4.805227 (-3.980893) | 0.166642 / 6.500664 (-6.334022) | 0.062774 / 0.075469 (-0.012695) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.367573 / 1.841788 (-0.474215) | 15.913637 / 8.074308 (7.839328) | 13.397411 / 10.191392 (3.206019) | 0.162599 / 0.680424 (-0.517825) | 0.020325 / 0.534201 (-0.513876) | 0.438745 / 0.579283 (-0.140538) | 0.449892 / 0.434364 (0.015528) | 0.556226 / 0.540337 (0.015888) | 0.672661 / 1.386936 (-0.714275) |

</details>
</details>

![](https://cml.dev/watermark.png#5f810b7011a8a4ab077a1847c024d2d9e267b065 "CML watermark")