Title: Temporarily pin pyarrow < 14.0.0
Author: albertvillanova (MEMBER)
State: closed
Created: 2023-11-02T09:48:58Z

Temporarily pin `pyarrow` < 14.0.0 until permanent solution is found.

Hot fix #6374.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008947 / 0.011353 (-0.002406) | 0.005602 / 0.011008 (-0.005406) | 0.111208 / 0.038508 (0.072700) | 0.082750 / 0.023109 (0.059641) | 0.453277 / 0.275898 (0.177379) | 0.480072 / 0.323480 (0.156592) | 0.005254 / 0.007986 (-0.002731) | 0.005421 / 0.004328 (0.001092) | 0.082899 / 0.004250 (0.078648) | 0.062859 / 0.037052 (0.025807) | 0.466703 / 0.258489 (0.208214) | 0.478241 / 0.293841 (0.184400) | 0.050754 / 0.128546 (-0.077792) | 0.017726 / 0.075646 (-0.057920) | 0.374830 / 0.419271 (-0.044442) | 0.068577 / 0.043533 (0.025044) | 0.453643 / 0.255139 (0.198504) | 0.453736 / 0.283200 (0.170537) | 0.037313 / 0.141683 (-0.104369) | 1.741215 / 1.452155 (0.289060) | 1.862247 / 1.492716 (0.369531) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.314174 / 0.018006 (0.296168) | 0.644439 / 0.000490 (0.643949) | 0.013914 / 0.000200 (0.013715) | 0.000478 / 0.000054 (0.000424) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030462 / 0.037411 (-0.006949) | 0.096789 / 0.014526 (0.082263) | 0.109999 / 0.176557 (-0.066557) | 0.184610 / 0.737135 (-0.552525) | 0.113846 / 0.296338 (-0.182493) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.586508 / 0.215209 (0.371299) | 5.785138 / 2.077655 (3.707484) | 2.578512 / 1.504120 (1.074392) | 2.266981 / 1.541195 (0.725786) | 2.442463 / 1.468490 (0.973973) | 0.880973 / 4.584777 (-3.703804) | 5.410327 / 3.745712 (1.664615) | 4.976842 / 5.269862 (-0.293020) | 3.020535 / 4.565676 (-1.545142) | 0.089640 / 0.424275 (-0.334635) | 0.009126 / 0.007607 (0.001519) | 0.682364 / 0.226044 (0.456319) | 6.840507 / 2.268929 (4.571579) | 3.313314 / 55.444624 (-52.131310) | 2.815313 / 6.876477 (-4.061164) | 2.851787 / 2.142072 (0.709715) | 1.044916 / 4.805227 (-3.760312) | 0.218346 / 6.500664 (-6.282318) | 0.075655 / 0.075469 (0.000186) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.641767 / 1.841788 (-0.200020) | 24.618096 / 8.074308 (16.543788) | 21.557652 / 10.191392 (11.366260) | 0.211622 / 0.680424 (-0.468801) | 0.028775 / 0.534201 (-0.505426) | 0.480469 / 0.579283 (-0.098814) | 0.593311 / 0.434364 (0.158948) | 0.560620 / 0.540337 (0.020283) | 0.827026 / 1.386936 (-0.559910) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009347 / 0.011353 (-0.002006) | 0.005184 / 0.011008 (-0.005824) | 0.078878 / 0.038508 (0.040370) | 0.083067 / 0.023109 (0.059957) | 0.446591 / 0.275898 (0.170693) | 0.512934 / 0.323480 (0.189454) | 0.006614 / 0.007986 (-0.001372) | 0.004477 / 0.004328 (0.000148) | 0.087403 / 0.004250 (0.083153) | 0.060710 / 0.037052 (0.023658) | 0.451811 / 0.258489 (0.193322) | 0.482031 / 0.293841 (0.188190) | 0.051685 / 0.128546 (-0.076862) | 0.013436 / 0.075646 (-0.062210) | 0.109012 / 0.419271 (-0.310259) | 0.059654 / 0.043533 (0.016121) | 0.439041 / 0.255139 (0.183902) | 0.481708 / 0.283200 (0.198508) | 0.037393 / 0.141683 (-0.104290) | 1.761704 / 1.452155 (0.309549) | 1.946711 / 1.492716 (0.453995) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.287981 / 0.018006 (0.269975) | 0.610219 / 0.000490 (0.609729) | 0.006733 / 0.000200 (0.006533) | 0.000128 / 0.000054 (0.000074) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038999 / 0.037411 (0.001588) | 0.100613 / 0.014526 (0.086087) | 0.126445 / 0.176557 (-0.050111) | 0.187596 / 0.737135 (-0.549540) | 0.122130 / 0.296338 (-0.174208) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.647686 / 0.215209 (0.432477) | 6.176079 / 2.077655 (4.098424) | 2.800232 / 1.504120 (1.296112) | 2.434625 / 1.541195 (0.893430) | 2.460646 / 1.468490 (0.992155) | 0.923736 / 4.584777 (-3.661041) | 5.480197 / 3.745712 (1.734485) | 4.849250 / 5.269862 (-0.420612) | 3.031576 / 4.565676 (-1.534101) | 0.102525 / 0.424275 (-0.321750) | 0.008688 / 0.007607 (0.001081) | 0.766097 / 0.226044 (0.540052) | 7.626822 / 2.268929 (5.357893) | 3.719155 / 55.444624 (-51.725469) | 2.967121 / 6.876477 (-3.909356) | 3.182464 / 2.142072 (1.040392) | 1.018315 / 4.805227 (-3.786912) | 0.211300 / 6.500664 (-6.289364) | 0.083055 / 0.075469 (0.007586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.731619 / 1.841788 (-0.110168) | 25.315978 / 8.074308 (17.241669) | 22.736306 / 10.191392 (12.544914) | 0.270330 / 0.680424 (-0.410094) | 0.034790 / 0.534201 (-0.499411) | 0.488675 / 0.579283 (-0.090608) | 0.603426 / 0.434364 (0.169062) | 0.572547 / 0.540337 (0.032210) | 0.825719 / 1.386936 (-0.561217) |

</details>
</details>

![](https://cml.dev/watermark.png#1eaa85a4ad79aa0e411218d61a8894cc14a75fa0 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008992 / 0.011353 (-0.002360) | 0.005086 / 0.011008 (-0.005923) | 0.107400 / 0.038508 (0.068892) | 0.091894 / 0.023109 (0.068785) | 0.382347 / 0.275898 (0.106449) | 0.446581 / 0.323480 (0.123101) | 0.005179 / 0.007986 (-0.002807) | 0.006356 / 0.004328 (0.002028) | 0.084979 / 0.004250 (0.080729) | 0.060647 / 0.037052 (0.023594) | 0.385940 / 0.258489 (0.127451) | 0.444817 / 0.293841 (0.150976) | 0.049484 / 0.128546 (-0.079062) | 0.014173 / 0.075646 (-0.061473) | 0.345704 / 0.419271 (-0.073567) | 0.068082 / 0.043533 (0.024550) | 0.377170 / 0.255139 (0.122031) | 0.411816 / 0.283200 (0.128616) | 0.043049 / 0.141683 (-0.098633) | 1.681499 / 1.452155 (0.229344) | 1.805428 / 1.492716 (0.312712) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.323170 / 0.018006 (0.305164) | 0.693845 / 0.000490 (0.693355) | 0.015499 / 0.000200 (0.015299) | 0.000603 / 0.000054 (0.000548) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031629 / 0.037411 (-0.005783) | 0.093511 / 0.014526 (0.078985) | 0.112400 / 0.176557 (-0.064157) | 0.173731 / 0.737135 (-0.563405) | 0.116013 / 0.296338 (-0.180325) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.576724 / 0.215209 (0.361515) | 5.775055 / 2.077655 (3.697400) | 2.755869 / 1.504120 (1.251749) | 2.430253 / 1.541195 (0.889058) | 2.479629 / 1.468490 (1.011139) | 0.841472 / 4.584777 (-3.743305) | 5.120536 / 3.745712 (1.374824) | 4.813281 / 5.269862 (-0.456581) | 3.054617 / 4.565676 (-1.511059) | 0.091459 / 0.424275 (-0.332816) | 0.009072 / 0.007607 (0.001465) | 0.742674 / 0.226044 (0.516629) | 7.137861 / 2.268929 (4.868933) | 3.497568 / 55.444624 (-51.947056) | 2.814658 / 6.876477 (-4.061819) | 2.934415 / 2.142072 (0.792343) | 0.970855 / 4.805227 (-3.834372) | 0.213366 / 6.500664 (-6.287299) | 0.078763 / 0.075469 (0.003293) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.584716 / 1.841788 (-0.257072) | 24.098173 / 8.074308 (16.023865) | 20.746352 / 10.191392 (10.554960) | 0.215313 / 0.680424 (-0.465111) | 0.029538 / 0.534201 (-0.504663) | 0.448672 / 0.579283 (-0.130611) | 0.580023 / 0.434364 (0.145659) | 0.537867 / 0.540337 (-0.002471) | 0.804622 / 1.386936 (-0.582314) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008965 / 0.011353 (-0.002388) | 0.005544 / 0.011008 (-0.005464) | 0.076806 / 0.038508 (0.038298) | 0.085333 / 0.023109 (0.062224) | 0.509974 / 0.275898 (0.234076) | 0.511548 / 0.323480 (0.188068) | 0.007136 / 0.007986 (-0.000849) | 0.004491 / 0.004328 (0.000163) | 0.086687 / 0.004250 (0.082437) | 0.066539 / 0.037052 (0.029486) | 0.483663 / 0.258489 (0.225174) | 0.529480 / 0.293841 (0.235639) | 0.046296 / 0.128546 (-0.082250) | 0.014736 / 0.075646 (-0.060910) | 0.088261 / 0.419271 (-0.331010) | 0.056753 / 0.043533 (0.013220) | 0.511698 / 0.255139 (0.256559) | 0.497956 / 0.283200 (0.214756) | 0.034753 / 0.141683 (-0.106930) | 1.828354 / 1.452155 (0.376199) | 1.799211 / 1.492716 (0.306494) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.389652 / 0.018006 (0.371645) | 0.602522 / 0.000490 (0.602033) | 0.068363 / 0.000200 (0.068163) | 0.000493 / 0.000054 (0.000439) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036431 / 0.037411 (-0.000980) | 0.102162 / 0.014526 (0.087636) | 0.122466 / 0.176557 (-0.054091) | 0.181001 / 0.737135 (-0.556134) | 0.125743 / 0.296338 (-0.170596) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.583847 / 0.215209 (0.368638) | 5.913008 / 2.077655 (3.835354) | 2.716088 / 1.504120 (1.211968) | 2.328631 / 1.541195 (0.787437) | 2.459953 / 1.468490 (0.991463) | 0.792829 / 4.584777 (-3.791948) | 5.183965 / 3.745712 (1.438253) | 4.508264 / 5.269862 (-0.761598) | 2.855444 / 4.565676 (-1.710232) | 0.090704 / 0.424275 (-0.333571) | 0.009303 / 0.007607 (0.001696) | 0.694303 / 0.226044 (0.468258) | 6.951876 / 2.268929 (4.682947) | 3.418244 / 55.444624 (-52.026381) | 2.799743 / 6.876477 (-4.076734) | 3.043657 / 2.142072 (0.901584) | 0.921537 / 4.805227 (-3.883691) | 0.191774 / 6.500664 (-6.308890) | 0.068602 / 0.075469 (-0.006867) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.624842 / 1.841788 (-0.216946) | 24.570622 / 8.074308 (16.496314) | 21.207566 / 10.191392 (11.016174) | 0.217734 / 0.680424 (-0.462689) | 0.033109 / 0.534201 (-0.501091) | 0.451651 / 0.579283 (-0.127632) | 0.590890 / 0.434364 (0.156526) | 0.546195 / 0.540337 (0.005858) | 0.730298 / 1.386936 (-0.656638) |

</details>
</details>

![](https://cml.dev/watermark.png#f6bdecff73303cf97f279a4e36622faf53133f9c "CML watermark")