Title: Temporarily pin pandas < 2.1.0
Author: albertvillanova (MEMBER)
State: closed
Created: 2023-08-31T09:45:17Z

Temporarily pin `pandas` < 2.1.0 until permanent solution is found.

Hot fix #6197.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008978 / 0.011353 (-0.002375) | 0.005143 / 0.011008 (-0.005865) | 0.104787 / 0.038508 (0.066279) | 0.077069 / 0.023109 (0.053960) | 0.427703 / 0.275898 (0.151805) | 0.469865 / 0.323480 (0.146386) | 0.004618 / 0.007986 (-0.003368) | 0.004074 / 0.004328 (-0.000255) | 0.088656 / 0.004250 (0.084405) | 0.059798 / 0.037052 (0.022746) | 0.465906 / 0.258489 (0.207417) | 0.510281 / 0.293841 (0.216440) | 0.051192 / 0.128546 (-0.077354) | 0.013623 / 0.075646 (-0.062024) | 0.379339 / 0.419271 (-0.039932) | 0.077393 / 0.043533 (0.033860) | 0.445165 / 0.255139 (0.190026) | 0.473577 / 0.283200 (0.190378) | 0.038125 / 0.141683 (-0.103558) | 1.858635 / 1.452155 (0.406480) | 1.869033 / 1.492716 (0.376316) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209011 / 0.018006 (0.191004) | 0.550978 / 0.000490 (0.550488) | 0.004904 / 0.000200 (0.004704) | 0.000106 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031418 / 0.037411 (-0.005993) | 0.089623 / 0.014526 (0.075098) | 0.103491 / 0.176557 (-0.073066) | 0.178158 / 0.737135 (-0.558978) | 0.108515 / 0.296338 (-0.187824) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.648293 / 0.215209 (0.433084) | 6.332361 / 2.077655 (4.254707) | 2.469076 / 1.504120 (0.964956) | 2.286228 / 1.541195 (0.745033) | 2.257408 / 1.468490 (0.788918) | 0.918027 / 4.584777 (-3.666750) | 5.229539 / 3.745712 (1.483827) | 4.676150 / 5.269862 (-0.593712) | 3.220411 / 4.565676 (-1.345266) | 0.095863 / 0.424275 (-0.328413) | 0.008696 / 0.007607 (0.001089) | 0.722356 / 0.226044 (0.496312) | 7.796690 / 2.268929 (5.527762) | 3.715044 / 55.444624 (-51.729581) | 2.852696 / 6.876477 (-4.023780) | 2.891838 / 2.142072 (0.749766) | 1.195536 / 4.805227 (-3.609691) | 0.246908 / 6.500664 (-6.253756) | 0.079454 / 0.075469 (0.003984) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.652740 / 1.841788 (-0.189047) | 23.791791 / 8.074308 (15.717482) | 22.778999 / 10.191392 (12.587607) | 0.253878 / 0.680424 (-0.426546) | 0.031367 / 0.534201 (-0.502834) | 0.509460 / 0.579283 (-0.069823) | 0.603085 / 0.434364 (0.168721) | 0.603890 / 0.540337 (0.063553) | 0.826606 / 1.386936 (-0.560330) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010407 / 0.011353 (-0.000946) | 0.004751 / 0.011008 (-0.006257) | 0.086761 / 0.038508 (0.048253) | 0.087281 / 0.023109 (0.064172) | 0.498409 / 0.275898 (0.222511) | 0.560727 / 0.323480 (0.237247) | 0.006563 / 0.007986 (-0.001423) | 0.004078 / 0.004328 (-0.000251) | 0.086383 / 0.004250 (0.082133) | 0.065915 / 0.037052 (0.028862) | 0.521871 / 0.258489 (0.263382) | 0.582281 / 0.293841 (0.288440) | 0.057189 / 0.128546 (-0.071357) | 0.015514 / 0.075646 (-0.060133) | 0.102574 / 0.419271 (-0.316697) | 0.069155 / 0.043533 (0.025622) | 0.525000 / 0.255139 (0.269861) | 0.557968 / 0.283200 (0.274769) | 0.036934 / 0.141683 (-0.104749) | 1.919335 / 1.452155 (0.467181) | 1.870948 / 1.492716 (0.378231) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.241932 / 0.018006 (0.223926) | 0.560136 / 0.000490 (0.559646) | 0.006438 / 0.000200 (0.006238) | 0.000108 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036192 / 0.037411 (-0.001220) | 0.106829 / 0.014526 (0.092303) | 0.128667 / 0.176557 (-0.047890) | 0.200514 / 0.737135 (-0.536621) | 0.127542 / 0.296338 (-0.168797) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.754556 / 0.215209 (0.539347) | 7.237324 / 2.077655 (5.159670) | 3.267424 / 1.504120 (1.763304) | 2.789601 / 1.541195 (1.248407) | 2.875728 / 1.468490 (1.407238) | 0.894274 / 4.584777 (-3.690503) | 5.394556 / 3.745712 (1.648844) | 4.818523 / 5.269862 (-0.451338) | 2.965827 / 4.565676 (-1.599850) | 0.101967 / 0.424275 (-0.322308) | 0.008506 / 0.007607 (0.000899) | 0.803476 / 0.226044 (0.577432) | 8.614426 / 2.268929 (6.345497) | 4.169113 / 55.444624 (-51.275511) | 3.346346 / 6.876477 (-3.530130) | 3.418206 / 2.142072 (1.276134) | 1.111718 / 4.805227 (-3.693509) | 0.211302 / 6.500664 (-6.289362) | 0.072524 / 0.075469 (-0.002945) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.792705 / 1.841788 (-0.049083) | 24.442484 / 8.074308 (16.368176) | 23.375008 / 10.191392 (13.183616) | 0.227946 / 0.680424 (-0.452478) | 0.034376 / 0.534201 (-0.499825) | 0.489260 / 0.579283 (-0.090023) | 0.563220 / 0.434364 (0.128856) | 0.617405 / 0.540337 (0.077068) | 0.850577 / 1.386936 (-0.536359) |

</details>
</details>

![](https://cml.dev/watermark.png#b4413ea1eaca5023ace1e62ddf1070de2d41b4f6 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006594 / 0.011353 (-0.004759) | 0.004366 / 0.011008 (-0.006642) | 0.084241 / 0.038508 (0.045733) | 0.071876 / 0.023109 (0.048767) | 0.321604 / 0.275898 (0.045706) | 0.343501 / 0.323480 (0.020021) | 0.004069 / 0.007986 (-0.003917) | 0.003311 / 0.004328 (-0.001017) | 0.065079 / 0.004250 (0.060829) | 0.053754 / 0.037052 (0.016702) | 0.326199 / 0.258489 (0.067710) | 0.356552 / 0.293841 (0.062711) | 0.031568 / 0.128546 (-0.096979) | 0.008581 / 0.075646 (-0.067065) | 0.289170 / 0.419271 (-0.130101) | 0.053097 / 0.043533 (0.009564) | 0.309678 / 0.255139 (0.054539) | 0.345717 / 0.283200 (0.062517) | 0.024144 / 0.141683 (-0.117539) | 1.497351 / 1.452155 (0.045196) | 1.584691 / 1.492716 (0.091975) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206357 / 0.018006 (0.188351) | 0.459611 / 0.000490 (0.459121) | 0.002586 / 0.000200 (0.002386) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027459 / 0.037411 (-0.009952) | 0.082197 / 0.014526 (0.067671) | 0.095004 / 0.176557 (-0.081553) | 0.151063 / 0.737135 (-0.586072) | 0.095107 / 0.296338 (-0.201231) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384363 / 0.215209 (0.169154) | 3.836187 / 2.077655 (1.758533) | 1.898312 / 1.504120 (0.394192) | 1.727310 / 1.541195 (0.186115) | 1.803579 / 1.468490 (0.335089) | 0.485946 / 4.584777 (-4.098831) | 3.619134 / 3.745712 (-0.126578) | 3.255274 / 5.269862 (-2.014588) | 2.004603 / 4.565676 (-2.561074) | 0.057107 / 0.424275 (-0.367168) | 0.007601 / 0.007607 (-0.000006) | 0.456545 / 0.226044 (0.230500) | 4.556857 / 2.268929 (2.287929) | 2.379954 / 55.444624 (-53.064671) | 2.045874 / 6.876477 (-4.830603) | 2.203090 / 2.142072 (0.061018) | 0.585400 / 4.805227 (-4.219827) | 0.133018 / 6.500664 (-6.367646) | 0.059457 / 0.075469 (-0.016012) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.292581 / 1.841788 (-0.549207) | 19.360057 / 8.074308 (11.285749) | 14.105359 / 10.191392 (3.913967) | 0.166028 / 0.680424 (-0.514396) | 0.018243 / 0.534201 (-0.515958) | 0.392026 / 0.579283 (-0.187257) | 0.412735 / 0.434364 (-0.021629) | 0.459791 / 0.540337 (-0.080547) | 0.624539 / 1.386936 (-0.762397) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006677 / 0.011353 (-0.004676) | 0.003897 / 0.011008 (-0.007111) | 0.064139 / 0.038508 (0.025631) | 0.071346 / 0.023109 (0.048237) | 0.431180 / 0.275898 (0.155282) | 0.470870 / 0.323480 (0.147390) | 0.005562 / 0.007986 (-0.002423) | 0.003405 / 0.004328 (-0.000924) | 0.064532 / 0.004250 (0.060282) | 0.055317 / 0.037052 (0.018265) | 0.434667 / 0.258489 (0.176178) | 0.475765 / 0.293841 (0.181924) | 0.032392 / 0.128546 (-0.096154) | 0.008418 / 0.075646 (-0.067228) | 0.071069 / 0.419271 (-0.348203) | 0.047963 / 0.043533 (0.004430) | 0.440225 / 0.255139 (0.185086) | 0.454860 / 0.283200 (0.171661) | 0.022653 / 0.141683 (-0.119029) | 1.489444 / 1.452155 (0.037289) | 1.556913 / 1.492716 (0.064196) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226733 / 0.018006 (0.208727) | 0.452005 / 0.000490 (0.451516) | 0.004715 / 0.000200 (0.004515) | 0.000099 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032042 / 0.037411 (-0.005369) | 0.091226 / 0.014526 (0.076700) | 0.103639 / 0.176557 (-0.072917) | 0.157772 / 0.737135 (-0.579363) | 0.105466 / 0.296338 (-0.190872) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439751 / 0.215209 (0.224542) | 4.357102 / 2.077655 (2.279448) | 2.362857 / 1.504120 (0.858737) | 2.180559 / 1.541195 (0.639364) | 2.279601 / 1.468490 (0.811111) | 0.495161 / 4.584777 (-4.089616) | 3.729199 / 3.745712 (-0.016513) | 3.334839 / 5.269862 (-1.935023) | 2.099315 / 4.565676 (-2.466362) | 0.058178 / 0.424275 (-0.366097) | 0.007303 / 0.007607 (-0.000304) | 0.506968 / 0.226044 (0.280924) | 5.078600 / 2.268929 (2.809671) | 2.846420 / 55.444624 (-52.598204) | 2.480644 / 6.876477 (-4.395833) | 2.693204 / 2.142072 (0.551132) | 0.590118 / 4.805227 (-4.215109) | 0.132900 / 6.500664 (-6.367764) | 0.060053 / 0.075469 (-0.015416) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356713 / 1.841788 (-0.485075) | 20.380573 / 8.074308 (12.306265) | 15.066507 / 10.191392 (4.875115) | 0.180655 / 0.680424 (-0.499769) | 0.020954 / 0.534201 (-0.513247) | 0.399638 / 0.579283 (-0.179645) | 0.420694 / 0.434364 (-0.013670) | 0.476124 / 0.540337 (-0.064213) | 0.647192 / 1.386936 (-0.739744) |

</details>
</details>

![](https://cml.dev/watermark.png#0f8c58002481568eb1aa4f6f86c4509cf476800a "CML watermark")