handle future deprecation argument
Opened by winglian (closed)

getting this error:
```
/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/datasets/table.py:1387: FutureWarning: promote has been superseded by mode='default'.
  return cls._concat_blocks(pa_tables_to_concat_vertically, axis=0)
```

Since datasets supports arrow greater than 8.0.0, we need to handle both cases.

[Arrow v14 docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)
[Arrow v13 docs](https://arrow.apache.org/docs/13.0/python/generated/pyarrow.concat_tables.html)

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004368 / 0.011353 (-0.006985) | 0.002613 / 0.011008 (-0.008396) | 0.061365 / 0.038508 (0.022856) | 0.029553 / 0.023109 (0.006444) | 0.240535 / 0.275898 (-0.035363) | 0.280634 / 0.323480 (-0.042845) | 0.002923 / 0.007986 (-0.005063) | 0.003696 / 0.004328 (-0.000632) | 0.049824 / 0.004250 (0.045573) | 0.044935 / 0.037052 (0.007882) | 0.246870 / 0.258489 (-0.011619) | 0.317248 / 0.293841 (0.023407) | 0.022717 / 0.128546 (-0.105829) | 0.006933 / 0.075646 (-0.068713) | 0.201118 / 0.419271 (-0.218154) | 0.053422 / 0.043533 (0.009890) | 0.266262 / 0.255139 (0.011123) | 0.269114 / 0.283200 (-0.014086) | 0.016908 / 0.141683 (-0.124775) | 1.154296 / 1.452155 (-0.297859) | 1.218825 / 1.492716 (-0.273892) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089908 / 0.018006 (0.071902) | 0.300029 / 0.000490 (0.299539) | 0.000209 / 0.000200 (0.000009) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018518 / 0.037411 (-0.018894) | 0.062246 / 0.014526 (0.047720) | 0.073542 / 0.176557 (-0.103014) | 0.119386 / 0.737135 (-0.617749) | 0.075256 / 0.296338 (-0.221082) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280812 / 0.215209 (0.065603) | 2.701282 / 2.077655 (0.623628) | 1.455146 / 1.504120 (-0.048974) | 1.310198 / 1.541195 (-0.230996) | 1.335287 / 1.468490 (-0.133203) | 0.388245 / 4.584777 (-4.196532) | 2.357770 / 3.745712 (-1.387942) | 2.534640 / 5.269862 (-2.735222) | 1.541382 / 4.565676 (-3.024295) | 0.045597 / 0.424275 (-0.378678) | 0.004842 / 0.007607 (-0.002765) | 0.325416 / 0.226044 (0.099371) | 3.221873 / 2.268929 (0.952944) | 1.791061 / 55.444624 (-53.653563) | 1.485094 / 6.876477 (-5.391382) | 1.512354 / 2.142072 (-0.629718) | 0.471241 / 4.805227 (-4.333986) | 0.098672 / 6.500664 (-6.401992) | 0.041668 / 0.075469 (-0.033801) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.953553 / 1.841788 (-0.888234) | 11.378394 / 8.074308 (3.304086) | 10.355970 / 10.191392 (0.164578) | 0.126891 / 0.680424 (-0.553533) | 0.013808 / 0.534201 (-0.520393) | 0.267800 / 0.579283 (-0.311484) | 0.266436 / 0.434364 (-0.167928) | 0.306668 / 0.540337 (-0.233670) | 0.427666 / 1.386936 (-0.959270) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004908 / 0.011353 (-0.006445) | 0.002698 / 0.011008 (-0.008310) | 0.047492 / 0.038508 (0.008984) | 0.049906 / 0.023109 (0.026797) | 0.271466 / 0.275898 (-0.004432) | 0.291030 / 0.323480 (-0.032449) | 0.003938 / 0.007986 (-0.004047) | 0.002457 / 0.004328 (-0.001871) | 0.047347 / 0.004250 (0.043096) | 0.038599 / 0.037052 (0.001547) | 0.269950 / 0.258489 (0.011461) | 0.303026 / 0.293841 (0.009185) | 0.024196 / 0.128546 (-0.104351) | 0.006889 / 0.075646 (-0.068757) | 0.053357 / 0.419271 (-0.365914) | 0.032249 / 0.043533 (-0.011284) | 0.271660 / 0.255139 (0.016521) | 0.286395 / 0.283200 (0.003196) | 0.017914 / 0.141683 (-0.123769) | 1.128762 / 1.452155 (-0.323393) | 1.206495 / 1.492716 (-0.286221) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093384 / 0.018006 (0.075378) | 0.305504 / 0.000490 (0.305014) | 0.000227 / 0.000200 (0.000027) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021183 / 0.037411 (-0.016229) | 0.070113 / 0.014526 (0.055587) | 0.080288 / 0.176557 (-0.096269) | 0.120798 / 0.737135 (-0.616337) | 0.082896 / 0.296338 (-0.213442) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292758 / 0.215209 (0.077549) | 2.893975 / 2.077655 (0.816320) | 1.584909 / 1.504120 (0.080789) | 1.455509 / 1.541195 (-0.085686) | 1.501625 / 1.468490 (0.033135) | 0.400772 / 4.584777 (-4.184005) | 2.446319 / 3.745712 (-1.299393) | 2.530690 / 5.269862 (-2.739172) | 1.525957 / 4.565676 (-3.039719) | 0.046070 / 0.424275 (-0.378205) | 0.004756 / 0.007607 (-0.002851) | 0.343039 / 0.226044 (0.116995) | 3.366772 / 2.268929 (1.097844) | 1.948895 / 55.444624 (-53.495729) | 1.666419 / 6.876477 (-5.210058) | 1.658258 / 2.142072 (-0.483814) | 0.470835 / 4.805227 (-4.334392) | 0.098008 / 6.500664 (-6.402656) | 0.040743 / 0.075469 (-0.034726) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.978025 / 1.841788 (-0.863763) | 11.945229 / 8.074308 (3.870920) | 11.025810 / 10.191392 (0.834418) | 0.129706 / 0.680424 (-0.550717) | 0.015148 / 0.534201 (-0.519053) | 0.269160 / 0.579283 (-0.310123) | 0.284306 / 0.434364 (-0.150058) | 0.307154 / 0.540337 (-0.233183) | 0.409153 / 1.386936 (-0.977783) |

</details>
</details>

![](https://cml.dev/watermark.png#9c75c104fd79cbf53be25f0fbbeb001e535f7e9b "CML watermark")