Title: Allow polars as valid output type
Author: psmyth94 (CONTRIBUTOR)
State: closed
Created: 2024-03-28T13:40:28Z

I was trying out polars as an output for a map function and found that it wasn't a valid return type in `validate_function_output`. Thought that we should accommodate this by creating and adding it to the `allowed_processed_input_types` variable.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hello @lhoestq, I added the test and modified [update_data](https://github.com/huggingface/datasets/blob/bececdac927160b5c7e883736d7cc79d5699ad0a/src/datasets/arrow_dataset.py#L3437) to include `polars` as an updatable type. Although, it seems pretty redundant to do the type checks both before `validate_function_output` and then immediately afterward within the call stack. Could consider adding `allowable_types` in `validation_function_output`.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005530 / 0.011353 (-0.005823) | 0.004012 / 0.011008 (-0.006996) | 0.062474 / 0.038508 (0.023966) | 0.031896 / 0.023109 (0.008787) | 0.239620 / 0.275898 (-0.036278) | 0.264694 / 0.323480 (-0.058785) | 0.003199 / 0.007986 (-0.004786) | 0.003141 / 0.004328 (-0.001187) | 0.048726 / 0.004250 (0.044475) | 0.044795 / 0.037052 (0.007743) | 0.250661 / 0.258489 (-0.007828) | 0.279658 / 0.293841 (-0.014183) | 0.029857 / 0.128546 (-0.098689) | 0.012293 / 0.075646 (-0.063353) | 0.203626 / 0.419271 (-0.215646) | 0.036284 / 0.043533 (-0.007249) | 0.241678 / 0.255139 (-0.013461) | 0.259380 / 0.283200 (-0.023820) | 0.020400 / 0.141683 (-0.121283) | 1.142334 / 1.452155 (-0.309821) | 1.199068 / 1.492716 (-0.293648) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097348 / 0.018006 (0.079341) | 0.303468 / 0.000490 (0.302978) | 0.000219 / 0.000200 (0.000019) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018646 / 0.037411 (-0.018766) | 0.062374 / 0.014526 (0.047848) | 0.074585 / 0.176557 (-0.101972) | 0.120380 / 0.737135 (-0.616755) | 0.075685 / 0.296338 (-0.220653) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277488 / 0.215209 (0.062279) | 2.741734 / 2.077655 (0.664080) | 1.451901 / 1.504120 (-0.052219) | 1.341712 / 1.541195 (-0.199482) | 1.395209 / 1.468490 (-0.073282) | 0.736334 / 4.584777 (-3.848443) | 2.358225 / 3.745712 (-1.387487) | 2.951838 / 5.269862 (-2.318023) | 1.892027 / 4.565676 (-2.673649) | 0.077913 / 0.424275 (-0.346362) | 0.005188 / 0.007607 (-0.002419) | 0.328790 / 0.226044 (0.102745) | 3.259387 / 2.268929 (0.990459) | 1.826102 / 55.444624 (-53.618522) | 1.526635 / 6.876477 (-5.349842) | 1.576392 / 2.142072 (-0.565680) | 0.786244 / 4.805227 (-4.018983) | 0.133909 / 6.500664 (-6.366756) | 0.044544 / 0.075469 (-0.030925) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965314 / 1.841788 (-0.876474) | 11.786831 / 8.074308 (3.712523) | 9.568519 / 10.191392 (-0.622873) | 0.140628 / 0.680424 (-0.539796) | 0.014442 / 0.534201 (-0.519759) | 0.300876 / 0.579283 (-0.278407) | 0.262647 / 0.434364 (-0.171717) | 0.339141 / 0.540337 (-0.201196) | 0.430254 / 1.386936 (-0.956683) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006020 / 0.011353 (-0.005333) | 0.004191 / 0.011008 (-0.006818) | 0.050006 / 0.038508 (0.011498) | 0.033247 / 0.023109 (0.010138) | 0.270677 / 0.275898 (-0.005221) | 0.299539 / 0.323480 (-0.023941) | 0.004391 / 0.007986 (-0.003595) | 0.002825 / 0.004328 (-0.001504) | 0.048573 / 0.004250 (0.044322) | 0.042461 / 0.037052 (0.005409) | 0.283812 / 0.258489 (0.025323) | 0.324302 / 0.293841 (0.030461) | 0.033264 / 0.128546 (-0.095282) | 0.012405 / 0.075646 (-0.063241) | 0.060298 / 0.419271 (-0.358973) | 0.034833 / 0.043533 (-0.008700) | 0.271133 / 0.255139 (0.015994) | 0.290712 / 0.283200 (0.007512) | 0.019762 / 0.141683 (-0.121920) | 1.138644 / 1.452155 (-0.313511) | 1.204628 / 1.492716 (-0.288088) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096171 / 0.018006 (0.078164) | 0.308916 / 0.000490 (0.308427) | 0.000213 / 0.000200 (0.000013) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023077 / 0.037411 (-0.014334) | 0.078865 / 0.014526 (0.064339) | 0.091031 / 0.176557 (-0.085526) | 0.133536 / 0.737135 (-0.603599) | 0.093308 / 0.296338 (-0.203030) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301466 / 0.215209 (0.086257) | 2.995190 / 2.077655 (0.917535) | 1.616545 / 1.504120 (0.112425) | 1.472572 / 1.541195 (-0.068622) | 1.477191 / 1.468490 (0.008701) | 0.730240 / 4.584777 (-3.854537) | 0.966591 / 3.745712 (-2.779121) | 2.979970 / 5.269862 (-2.289892) | 1.908275 / 4.565676 (-2.657401) | 0.081346 / 0.424275 (-0.342929) | 0.005150 / 0.007607 (-0.002458) | 0.349066 / 0.226044 (0.123022) | 3.504363 / 2.268929 (1.235435) | 1.973355 / 55.444624 (-53.471270) | 1.659337 / 6.876477 (-5.217139) | 1.701282 / 2.142072 (-0.440790) | 0.813493 / 4.805227 (-3.991735) | 0.133537 / 6.500664 (-6.367127) | 0.041207 / 0.075469 (-0.034262) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.020368 / 1.841788 (-0.821420) | 12.444848 / 8.074308 (4.370540) | 10.113832 / 10.191392 (-0.077560) | 0.137782 / 0.680424 (-0.542642) | 0.015217 / 0.534201 (-0.518984) | 0.300419 / 0.579283 (-0.278864) | 0.128868 / 0.434364 (-0.305496) | 0.342831 / 0.540337 (-0.197506) | 0.443036 / 1.386936 (-0.943900) |

</details>
</details>

![](https://cml.dev/watermark.png#5f42139a2c5583a55d34a2f60d537f5fba285c28 "CML watermark")