Title: Support PyArrow arrays as column values in `from_dict`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-03-15T19:32:40Z

For consistency with `pa.Table.from_pydict`, which supports both Python lists and PyArrow arrays as column values.

"Fixes" https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006665 / 0.011353 (-0.004688) | 0.004842 / 0.011008 (-0.006166) | 0.097802 / 0.038508 (0.059294) | 0.032292 / 0.023109 (0.009182) | 0.327522 / 0.275898 (0.051624) | 0.351851 / 0.323480 (0.028371) | 0.005197 / 0.007986 (-0.002789) | 0.003781 / 0.004328 (-0.000547) | 0.073213 / 0.004250 (0.068963) | 0.045819 / 0.037052 (0.008767) | 0.331323 / 0.258489 (0.072834) | 0.376978 / 0.293841 (0.083137) | 0.035014 / 0.128546 (-0.093532) | 0.011853 / 0.075646 (-0.063793) | 0.344031 / 0.419271 (-0.075240) | 0.049094 / 0.043533 (0.005561) | 0.327054 / 0.255139 (0.071915) | 0.349053 / 0.283200 (0.065853) | 0.095413 / 0.141683 (-0.046269) | 1.451593 / 1.452155 (-0.000562) | 1.505568 / 1.492716 (0.012851) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211624 / 0.018006 (0.193618) | 0.437569 / 0.000490 (0.437079) | 0.003775 / 0.000200 (0.003575) | 0.000082 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025915 / 0.037411 (-0.011496) | 0.104085 / 0.014526 (0.089559) | 0.111064 / 0.176557 (-0.065493) | 0.167316 / 0.737135 (-0.569819) | 0.117255 / 0.296338 (-0.179084) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424241 / 0.215209 (0.209032) | 4.251365 / 2.077655 (2.173710) | 2.074036 / 1.504120 (0.569916) | 1.858022 / 1.541195 (0.316828) | 1.819929 / 1.468490 (0.351439) | 0.704153 / 4.584777 (-3.880624) | 3.750506 / 3.745712 (0.004794) | 3.149836 / 5.269862 (-2.120026) | 1.729540 / 4.565676 (-2.836137) | 0.087287 / 0.424275 (-0.336988) | 0.012304 / 0.007607 (0.004697) | 0.513811 / 0.226044 (0.287767) | 5.129427 / 2.268929 (2.860498) | 2.489253 / 55.444624 (-52.955371) | 2.122746 / 6.876477 (-4.753730) | 2.208528 / 2.142072 (0.066456) | 0.843386 / 4.805227 (-3.961841) | 0.169320 / 6.500664 (-6.331344) | 0.064085 / 0.075469 (-0.011384) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.184361 / 1.841788 (-0.657427) | 14.013478 / 8.074308 (5.939170) | 13.936774 / 10.191392 (3.745382) | 0.138009 / 0.680424 (-0.542415) | 0.017192 / 0.534201 (-0.517009) | 0.420938 / 0.579283 (-0.158345) | 0.413390 / 0.434364 (-0.020974) | 0.500244 / 0.540337 (-0.040094) | 0.582499 / 1.386936 (-0.804437) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006709 / 0.011353 (-0.004643) | 0.004847 / 0.011008 (-0.006161) | 0.074740 / 0.038508 (0.036232) | 0.032126 / 0.023109 (0.009017) | 0.343248 / 0.275898 (0.067350) | 0.376822 / 0.323480 (0.053342) | 0.005547 / 0.007986 (-0.002439) | 0.005080 / 0.004328 (0.000752) | 0.074634 / 0.004250 (0.070384) | 0.044735 / 0.037052 (0.007682) | 0.357895 / 0.258489 (0.099406) | 0.401150 / 0.293841 (0.107310) | 0.035485 / 0.128546 (-0.093061) | 0.011978 / 0.075646 (-0.063668) | 0.087567 / 0.419271 (-0.331704) | 0.050233 / 0.043533 (0.006701) | 0.337476 / 0.255139 (0.082337) | 0.385064 / 0.283200 (0.101865) | 0.102733 / 0.141683 (-0.038950) | 1.456238 / 1.452155 (0.004083) | 1.539468 / 1.492716 (0.046752) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203156 / 0.018006 (0.185149) | 0.448898 / 0.000490 (0.448408) | 0.002843 / 0.000200 (0.002644) | 0.000222 / 0.000054 (0.000168) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027836 / 0.037411 (-0.009576) | 0.109889 / 0.014526 (0.095364) | 0.119378 / 0.176557 (-0.057179) | 0.171208 / 0.737135 (-0.565927) | 0.124240 / 0.296338 (-0.172098) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425374 / 0.215209 (0.210165) | 4.252994 / 2.077655 (2.175339) | 2.006410 / 1.504120 (0.502290) | 1.812821 / 1.541195 (0.271626) | 1.857618 / 1.468490 (0.389128) | 0.714564 / 4.584777 (-3.870213) | 3.803040 / 3.745712 (0.057328) | 2.075452 / 5.269862 (-3.194410) | 1.344868 / 4.565676 (-3.220809) | 0.088705 / 0.424275 (-0.335570) | 0.012481 / 0.007607 (0.004874) | 0.528022 / 0.226044 (0.301977) | 5.268878 / 2.268929 (2.999949) | 2.467858 / 55.444624 (-52.976767) | 2.138681 / 6.876477 (-4.737796) | 2.134928 / 2.142072 (-0.007145) | 0.851518 / 4.805227 (-3.953709) | 0.175085 / 6.500664 (-6.325579) | 0.063555 / 0.075469 (-0.011914) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.265788 / 1.841788 (-0.576000) | 14.683444 / 8.074308 (6.609136) | 14.055848 / 10.191392 (3.864456) | 0.145260 / 0.680424 (-0.535164) | 0.017064 / 0.534201 (-0.517137) | 0.424836 / 0.579283 (-0.154447) | 0.418345 / 0.434364 (-0.016019) | 0.491408 / 0.540337 (-0.048930) | 0.594387 / 1.386936 (-0.792549) |

</details>
</details>

![](https://cml.dev/watermark.png#10c3f32c228cc7011ce456498942e6a2a5dc3086 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006870 / 0.011353 (-0.004483) | 0.004602 / 0.011008 (-0.006406) | 0.100075 / 0.038508 (0.061567) | 0.028720 / 0.023109 (0.005611) | 0.304212 / 0.275898 (0.028314) | 0.348423 / 0.323480 (0.024943) | 0.005266 / 0.007986 (-0.002720) | 0.003473 / 0.004328 (-0.000855) | 0.077563 / 0.004250 (0.073313) | 0.040066 / 0.037052 (0.003013) | 0.304039 / 0.258489 (0.045550) | 0.348721 / 0.293841 (0.054881) | 0.032127 / 0.128546 (-0.096419) | 0.011583 / 0.075646 (-0.064063) | 0.326853 / 0.419271 (-0.092418) | 0.043158 / 0.043533 (-0.000375) | 0.310111 / 0.255139 (0.054973) | 0.332869 / 0.283200 (0.049670) | 0.088384 / 0.141683 (-0.053299) | 1.509245 / 1.452155 (0.057091) | 1.575393 / 1.492716 (0.082677) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212839 / 0.018006 (0.194833) | 0.431407 / 0.000490 (0.430918) | 0.002639 / 0.000200 (0.002439) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024945 / 0.037411 (-0.012466) | 0.101312 / 0.014526 (0.086787) | 0.107873 / 0.176557 (-0.068683) | 0.169579 / 0.737135 (-0.567556) | 0.109922 / 0.296338 (-0.186417) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422091 / 0.215209 (0.206882) | 4.227174 / 2.077655 (2.149519) | 1.957964 / 1.504120 (0.453844) | 1.812076 / 1.541195 (0.270882) | 1.966666 / 1.468490 (0.498176) | 0.698710 / 4.584777 (-3.886067) | 3.431824 / 3.745712 (-0.313888) | 1.898646 / 5.269862 (-3.371215) | 1.172096 / 4.565676 (-3.393581) | 0.083383 / 0.424275 (-0.340892) | 0.012793 / 0.007607 (0.005186) | 0.522501 / 0.226044 (0.296457) | 5.240049 / 2.268929 (2.971121) | 2.349286 / 55.444624 (-53.095338) | 2.051117 / 6.876477 (-4.825360) | 2.255652 / 2.142072 (0.113580) | 0.813668 / 4.805227 (-3.991560) | 0.153770 / 6.500664 (-6.346894) | 0.068323 / 0.075469 (-0.007146) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.197204 / 1.841788 (-0.644584) | 14.146212 / 8.074308 (6.071904) | 14.469765 / 10.191392 (4.278373) | 0.130024 / 0.680424 (-0.550400) | 0.016858 / 0.534201 (-0.517343) | 0.382949 / 0.579283 (-0.196334) | 0.393414 / 0.434364 (-0.040950) | 0.447910 / 0.540337 (-0.092427) | 0.529842 / 1.386936 (-0.857094) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006903 / 0.011353 (-0.004450) | 0.004695 / 0.011008 (-0.006313) | 0.077457 / 0.038508 (0.038949) | 0.028624 / 0.023109 (0.005514) | 0.340767 / 0.275898 (0.064869) | 0.378811 / 0.323480 (0.055331) | 0.005996 / 0.007986 (-0.001990) | 0.003481 / 0.004328 (-0.000848) | 0.076284 / 0.004250 (0.072034) | 0.042564 / 0.037052 (0.005511) | 0.340908 / 0.258489 (0.082419) | 0.384952 / 0.293841 (0.091111) | 0.032057 / 0.128546 (-0.096489) | 0.011697 / 0.075646 (-0.063949) | 0.085941 / 0.419271 (-0.333331) | 0.042464 / 0.043533 (-0.001069) | 0.339309 / 0.255139 (0.084170) | 0.368105 / 0.283200 (0.084905) | 0.093382 / 0.141683 (-0.048301) | 1.467220 / 1.452155 (0.015065) | 1.563105 / 1.492716 (0.070389) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260631 / 0.018006 (0.242625) | 0.418155 / 0.000490 (0.417665) | 0.009539 / 0.000200 (0.009339) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025494 / 0.037411 (-0.011917) | 0.106034 / 0.014526 (0.091508) | 0.109878 / 0.176557 (-0.066678) | 0.160754 / 0.737135 (-0.576382) | 0.113226 / 0.296338 (-0.183112) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442989 / 0.215209 (0.227780) | 4.447040 / 2.077655 (2.369385) | 2.082529 / 1.504120 (0.578409) | 1.876952 / 1.541195 (0.335757) | 1.968341 / 1.468490 (0.499851) | 0.704317 / 4.584777 (-3.880460) | 3.466190 / 3.745712 (-0.279523) | 1.924954 / 5.269862 (-3.344908) | 1.199763 / 4.565676 (-3.365913) | 0.084320 / 0.424275 (-0.339955) | 0.012956 / 0.007607 (0.005349) | 0.538905 / 0.226044 (0.312861) | 5.426593 / 2.268929 (3.157665) | 2.509287 / 55.444624 (-52.935338) | 2.174829 / 6.876477 (-4.701648) | 2.239214 / 2.142072 (0.097141) | 0.810031 / 4.805227 (-3.995196) | 0.153534 / 6.500664 (-6.347130) | 0.069578 / 0.075469 (-0.005891) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.294068 / 1.841788 (-0.547720) | 14.601899 / 8.074308 (6.527591) | 14.469282 / 10.191392 (4.277890) | 0.130024 / 0.680424 (-0.550400) | 0.016895 / 0.534201 (-0.517306) | 0.382583 / 0.579283 (-0.196700) | 0.388938 / 0.434364 (-0.045426) | 0.448416 / 0.540337 (-0.091922) | 0.533261 / 1.386936 (-0.853675) |

</details>
</details>

![](https://cml.dev/watermark.png#7b2af47647152d39a3acade256da898cb396e4d9 "CML watermark")