Title: Allow null values in dict columns
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2024-03-19T16:54:22Z

Fix #6738

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005013 / 0.011353 (-0.006340) | 0.003228 / 0.011008 (-0.007780) | 0.062763 / 0.038508 (0.024255) | 0.028937 / 0.023109 (0.005828) | 0.240777 / 0.275898 (-0.035121) | 0.266972 / 0.323480 (-0.056508) | 0.003073 / 0.007986 (-0.004913) | 0.002769 / 0.004328 (-0.001560) | 0.049265 / 0.004250 (0.045015) | 0.042061 / 0.037052 (0.005009) | 0.261714 / 0.258489 (0.003225) | 0.284896 / 0.293841 (-0.008944) | 0.027717 / 0.128546 (-0.100829) | 0.010430 / 0.075646 (-0.065216) | 0.209022 / 0.419271 (-0.210249) | 0.035941 / 0.043533 (-0.007591) | 0.246849 / 0.255139 (-0.008290) | 0.263205 / 0.283200 (-0.019994) | 0.019489 / 0.141683 (-0.122193) | 1.102595 / 1.452155 (-0.349559) | 1.170493 / 1.492716 (-0.322223) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093611 / 0.018006 (0.075604) | 0.302041 / 0.000490 (0.301551) | 0.000223 / 0.000200 (0.000023) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018720 / 0.037411 (-0.018692) | 0.062199 / 0.014526 (0.047673) | 0.074888 / 0.176557 (-0.101669) | 0.120184 / 0.737135 (-0.616951) | 0.076756 / 0.296338 (-0.219583) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287484 / 0.215209 (0.072275) | 2.787777 / 2.077655 (0.710123) | 1.488957 / 1.504120 (-0.015163) | 1.362678 / 1.541195 (-0.178517) | 1.364571 / 1.468490 (-0.103919) | 0.563139 / 4.584777 (-4.021638) | 2.422224 / 3.745712 (-1.323488) | 2.798011 / 5.269862 (-2.471850) | 1.751159 / 4.565676 (-2.814517) | 0.062740 / 0.424275 (-0.361536) | 0.004918 / 0.007607 (-0.002689) | 0.338285 / 0.226044 (0.112240) | 3.316012 / 2.268929 (1.047083) | 1.845975 / 55.444624 (-53.598650) | 1.553187 / 6.876477 (-5.323290) | 1.564582 / 2.142072 (-0.577490) | 0.645987 / 4.805227 (-4.159240) | 0.118216 / 6.500664 (-6.382448) | 0.041243 / 0.075469 (-0.034226) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970265 / 1.841788 (-0.871522) | 11.783152 / 8.074308 (3.708844) | 9.516584 / 10.191392 (-0.674808) | 0.148086 / 0.680424 (-0.532338) | 0.013689 / 0.534201 (-0.520512) | 0.289657 / 0.579283 (-0.289626) | 0.265966 / 0.434364 (-0.168398) | 0.328483 / 0.540337 (-0.211854) | 0.433544 / 1.386936 (-0.953392) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005235 / 0.011353 (-0.006118) | 0.003515 / 0.011008 (-0.007493) | 0.049484 / 0.038508 (0.010976) | 0.029264 / 0.023109 (0.006154) | 0.278518 / 0.275898 (0.002620) | 0.298948 / 0.323480 (-0.024532) | 0.004308 / 0.007986 (-0.003678) | 0.002751 / 0.004328 (-0.001577) | 0.048952 / 0.004250 (0.044701) | 0.045379 / 0.037052 (0.008327) | 0.292633 / 0.258489 (0.034144) | 0.319405 / 0.293841 (0.025564) | 0.030201 / 0.128546 (-0.098345) | 0.010657 / 0.075646 (-0.064990) | 0.057842 / 0.419271 (-0.361430) | 0.053359 / 0.043533 (0.009826) | 0.281136 / 0.255139 (0.025997) | 0.295388 / 0.283200 (0.012188) | 0.018786 / 0.141683 (-0.122897) | 1.187181 / 1.452155 (-0.264974) | 1.198394 / 1.492716 (-0.294323) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093861 / 0.018006 (0.075855) | 0.304019 / 0.000490 (0.303529) | 0.000220 / 0.000200 (0.000020) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021582 / 0.037411 (-0.015829) | 0.075381 / 0.014526 (0.060855) | 0.087886 / 0.176557 (-0.088671) | 0.125078 / 0.737135 (-0.612057) | 0.089339 / 0.296338 (-0.206999) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295797 / 0.215209 (0.080588) | 2.912021 / 2.077655 (0.834367) | 1.592191 / 1.504120 (0.088071) | 1.471270 / 1.541195 (-0.069925) | 1.475535 / 1.468490 (0.007045) | 0.564114 / 4.584777 (-4.020663) | 2.442882 / 3.745712 (-1.302830) | 2.679433 / 5.269862 (-2.590428) | 1.752097 / 4.565676 (-2.813579) | 0.062748 / 0.424275 (-0.361527) | 0.005068 / 0.007607 (-0.002539) | 0.345554 / 0.226044 (0.119509) | 3.456929 / 2.268929 (1.188000) | 1.962781 / 55.444624 (-53.481844) | 1.688313 / 6.876477 (-5.188164) | 1.817392 / 2.142072 (-0.324681) | 0.639588 / 4.805227 (-4.165639) | 0.116148 / 6.500664 (-6.384516) | 0.040851 / 0.075469 (-0.034618) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009852 / 1.841788 (-0.831936) | 12.031749 / 8.074308 (3.957440) | 10.305107 / 10.191392 (0.113715) | 0.132960 / 0.680424 (-0.547464) | 0.014779 / 0.534201 (-0.519422) | 0.288903 / 0.579283 (-0.290381) | 0.275417 / 0.434364 (-0.158947) | 0.322628 / 0.540337 (-0.217709) | 0.445060 / 1.386936 (-0.941876) |

</details>
</details>

![](https://cml.dev/watermark.png#f234fce40d5ffc96fac5198d8cc89817970d87ee "CML watermark")


notify https://huggingface.co/datasets/chaoyi-wu/PMC-Inline/discussions/1 once it's merged in dataset-viewer