Title: Make JSON builder support an array of strings
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-02-26T13:18:31Z

Support JSON file with an array of strings.

Fix #6695.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6696). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005057 / 0.011353 (-0.006296) | 0.003665 / 0.011008 (-0.007343) | 0.063217 / 0.038508 (0.024709) | 0.028789 / 0.023109 (0.005679) | 0.233597 / 0.275898 (-0.042301) | 0.254792 / 0.323480 (-0.068687) | 0.003065 / 0.007986 (-0.004921) | 0.002686 / 0.004328 (-0.001642) | 0.050182 / 0.004250 (0.045932) | 0.042204 / 0.037052 (0.005151) | 0.254262 / 0.258489 (-0.004227) | 0.277099 / 0.293841 (-0.016742) | 0.027564 / 0.128546 (-0.100982) | 0.010768 / 0.075646 (-0.064878) | 0.207302 / 0.419271 (-0.211969) | 0.035737 / 0.043533 (-0.007796) | 0.242388 / 0.255139 (-0.012751) | 0.259833 / 0.283200 (-0.023367) | 0.019833 / 0.141683 (-0.121850) | 1.135928 / 1.452155 (-0.316227) | 1.162851 / 1.492716 (-0.329865) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089209 / 0.018006 (0.071202) | 0.300493 / 0.000490 (0.300003) | 0.000216 / 0.000200 (0.000016) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017968 / 0.037411 (-0.019444) | 0.061773 / 0.014526 (0.047247) | 0.073835 / 0.176557 (-0.102722) | 0.118592 / 0.737135 (-0.618544) | 0.073606 / 0.296338 (-0.222732) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287858 / 0.215209 (0.072649) | 2.822917 / 2.077655 (0.745262) | 1.485259 / 1.504120 (-0.018861) | 1.355922 / 1.541195 (-0.185273) | 1.364008 / 1.468490 (-0.104482) | 0.557713 / 4.584777 (-4.027064) | 2.378972 / 3.745712 (-1.366741) | 2.737218 / 5.269862 (-2.532643) | 1.718317 / 4.565676 (-2.847359) | 0.062362 / 0.424275 (-0.361913) | 0.004992 / 0.007607 (-0.002615) | 0.350765 / 0.226044 (0.124721) | 3.387579 / 2.268929 (1.118650) | 1.860408 / 55.444624 (-53.584216) | 1.569355 / 6.876477 (-5.307122) | 1.593013 / 2.142072 (-0.549059) | 0.639325 / 4.805227 (-4.165902) | 0.121769 / 6.500664 (-6.378895) | 0.042148 / 0.075469 (-0.033322) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.990594 / 1.841788 (-0.851194) | 11.460904 / 8.074308 (3.386596) | 9.438691 / 10.191392 (-0.752701) | 0.141884 / 0.680424 (-0.538540) | 0.013725 / 0.534201 (-0.520476) | 0.288847 / 0.579283 (-0.290436) | 0.278815 / 0.434364 (-0.155549) | 0.337108 / 0.540337 (-0.203229) | 0.441659 / 1.386936 (-0.945277) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005265 / 0.011353 (-0.006088) | 0.003734 / 0.011008 (-0.007274) | 0.049365 / 0.038508 (0.010857) | 0.030483 / 0.023109 (0.007373) | 0.275085 / 0.275898 (-0.000813) | 0.296004 / 0.323480 (-0.027475) | 0.004964 / 0.007986 (-0.003022) | 0.002542 / 0.004328 (-0.001787) | 0.048734 / 0.004250 (0.044483) | 0.044098 / 0.037052 (0.007046) | 0.292517 / 0.258489 (0.034028) | 0.319992 / 0.293841 (0.026151) | 0.029552 / 0.128546 (-0.098994) | 0.010669 / 0.075646 (-0.064977) | 0.058887 / 0.419271 (-0.360385) | 0.051163 / 0.043533 (0.007630) | 0.277266 / 0.255139 (0.022127) | 0.295347 / 0.283200 (0.012147) | 0.018403 / 0.141683 (-0.123280) | 1.151979 / 1.452155 (-0.300176) | 1.204583 / 1.492716 (-0.288134) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091157 / 0.018006 (0.073151) | 0.300109 / 0.000490 (0.299619) | 0.000211 / 0.000200 (0.000011) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021521 / 0.037411 (-0.015890) | 0.074954 / 0.014526 (0.060428) | 0.087010 / 0.176557 (-0.089546) | 0.125853 / 0.737135 (-0.611282) | 0.087877 / 0.296338 (-0.208461) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297890 / 0.215209 (0.082681) | 2.912159 / 2.077655 (0.834504) | 1.619311 / 1.504120 (0.115192) | 1.501726 / 1.541195 (-0.039468) | 1.494143 / 1.468490 (0.025652) | 0.566744 / 4.584777 (-4.018033) | 2.497594 / 3.745712 (-1.248118) | 2.631403 / 5.269862 (-2.638459) | 1.727896 / 4.565676 (-2.837780) | 0.065937 / 0.424275 (-0.358339) | 0.005023 / 0.007607 (-0.002585) | 0.345747 / 0.226044 (0.119702) | 3.417615 / 2.268929 (1.148686) | 1.949970 / 55.444624 (-53.494654) | 1.680019 / 6.876477 (-5.196457) | 1.789879 / 2.142072 (-0.352193) | 0.648053 / 4.805227 (-4.157174) | 0.117408 / 6.500664 (-6.383256) | 0.040681 / 0.075469 (-0.034788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.012535 / 1.841788 (-0.829252) | 11.935819 / 8.074308 (3.861511) | 10.241452 / 10.191392 (0.050060) | 0.130956 / 0.680424 (-0.549468) | 0.015396 / 0.534201 (-0.518805) | 0.289166 / 0.579283 (-0.290117) | 0.274149 / 0.434364 (-0.160215) | 0.325844 / 0.540337 (-0.214493) | 0.424919 / 1.386936 (-0.962017) |

</details>
</details>

![](https://cml.dev/watermark.png#cb834d9c63ab8cb14725ae8e4fc2da8672892a6d "CML watermark")