Title: Improve documentation of dataset.from_generator
Author: hartmans (CONTRIBUTOR)
State: closed
Created: 2023-10-05T14:34:49Z

Improve documentation to clarify sharding behavior (#6270)

Comments:
I have looked at the doc failures, and I do not think that my change caused the doc build failure, but I'm not 100% sure about that.
I have high confidence that the integration test failures are not something I introduced:-)

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008557 / 0.011353 (-0.002796) | 0.005224 / 0.011008 (-0.005784) | 0.109402 / 0.038508 (0.070893) | 0.075008 / 0.023109 (0.051899) | 0.388910 / 0.275898 (0.113012) | 0.425481 / 0.323480 (0.102002) | 0.005046 / 0.007986 (-0.002939) | 0.004166 / 0.004328 (-0.000162) | 0.079890 / 0.004250 (0.075639) | 0.061992 / 0.037052 (0.024940) | 0.409933 / 0.258489 (0.151444) | 0.444096 / 0.293841 (0.150255) | 0.043958 / 0.128546 (-0.084588) | 0.013655 / 0.075646 (-0.061991) | 0.402620 / 0.419271 (-0.016651) | 0.062784 / 0.043533 (0.019251) | 0.399653 / 0.255139 (0.144514) | 0.432926 / 0.283200 (0.149727) | 0.034631 / 0.141683 (-0.107052) | 1.801450 / 1.452155 (0.349296) | 1.965007 / 1.492716 (0.472290) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305744 / 0.018006 (0.287738) | 0.590825 / 0.000490 (0.590335) | 0.014561 / 0.000200 (0.014361) | 0.000430 / 0.000054 (0.000375) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030449 / 0.037411 (-0.006962) | 0.091753 / 0.014526 (0.077227) | 0.106259 / 0.176557 (-0.070298) | 0.174599 / 0.737135 (-0.562537) | 0.107069 / 0.296338 (-0.189269) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.607544 / 0.215209 (0.392335) | 6.182592 / 2.077655 (4.104937) | 2.699782 / 1.504120 (1.195663) | 2.386915 / 1.541195 (0.845720) | 2.441763 / 1.468490 (0.973273) | 0.811360 / 4.584777 (-3.773417) | 5.253799 / 3.745712 (1.508087) | 4.762054 / 5.269862 (-0.507807) | 3.045161 / 4.565676 (-1.520515) | 0.095983 / 0.424275 (-0.328292) | 0.008653 / 0.007607 (0.001046) | 0.714218 / 0.226044 (0.488174) | 7.279325 / 2.268929 (5.010397) | 3.356107 / 55.444624 (-52.088517) | 2.765867 / 6.876477 (-4.110610) | 2.997756 / 2.142072 (0.855684) | 1.008740 / 4.805227 (-3.796487) | 0.201462 / 6.500664 (-6.299202) | 0.075780 / 0.075469 (0.000311) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.677034 / 1.841788 (-0.164754) | 23.546919 / 8.074308 (15.472610) | 21.576985 / 10.191392 (11.385593) | 0.239253 / 0.680424 (-0.441171) | 0.028740 / 0.534201 (-0.505460) | 0.468519 / 0.579283 (-0.110765) | 0.593935 / 0.434364 (0.159571) | 0.536830 / 0.540337 (-0.003507) | 0.779925 / 1.386936 (-0.607011) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009582 / 0.011353 (-0.001771) | 0.004971 / 0.011008 (-0.006037) | 0.081304 / 0.038508 (0.042796) | 0.077588 / 0.023109 (0.054478) | 0.486610 / 0.275898 (0.210712) | 0.580228 / 0.323480 (0.256748) | 0.006707 / 0.007986 (-0.001279) | 0.004325 / 0.004328 (-0.000004) | 0.086170 / 0.004250 (0.081920) | 0.060591 / 0.037052 (0.023539) | 0.501723 / 0.258489 (0.243234) | 0.548633 / 0.293841 (0.254793) | 0.050306 / 0.128546 (-0.078240) | 0.017458 / 0.075646 (-0.058188) | 0.093295 / 0.419271 (-0.325977) | 0.064588 / 0.043533 (0.021056) | 0.519395 / 0.255139 (0.264256) | 0.526021 / 0.283200 (0.242821) | 0.035795 / 0.141683 (-0.105888) | 1.792927 / 1.452155 (0.340772) | 1.956499 / 1.492716 (0.463783) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296249 / 0.018006 (0.278243) | 0.594482 / 0.000490 (0.593992) | 0.007318 / 0.000200 (0.007118) | 0.000182 / 0.000054 (0.000128) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036110 / 0.037411 (-0.001301) | 0.107924 / 0.014526 (0.093399) | 0.119975 / 0.176557 (-0.056582) | 0.177499 / 0.737135 (-0.559636) | 0.123299 / 0.296338 (-0.173039) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.632994 / 0.215209 (0.417785) | 6.481663 / 2.077655 (4.404008) | 3.231259 / 1.504120 (1.727139) | 2.768298 / 1.541195 (1.227103) | 2.694543 / 1.468490 (1.226053) | 0.837384 / 4.584777 (-3.747393) | 5.405278 / 3.745712 (1.659566) | 4.639424 / 5.269862 (-0.630437) | 2.944251 / 4.565676 (-1.621426) | 0.094978 / 0.424275 (-0.329297) | 0.008716 / 0.007607 (0.001108) | 0.795820 / 0.226044 (0.569776) | 8.514233 / 2.268929 (6.245304) | 3.800463 / 55.444624 (-51.644161) | 3.000005 / 6.876477 (-3.876472) | 3.298853 / 2.142072 (1.156781) | 0.994112 / 4.805227 (-3.811115) | 0.209435 / 6.500664 (-6.291229) | 0.075610 / 0.075469 (0.000141) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.681127 / 1.841788 (-0.160661) | 23.874465 / 8.074308 (15.800156) | 21.638567 / 10.191392 (11.447175) | 0.233303 / 0.680424 (-0.447121) | 0.032504 / 0.534201 (-0.501697) | 0.460462 / 0.579283 (-0.118821) | 0.560043 / 0.434364 (0.125679) | 0.555059 / 0.540337 (0.014721) | 0.831444 / 1.386936 (-0.555492) |

</details>
</details>

![](https://cml.dev/watermark.png#faada1742e1f25fce9cc5691ec11d3f91d4aa120 "CML watermark")