Title: Add metadata ui screenshot in docs
Author: lhoestq (MEMBER)
State: closed
Created: 2023-07-11T12:16:29Z


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007633 / 0.011353 (-0.003720) | 0.004666 / 0.011008 (-0.006343) | 0.097768 / 0.038508 (0.059260) | 0.085153 / 0.023109 (0.062044) | 0.400315 / 0.275898 (0.124417) | 0.452903 / 0.323480 (0.129423) | 0.006227 / 0.007986 (-0.001759) | 0.003814 / 0.004328 (-0.000515) | 0.074586 / 0.004250 (0.070336) | 0.064295 / 0.037052 (0.027242) | 0.408082 / 0.258489 (0.149593) | 0.446921 / 0.293841 (0.153080) | 0.034593 / 0.128546 (-0.093953) | 0.009191 / 0.075646 (-0.066456) | 0.337099 / 0.419271 (-0.082173) | 0.075320 / 0.043533 (0.031787) | 0.403488 / 0.255139 (0.148349) | 0.435309 / 0.283200 (0.152109) | 0.035675 / 0.141683 (-0.106008) | 1.732642 / 1.452155 (0.280487) | 1.770238 / 1.492716 (0.277522) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235879 / 0.018006 (0.217873) | 0.500330 / 0.000490 (0.499841) | 0.005221 / 0.000200 (0.005021) | 0.000150 / 0.000054 (0.000096) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032479 / 0.037411 (-0.004933) | 0.095873 / 0.014526 (0.081348) | 0.107118 / 0.176557 (-0.069438) | 0.173809 / 0.737135 (-0.563326) | 0.109832 / 0.296338 (-0.186507) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444342 / 0.215209 (0.229133) | 4.459010 / 2.077655 (2.381355) | 2.209687 / 1.504120 (0.705567) | 2.007556 / 1.541195 (0.466362) | 2.113683 / 1.468490 (0.645193) | 0.544281 / 4.584777 (-4.040496) | 4.037151 / 3.745712 (0.291439) | 4.852644 / 5.269862 (-0.417217) | 3.134126 / 4.565676 (-1.431550) | 0.066815 / 0.424275 (-0.357460) | 0.008836 / 0.007607 (0.001229) | 0.560904 / 0.226044 (0.334859) | 5.302760 / 2.268929 (3.033832) | 2.750182 / 55.444624 (-52.694442) | 2.322595 / 6.876477 (-4.553882) | 2.547486 / 2.142072 (0.405414) | 0.665766 / 4.805227 (-4.139461) | 0.151613 / 6.500664 (-6.349051) | 0.071155 / 0.075469 (-0.004314) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.473717 / 1.841788 (-0.368071) | 22.584179 / 8.074308 (14.509871) | 15.888001 / 10.191392 (5.696609) | 0.181073 / 0.680424 (-0.499351) | 0.021395 / 0.534201 (-0.512806) | 0.452693 / 0.579283 (-0.126590) | 0.447709 / 0.434364 (0.013345) | 0.529599 / 0.540337 (-0.010738) | 0.699241 / 1.386936 (-0.687695) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007917 / 0.011353 (-0.003436) | 0.004544 / 0.011008 (-0.006464) | 0.074566 / 0.038508 (0.036058) | 0.087530 / 0.023109 (0.064421) | 0.419753 / 0.275898 (0.143854) | 0.452352 / 0.323480 (0.128872) | 0.005882 / 0.007986 (-0.002104) | 0.003904 / 0.004328 (-0.000425) | 0.073539 / 0.004250 (0.069289) | 0.071320 / 0.037052 (0.034267) | 0.432899 / 0.258489 (0.174409) | 0.470365 / 0.293841 (0.176524) | 0.036198 / 0.128546 (-0.092348) | 0.009342 / 0.075646 (-0.066304) | 0.080970 / 0.419271 (-0.338301) | 0.058769 / 0.043533 (0.015236) | 0.413397 / 0.255139 (0.158258) | 0.448362 / 0.283200 (0.165162) | 0.034177 / 0.141683 (-0.107506) | 1.706217 / 1.452155 (0.254063) | 1.776743 / 1.492716 (0.284026) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198779 / 0.018006 (0.180773) | 0.499862 / 0.000490 (0.499372) | 0.003891 / 0.000200 (0.003692) | 0.000108 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034671 / 0.037411 (-0.002740) | 0.103165 / 0.014526 (0.088639) | 0.115813 / 0.176557 (-0.060744) | 0.177407 / 0.737135 (-0.559728) | 0.117733 / 0.296338 (-0.178606) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.476859 / 0.215209 (0.261650) | 4.823063 / 2.077655 (2.745409) | 2.524133 / 1.504120 (1.020013) | 2.374482 / 1.541195 (0.833288) | 2.518047 / 1.468490 (1.049557) | 0.559131 / 4.584777 (-4.025646) | 4.126213 / 3.745712 (0.380501) | 6.488570 / 5.269862 (1.218708) | 3.816540 / 4.565676 (-0.749137) | 0.064742 / 0.424275 (-0.359533) | 0.008476 / 0.007607 (0.000869) | 0.576432 / 0.226044 (0.350387) | 5.835133 / 2.268929 (3.566205) | 3.237833 / 55.444624 (-52.206791) | 2.726596 / 6.876477 (-4.149880) | 2.799212 / 2.142072 (0.657139) | 0.661628 / 4.805227 (-4.143599) | 0.153997 / 6.500664 (-6.346667) | 0.070621 / 0.075469 (-0.004848) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.648505 / 1.841788 (-0.193282) | 22.454019 / 8.074308 (14.379711) | 16.077098 / 10.191392 (5.885706) | 0.217875 / 0.680424 (-0.462549) | 0.021285 / 0.534201 (-0.512916) | 0.459837 / 0.579283 (-0.119446) | 0.476211 / 0.434364 (0.041847) | 0.525903 / 0.540337 (-0.014435) | 0.717224 / 1.386936 (-0.669712) |

</details>
</details>

![](https://cml.dev/watermark.png#b767e9c3ef30f9da30d47cfcaccf9a7ac2500c43 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008929 / 0.011353 (-0.002424) | 0.004188 / 0.011008 (-0.006820) | 0.097030 / 0.038508 (0.058522) | 0.071363 / 0.023109 (0.048254) | 0.333116 / 0.275898 (0.057218) | 0.371272 / 0.323480 (0.047792) | 0.006430 / 0.007986 (-0.001555) | 0.003689 / 0.004328 (-0.000639) | 0.068666 / 0.004250 (0.064416) | 0.057562 / 0.037052 (0.020510) | 0.347208 / 0.258489 (0.088719) | 0.390514 / 0.293841 (0.096673) | 0.050560 / 0.128546 (-0.077987) | 0.013372 / 0.075646 (-0.062275) | 0.311345 / 0.419271 (-0.107927) | 0.068990 / 0.043533 (0.025457) | 0.363026 / 0.255139 (0.107887) | 0.379793 / 0.283200 (0.096593) | 0.036891 / 0.141683 (-0.104792) | 1.583481 / 1.452155 (0.131327) | 1.688727 / 1.492716 (0.196011) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209777 / 0.018006 (0.191771) | 0.507267 / 0.000490 (0.506777) | 0.003637 / 0.000200 (0.003438) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029309 / 0.037411 (-0.008102) | 0.088386 / 0.014526 (0.073861) | 0.104974 / 0.176557 (-0.071582) | 0.171999 / 0.737135 (-0.565137) | 0.110797 / 0.296338 (-0.185542) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.543465 / 0.215209 (0.328256) | 5.361491 / 2.077655 (3.283836) | 2.348712 / 1.504120 (0.844592) | 2.012527 / 1.541195 (0.471332) | 2.069776 / 1.468490 (0.601286) | 0.874262 / 4.584777 (-3.710515) | 4.877317 / 3.745712 (1.131605) | 5.327459 / 5.269862 (0.057597) | 3.336823 / 4.565676 (-1.228854) | 0.100456 / 0.424275 (-0.323819) | 0.008503 / 0.007607 (0.000895) | 0.692009 / 0.226044 (0.465965) | 6.912731 / 2.268929 (4.643802) | 3.110548 / 55.444624 (-52.334076) | 2.443665 / 6.876477 (-4.432811) | 2.528713 / 2.142072 (0.386641) | 1.076358 / 4.805227 (-3.728869) | 0.220352 / 6.500664 (-6.280312) | 0.080293 / 0.075469 (0.004824) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.538444 / 1.841788 (-0.303344) | 21.121221 / 8.074308 (13.046913) | 19.810609 / 10.191392 (9.619216) | 0.225406 / 0.680424 (-0.455018) | 0.026652 / 0.534201 (-0.507549) | 0.430372 / 0.579283 (-0.148911) | 0.510722 / 0.434364 (0.076358) | 0.514347 / 0.540337 (-0.025991) | 0.686050 / 1.386936 (-0.700886) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007675 / 0.011353 (-0.003678) | 0.004542 / 0.011008 (-0.006466) | 0.069655 / 0.038508 (0.031147) | 0.069338 / 0.023109 (0.046229) | 0.436505 / 0.275898 (0.160607) | 0.481806 / 0.323480 (0.158326) | 0.005315 / 0.007986 (-0.002670) | 0.004455 / 0.004328 (0.000127) | 0.072674 / 0.004250 (0.068424) | 0.058088 / 0.037052 (0.021035) | 0.445825 / 0.258489 (0.187336) | 0.501706 / 0.293841 (0.207865) | 0.047123 / 0.128546 (-0.081424) | 0.012943 / 0.075646 (-0.062703) | 0.093491 / 0.419271 (-0.325780) | 0.060169 / 0.043533 (0.016637) | 0.436530 / 0.255139 (0.181391) | 0.466873 / 0.283200 (0.183674) | 0.040453 / 0.141683 (-0.101230) | 1.586438 / 1.452155 (0.134283) | 1.671081 / 1.492716 (0.178365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180607 / 0.018006 (0.162601) | 0.520145 / 0.000490 (0.519655) | 0.004824 / 0.000200 (0.004624) | 0.000116 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029308 / 0.037411 (-0.008103) | 0.093652 / 0.014526 (0.079126) | 0.102332 / 0.176557 (-0.074224) | 0.162414 / 0.737135 (-0.574721) | 0.098017 / 0.296338 (-0.198321) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.583949 / 0.215209 (0.368740) | 6.035191 / 2.077655 (3.957536) | 2.801274 / 1.504120 (1.297155) | 2.566150 / 1.541195 (1.024955) | 2.437122 / 1.468490 (0.968632) | 0.865038 / 4.584777 (-3.719739) | 4.841727 / 3.745712 (1.096015) | 4.683919 / 5.269862 (-0.585943) | 2.941240 / 4.565676 (-1.624437) | 0.104888 / 0.424275 (-0.319387) | 0.007747 / 0.007607 (0.000140) | 0.780041 / 0.226044 (0.553997) | 7.771314 / 2.268929 (5.502385) | 3.680814 / 55.444624 (-51.763811) | 2.938472 / 6.876477 (-3.938004) | 2.981740 / 2.142072 (0.839668) | 1.065411 / 4.805227 (-3.739816) | 0.222265 / 6.500664 (-6.278399) | 0.082428 / 0.075469 (0.006959) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.626774 / 1.841788 (-0.215014) | 21.618284 / 8.074308 (13.543976) | 20.596743 / 10.191392 (10.405351) | 0.240969 / 0.680424 (-0.439454) | 0.025630 / 0.534201 (-0.508570) | 0.481981 / 0.579283 (-0.097302) | 0.547914 / 0.434364 (0.113550) | 0.522296 / 0.540337 (-0.018041) | 0.729174 / 1.386936 (-0.657762) |

</details>
</details>

![](https://cml.dev/watermark.png#b8067c0262073891180869f700ebef5ac3dc5cce "CML watermark")