Title: Refactor extensions
Author: albertvillanova (MEMBER)
State: closed
Created: 2023-05-31T08:33:02Z

Related to:
- #5850

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008358 / 0.011353 (-0.002995) | 0.005673 / 0.011008 (-0.005335) | 0.124034 / 0.038508 (0.085526) | 0.037550 / 0.023109 (0.014441) | 0.331301 / 0.275898 (0.055403) | 0.383542 / 0.323480 (0.060062) | 0.006940 / 0.007986 (-0.001046) | 0.005959 / 0.004328 (0.001631) | 0.084670 / 0.004250 (0.080419) | 0.054214 / 0.037052 (0.017162) | 0.359897 / 0.258489 (0.101408) | 0.383260 / 0.293841 (0.089419) | 0.047642 / 0.128546 (-0.080904) | 0.013902 / 0.075646 (-0.061744) | 0.380232 / 0.419271 (-0.039040) | 0.077790 / 0.043533 (0.034257) | 0.376648 / 0.255139 (0.121509) | 0.387536 / 0.283200 (0.104336) | 0.104644 / 0.141683 (-0.037038) | 1.618560 / 1.452155 (0.166406) | 1.742569 / 1.492716 (0.249853) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.257218 / 0.018006 (0.239212) | 0.636801 / 0.000490 (0.636311) | 0.000634 / 0.000200 (0.000434) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037874 / 0.037411 (0.000462) | 0.107454 / 0.014526 (0.092928) | 0.117855 / 0.176557 (-0.058702) | 0.204067 / 0.737135 (-0.533068) | 0.134029 / 0.296338 (-0.162310) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.583657 / 0.215209 (0.368447) | 5.761289 / 2.077655 (3.683635) | 2.280201 / 1.504120 (0.776081) | 2.033442 / 1.541195 (0.492247) | 2.035343 / 1.468490 (0.566853) | 0.868122 / 4.584777 (-3.716655) | 5.352591 / 3.745712 (1.606879) | 2.432814 / 5.269862 (-2.837047) | 1.560765 / 4.565676 (-3.004911) | 0.098793 / 0.424275 (-0.325482) | 0.017327 / 0.007607 (0.009720) | 0.734676 / 0.226044 (0.508631) | 7.070318 / 2.268929 (4.801390) | 2.972701 / 55.444624 (-52.471924) | 2.442189 / 6.876477 (-4.434288) | 2.604379 / 2.142072 (0.462307) | 1.028853 / 4.805227 (-3.776374) | 0.210390 / 6.500664 (-6.290274) | 0.069329 / 0.075469 (-0.006140) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.469586 / 1.841788 (-0.372202) | 16.570305 / 8.074308 (8.495997) | 19.187845 / 10.191392 (8.996453) | 0.219162 / 0.680424 (-0.461262) | 0.026356 / 0.534201 (-0.507845) | 0.447370 / 0.579283 (-0.131913) | 0.555893 / 0.434364 (0.121529) | 0.574958 / 0.540337 (0.034621) | 0.639166 / 1.386936 (-0.747770) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008166 / 0.011353 (-0.003187) | 0.005577 / 0.011008 (-0.005431) | 0.103578 / 0.038508 (0.065070) | 0.040563 / 0.023109 (0.017454) | 0.441996 / 0.275898 (0.166098) | 0.483594 / 0.323480 (0.160114) | 0.007329 / 0.007986 (-0.000657) | 0.004546 / 0.004328 (0.000218) | 0.090471 / 0.004250 (0.086220) | 0.052740 / 0.037052 (0.015688) | 0.442197 / 0.258489 (0.183708) | 0.524310 / 0.293841 (0.230469) | 0.042487 / 0.128546 (-0.086060) | 0.012917 / 0.075646 (-0.062730) | 0.103992 / 0.419271 (-0.315280) | 0.060570 / 0.043533 (0.017037) | 0.441956 / 0.255139 (0.186817) | 0.477084 / 0.283200 (0.193885) | 0.103815 / 0.141683 (-0.037868) | 1.696963 / 1.452155 (0.244809) | 1.747849 / 1.492716 (0.255132) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292465 / 0.018006 (0.274458) | 0.571518 / 0.000490 (0.571028) | 0.000476 / 0.000200 (0.000276) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028697 / 0.037411 (-0.008714) | 0.111671 / 0.014526 (0.097145) | 0.138826 / 0.176557 (-0.037731) | 0.189697 / 0.737135 (-0.547439) | 0.125454 / 0.296338 (-0.170884) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.619273 / 0.215209 (0.404064) | 6.138669 / 2.077655 (4.061015) | 2.558622 / 1.504120 (1.054502) | 2.201550 / 1.541195 (0.660356) | 2.279034 / 1.468490 (0.810544) | 0.850752 / 4.584777 (-3.734025) | 5.438185 / 3.745712 (1.692473) | 2.529343 / 5.269862 (-2.740518) | 1.572178 / 4.565676 (-2.993499) | 0.100768 / 0.424275 (-0.323507) | 0.013902 / 0.007607 (0.006295) | 0.726660 / 0.226044 (0.500616) | 7.794918 / 2.268929 (5.525990) | 3.311695 / 55.444624 (-52.132930) | 2.729167 / 6.876477 (-4.147310) | 2.630984 / 2.142072 (0.488911) | 1.018534 / 4.805227 (-3.786693) | 0.194602 / 6.500664 (-6.306062) | 0.070876 / 0.075469 (-0.004593) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.573005 / 1.841788 (-0.268783) | 17.042710 / 8.074308 (8.968401) | 19.615320 / 10.191392 (9.423928) | 0.229405 / 0.680424 (-0.451019) | 0.027560 / 0.534201 (-0.506641) | 0.447984 / 0.579283 (-0.131299) | 0.598392 / 0.434364 (0.164028) | 0.571769 / 0.540337 (0.031431) | 0.653025 / 1.386936 (-0.733911) |

</details>
</details>

![](https://cml.dev/watermark.png#9dca2ff89a8589595313e9535d16597ce10e3700 "CML watermark")