Title: Lint code with `ruff`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-02-09T17:50:21Z

EDIT:
Use `ruff` for linting instead of `isort` and `flake8` ~~`black`~~ to be consistent with [`transformers`](https://github.com/huggingface/transformers/pull/21480) and [`hfh`](https://github.com/huggingface/huggingface_hub/pull/1323).

TODO:
- [x] ~Merge the community contributors' PR to avoid having to run `make style` on their PR branches~ (we have some new PRs, but fixing those shouldn't be too big of a problem)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009729 / 0.011353 (-0.001624) | 0.005342 / 0.011008 (-0.005666) | 0.100194 / 0.038508 (0.061686) | 0.036391 / 0.023109 (0.013282) | 0.294163 / 0.275898 (0.018264) | 0.364117 / 0.323480 (0.040637) | 0.008231 / 0.007986 (0.000246) | 0.005954 / 0.004328 (0.001626) | 0.076484 / 0.004250 (0.072234) | 0.045028 / 0.037052 (0.007976) | 0.308163 / 0.258489 (0.049674) | 0.339473 / 0.293841 (0.045632) | 0.039268 / 0.128546 (-0.089279) | 0.012357 / 0.075646 (-0.063289) | 0.334176 / 0.419271 (-0.085096) | 0.049502 / 0.043533 (0.005969) | 0.294134 / 0.255139 (0.038995) | 0.319370 / 0.283200 (0.036170) | 0.113040 / 0.141683 (-0.028643) | 1.450750 / 1.452155 (-0.001405) | 1.490265 / 1.492716 (-0.002452) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.252860 / 0.018006 (0.234854) | 0.554299 / 0.000490 (0.553810) | 0.002105 / 0.000200 (0.001905) | 0.000091 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026557 / 0.037411 (-0.010854) | 0.104464 / 0.014526 (0.089938) | 0.116724 / 0.176557 (-0.059833) | 0.154736 / 0.737135 (-0.582399) | 0.122017 / 0.296338 (-0.174322) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398170 / 0.215209 (0.182961) | 3.979309 / 2.077655 (1.901654) | 1.773051 / 1.504120 (0.268931) | 1.587247 / 1.541195 (0.046053) | 1.620446 / 1.468490 (0.151956) | 0.692152 / 4.584777 (-3.892625) | 3.724821 / 3.745712 (-0.020891) | 2.133122 / 5.269862 (-3.136739) | 1.455612 / 4.565676 (-3.110065) | 0.084721 / 0.424275 (-0.339554) | 0.012461 / 0.007607 (0.004854) | 0.498909 / 0.226044 (0.272865) | 4.983837 / 2.268929 (2.714908) | 2.258489 / 55.444624 (-53.186135) | 1.891690 / 6.876477 (-4.984786) | 1.976944 / 2.142072 (-0.165128) | 0.836950 / 4.805227 (-3.968277) | 0.165401 / 6.500664 (-6.335263) | 0.061623 / 0.075469 (-0.013846) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.205945 / 1.841788 (-0.635842) | 15.101603 / 8.074308 (7.027295) | 14.393739 / 10.191392 (4.202347) | 0.176313 / 0.680424 (-0.504110) | 0.029102 / 0.534201 (-0.505099) | 0.439785 / 0.579283 (-0.139498) | 0.437360 / 0.434364 (0.002996) | 0.539668 / 0.540337 (-0.000669) | 0.641452 / 1.386936 (-0.745484) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007184 / 0.011353 (-0.004169) | 0.005215 / 0.011008 (-0.005793) | 0.074617 / 0.038508 (0.036109) | 0.033209 / 0.023109 (0.010100) | 0.334304 / 0.275898 (0.058406) | 0.370270 / 0.323480 (0.046790) | 0.005851 / 0.007986 (-0.002135) | 0.004106 / 0.004328 (-0.000222) | 0.075487 / 0.004250 (0.071237) | 0.051133 / 0.037052 (0.014080) | 0.335401 / 0.258489 (0.076912) | 0.391457 / 0.293841 (0.097616) | 0.036525 / 0.128546 (-0.092021) | 0.012423 / 0.075646 (-0.063223) | 0.086446 / 0.419271 (-0.332825) | 0.050707 / 0.043533 (0.007174) | 0.336186 / 0.255139 (0.081047) | 0.353273 / 0.283200 (0.070074) | 0.105625 / 0.141683 (-0.036057) | 1.486118 / 1.452155 (0.033963) | 1.584931 / 1.492716 (0.092214) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237589 / 0.018006 (0.219583) | 0.552030 / 0.000490 (0.551540) | 0.002863 / 0.000200 (0.002663) | 0.000097 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028078 / 0.037411 (-0.009333) | 0.112516 / 0.014526 (0.097990) | 0.121119 / 0.176557 (-0.055438) | 0.158874 / 0.737135 (-0.578262) | 0.129501 / 0.296338 (-0.166837) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419479 / 0.215209 (0.204270) | 4.192216 / 2.077655 (2.114561) | 1.990513 / 1.504120 (0.486393) | 1.792892 / 1.541195 (0.251697) | 1.853904 / 1.468490 (0.385413) | 0.712702 / 4.584777 (-3.872074) | 3.820682 / 3.745712 (0.074970) | 2.143695 / 5.269862 (-3.126166) | 1.369621 / 4.565676 (-3.196055) | 0.087451 / 0.424275 (-0.336824) | 0.012622 / 0.007607 (0.005014) | 0.521056 / 0.226044 (0.295011) | 5.204873 / 2.268929 (2.935944) | 2.481169 / 55.444624 (-52.963455) | 2.112134 / 6.876477 (-4.764342) | 2.200681 / 2.142072 (0.058609) | 0.860323 / 4.805227 (-3.944904) | 0.171452 / 6.500664 (-6.329212) | 0.065235 / 0.075469 (-0.010234) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.241047 / 1.841788 (-0.600741) | 14.977890 / 8.074308 (6.903582) | 13.584265 / 10.191392 (3.392873) | 0.180050 / 0.680424 (-0.500374) | 0.018247 / 0.534201 (-0.515954) | 0.429585 / 0.579283 (-0.149698) | 0.429448 / 0.434364 (-0.004916) | 0.542663 / 0.540337 (0.002326) | 0.649525 / 1.386936 (-0.737411) |

</details>
</details>

![](https://cml.dev/watermark.png#26cf1d2548eb313a06565d36bd400436e350bc86 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011289 / 0.011353 (-0.000064) | 0.005841 / 0.011008 (-0.005167) | 0.120994 / 0.038508 (0.082486) | 0.043627 / 0.023109 (0.020517) | 0.353254 / 0.275898 (0.077356) | 0.394685 / 0.323480 (0.071205) | 0.009520 / 0.007986 (0.001535) | 0.004770 / 0.004328 (0.000442) | 0.088857 / 0.004250 (0.084607) | 0.048426 / 0.037052 (0.011373) | 0.353815 / 0.258489 (0.095326) | 0.404109 / 0.293841 (0.110268) | 0.060079 / 0.128546 (-0.068467) | 0.013840 / 0.075646 (-0.061806) | 0.403133 / 0.419271 (-0.016139) | 0.072227 / 0.043533 (0.028694) | 0.354585 / 0.255139 (0.099446) | 0.377937 / 0.283200 (0.094737) | 0.139080 / 0.141683 (-0.002602) | 1.733266 / 1.452155 (0.281112) | 1.828402 / 1.492716 (0.335686) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215095 / 0.018006 (0.197088) | 0.486669 / 0.000490 (0.486179) | 0.001425 / 0.000200 (0.001225) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032832 / 0.037411 (-0.004579) | 0.136335 / 0.014526 (0.121809) | 0.141827 / 0.176557 (-0.034730) | 0.185917 / 0.737135 (-0.551218) | 0.149046 / 0.296338 (-0.147293) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474587 / 0.215209 (0.259378) | 4.753686 / 2.077655 (2.676031) | 2.152147 / 1.504120 (0.648027) | 1.941762 / 1.541195 (0.400567) | 2.077493 / 1.468490 (0.609003) | 0.822432 / 4.584777 (-3.762345) | 4.860151 / 3.745712 (1.114439) | 2.527292 / 5.269862 (-2.742569) | 1.580442 / 4.565676 (-2.985234) | 0.102104 / 0.424275 (-0.322171) | 0.015060 / 0.007607 (0.007453) | 0.598780 / 0.226044 (0.372736) | 5.998318 / 2.268929 (3.729390) | 2.754115 / 55.444624 (-52.690509) | 2.317509 / 6.876477 (-4.558967) | 2.409942 / 2.142072 (0.267870) | 1.008830 / 4.805227 (-3.796397) | 0.196203 / 6.500664 (-6.304461) | 0.075378 / 0.075469 (-0.000091) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.430676 / 1.841788 (-0.411112) | 19.597628 / 8.074308 (11.523320) | 17.364673 / 10.191392 (7.173281) | 0.216621 / 0.680424 (-0.463803) | 0.039505 / 0.534201 (-0.494696) | 0.529027 / 0.579283 (-0.050256) | 0.572014 / 0.434364 (0.137650) | 0.702898 / 0.540337 (0.162560) | 0.785748 / 1.386936 (-0.601188) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009150 / 0.011353 (-0.002203) | 0.006088 / 0.011008 (-0.004920) | 0.090629 / 0.038508 (0.052121) | 0.044284 / 0.023109 (0.021174) | 0.411363 / 0.275898 (0.135465) | 0.445499 / 0.323480 (0.122020) | 0.007129 / 0.007986 (-0.000856) | 0.004843 / 0.004328 (0.000515) | 0.087919 / 0.004250 (0.083668) | 0.060329 / 0.037052 (0.023277) | 0.405802 / 0.258489 (0.147313) | 0.468301 / 0.293841 (0.174460) | 0.044271 / 0.128546 (-0.084275) | 0.014895 / 0.075646 (-0.060751) | 0.103728 / 0.419271 (-0.315544) | 0.084190 / 0.043533 (0.040657) | 0.407210 / 0.255139 (0.152071) | 0.432585 / 0.283200 (0.149386) | 0.137132 / 0.141683 (-0.004550) | 1.720261 / 1.452155 (0.268107) | 1.858575 / 1.492716 (0.365858) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.331395 / 0.018006 (0.313389) | 0.494757 / 0.000490 (0.494267) | 0.043426 / 0.000200 (0.043226) | 0.000470 / 0.000054 (0.000415) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035288 / 0.037411 (-0.002123) | 0.140856 / 0.014526 (0.126330) | 0.146597 / 0.176557 (-0.029959) | 0.192775 / 0.737135 (-0.544360) | 0.155307 / 0.296338 (-0.141032) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.504000 / 0.215209 (0.288791) | 5.011081 / 2.077655 (2.933427) | 2.380420 / 1.504120 (0.876300) | 2.154819 / 1.541195 (0.613624) | 2.293883 / 1.468490 (0.825393) | 0.864429 / 4.584777 (-3.720348) | 5.134475 / 3.745712 (1.388763) | 4.984024 / 5.269862 (-0.285837) | 2.333754 / 4.565676 (-2.231923) | 0.105854 / 0.424275 (-0.318422) | 0.015833 / 0.007607 (0.008226) | 0.633614 / 0.226044 (0.407569) | 6.330974 / 2.268929 (4.062046) | 3.020498 / 55.444624 (-52.424126) | 2.578234 / 6.876477 (-4.298243) | 2.654429 / 2.142072 (0.512357) | 1.022041 / 4.805227 (-3.783186) | 0.205085 / 6.500664 (-6.295579) | 0.081122 / 0.075469 (0.005653) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.538929 / 1.841788 (-0.302859) | 19.907799 / 8.074308 (11.833490) | 17.174568 / 10.191392 (6.983176) | 0.228165 / 0.680424 (-0.452258) | 0.024688 / 0.534201 (-0.509513) | 0.508958 / 0.579283 (-0.070326) | 0.544469 / 0.434364 (0.110105) | 0.590805 / 0.540337 (0.050468) | 0.705947 / 1.386936 (-0.680989) |

</details>
</details>

![](https://cml.dev/watermark.png#2573861afb170fd575dbe67270294a4e88ab4be6 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008377 / 0.011353 (-0.002975) | 0.004445 / 0.011008 (-0.006563) | 0.100671 / 0.038508 (0.062163) | 0.029216 / 0.023109 (0.006107) | 0.300311 / 0.275898 (0.024413) | 0.356907 / 0.323480 (0.033427) | 0.006921 / 0.007986 (-0.001065) | 0.003384 / 0.004328 (-0.000944) | 0.078529 / 0.004250 (0.074278) | 0.034689 / 0.037052 (-0.002364) | 0.304647 / 0.258489 (0.046158) | 0.343584 / 0.293841 (0.049743) | 0.032700 / 0.128546 (-0.095846) | 0.011403 / 0.075646 (-0.064244) | 0.321540 / 0.419271 (-0.097732) | 0.040770 / 0.043533 (-0.002762) | 0.306900 / 0.255139 (0.051761) | 0.322482 / 0.283200 (0.039282) | 0.085396 / 0.141683 (-0.056287) | 1.450735 / 1.452155 (-0.001419) | 1.491829 / 1.492716 (-0.000888) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.009439 / 0.018006 (-0.008567) | 0.406805 / 0.000490 (0.406315) | 0.002993 / 0.000200 (0.002793) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025034 / 0.037411 (-0.012378) | 0.100567 / 0.014526 (0.086042) | 0.107267 / 0.176557 (-0.069290) | 0.149945 / 0.737135 (-0.587190) | 0.111150 / 0.296338 (-0.185189) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418387 / 0.215209 (0.203178) | 4.177979 / 2.077655 (2.100324) | 1.886650 / 1.504120 (0.382530) | 1.685692 / 1.541195 (0.144497) | 1.728270 / 1.468490 (0.259780) | 0.700904 / 4.584777 (-3.883873) | 3.379998 / 3.745712 (-0.365714) | 1.874779 / 5.269862 (-3.395083) | 1.170366 / 4.565676 (-3.395310) | 0.083190 / 0.424275 (-0.341085) | 0.012506 / 0.007607 (0.004899) | 0.528633 / 0.226044 (0.302589) | 5.301793 / 2.268929 (3.032865) | 2.334050 / 55.444624 (-53.110574) | 1.986988 / 6.876477 (-4.889488) | 2.020508 / 2.142072 (-0.121565) | 0.817227 / 4.805227 (-3.988000) | 0.150284 / 6.500664 (-6.350380) | 0.065489 / 0.075469 (-0.009980) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.224216 / 1.841788 (-0.617572) | 13.729808 / 8.074308 (5.655500) | 14.283402 / 10.191392 (4.092010) | 0.159434 / 0.680424 (-0.520990) | 0.028471 / 0.534201 (-0.505730) | 0.395102 / 0.579283 (-0.184181) | 0.402733 / 0.434364 (-0.031631) | 0.470852 / 0.540337 (-0.069485) | 0.568530 / 1.386936 (-0.818406) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006750 / 0.011353 (-0.004603) | 0.004479 / 0.011008 (-0.006529) | 0.074926 / 0.038508 (0.036418) | 0.027619 / 0.023109 (0.004510) | 0.342070 / 0.275898 (0.066172) | 0.372452 / 0.323480 (0.048972) | 0.005094 / 0.007986 (-0.002892) | 0.003494 / 0.004328 (-0.000834) | 0.074963 / 0.004250 (0.070713) | 0.038457 / 0.037052 (0.001405) | 0.340587 / 0.258489 (0.082098) | 0.381212 / 0.293841 (0.087371) | 0.031597 / 0.128546 (-0.096950) | 0.011631 / 0.075646 (-0.064015) | 0.084646 / 0.419271 (-0.334626) | 0.042072 / 0.043533 (-0.001461) | 0.340977 / 0.255139 (0.085838) | 0.366502 / 0.283200 (0.083302) | 0.091181 / 0.141683 (-0.050502) | 1.435119 / 1.452155 (-0.017035) | 1.520426 / 1.492716 (0.027710) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211320 / 0.018006 (0.193313) | 0.466154 / 0.000490 (0.465664) | 0.002901 / 0.000200 (0.002701) | 0.000080 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025122 / 0.037411 (-0.012289) | 0.098929 / 0.014526 (0.084403) | 0.106551 / 0.176557 (-0.070005) | 0.142820 / 0.737135 (-0.594316) | 0.110701 / 0.296338 (-0.185637) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445187 / 0.215209 (0.229978) | 4.457524 / 2.077655 (2.379870) | 2.088323 / 1.504120 (0.584203) | 1.888076 / 1.541195 (0.346881) | 1.923340 / 1.468490 (0.454850) | 0.723354 / 4.584777 (-3.861423) | 3.428479 / 3.745712 (-0.317233) | 1.914580 / 5.269862 (-3.355281) | 1.191810 / 4.565676 (-3.373866) | 0.087008 / 0.424275 (-0.337267) | 0.013431 / 0.007607 (0.005824) | 0.545089 / 0.226044 (0.319044) | 5.465887 / 2.268929 (3.196958) | 2.527431 / 55.444624 (-52.917194) | 2.240622 / 6.876477 (-4.635854) | 2.232472 / 2.142072 (0.090399) | 0.815968 / 4.805227 (-3.989259) | 0.152842 / 6.500664 (-6.347822) | 0.067152 / 0.075469 (-0.008317) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.328360 / 1.841788 (-0.513427) | 14.163349 / 8.074308 (6.089040) | 13.814255 / 10.191392 (3.622863) | 0.131684 / 0.680424 (-0.548740) | 0.016980 / 0.534201 (-0.517221) | 0.396045 / 0.579283 (-0.183238) | 0.395078 / 0.434364 (-0.039286) | 0.471728 / 0.540337 (-0.068609) | 0.567830 / 1.386936 (-0.819106) |

</details>
</details>

![](https://cml.dev/watermark.png#82331b032891671c334afe30c5f3cc21245b2d72 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012630 / 0.011353 (0.001277) | 0.007038 / 0.011008 (-0.003970) | 0.158816 / 0.038508 (0.120308) | 0.044142 / 0.023109 (0.021032) | 0.389393 / 0.275898 (0.113495) | 0.479745 / 0.323480 (0.156265) | 0.009335 / 0.007986 (0.001349) | 0.005434 / 0.004328 (0.001105) | 0.107747 / 0.004250 (0.103497) | 0.048382 / 0.037052 (0.011330) | 0.398144 / 0.258489 (0.139655) | 0.446373 / 0.293841 (0.152532) | 0.066285 / 0.128546 (-0.062261) | 0.021174 / 0.075646 (-0.054472) | 0.449176 / 0.419271 (0.029905) | 0.063044 / 0.043533 (0.019511) | 0.390523 / 0.255139 (0.135384) | 0.451435 / 0.283200 (0.168236) | 0.116369 / 0.141683 (-0.025314) | 1.881269 / 1.452155 (0.429114) | 1.944527 / 1.492716 (0.451811) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227989 / 0.018006 (0.209983) | 0.538514 / 0.000490 (0.538024) | 0.009404 / 0.000200 (0.009204) | 0.000510 / 0.000054 (0.000455) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029826 / 0.037411 (-0.007585) | 0.129623 / 0.014526 (0.115098) | 0.142067 / 0.176557 (-0.034489) | 0.218586 / 0.737135 (-0.518549) | 0.160524 / 0.296338 (-0.135814) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.667195 / 0.215209 (0.451986) | 6.694192 / 2.077655 (4.616537) | 2.542493 / 1.504120 (1.038373) | 2.124042 / 1.541195 (0.582847) | 2.024854 / 1.468490 (0.556364) | 1.306222 / 4.584777 (-3.278555) | 5.631557 / 3.745712 (1.885845) | 3.405978 / 5.269862 (-1.863884) | 2.471399 / 4.565676 (-2.094278) | 0.165187 / 0.424275 (-0.259088) | 0.014880 / 0.007607 (0.007273) | 0.842718 / 0.226044 (0.616673) | 8.584358 / 2.268929 (6.315430) | 3.377228 / 55.444624 (-52.067396) | 2.667265 / 6.876477 (-4.209212) | 2.699462 / 2.142072 (0.557389) | 1.623115 / 4.805227 (-3.182112) | 0.253929 / 6.500664 (-6.246735) | 0.077189 / 0.075469 (0.001720) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.778962 / 1.841788 (-0.062825) | 18.997636 / 8.074308 (10.923328) | 24.255222 / 10.191392 (14.063830) | 0.304754 / 0.680424 (-0.375670) | 0.049656 / 0.534201 (-0.484545) | 0.590871 / 0.579283 (0.011588) | 0.649292 / 0.434364 (0.214928) | 0.751281 / 0.540337 (0.210943) | 0.872193 / 1.386936 (-0.514743) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010660 / 0.011353 (-0.000693) | 0.006492 / 0.011008 (-0.004516) | 0.112190 / 0.038508 (0.073682) | 0.045391 / 0.023109 (0.022281) | 0.439852 / 0.275898 (0.163954) | 0.486489 / 0.323480 (0.163009) | 0.007155 / 0.007986 (-0.000830) | 0.006323 / 0.004328 (0.001995) | 0.099775 / 0.004250 (0.095525) | 0.055762 / 0.037052 (0.018709) | 0.439457 / 0.258489 (0.180968) | 0.505322 / 0.293841 (0.211481) | 0.057019 / 0.128546 (-0.071527) | 0.031382 / 0.075646 (-0.044264) | 0.121211 / 0.419271 (-0.298061) | 0.066091 / 0.043533 (0.022558) | 0.499760 / 0.255139 (0.244622) | 0.508312 / 0.283200 (0.225113) | 0.146975 / 0.141683 (0.005292) | 1.916347 / 1.452155 (0.464193) | 2.065860 / 1.492716 (0.573144) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.247176 / 0.018006 (0.229170) | 0.565141 / 0.000490 (0.564652) | 0.004841 / 0.000200 (0.004641) | 0.000141 / 0.000054 (0.000087) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036378 / 0.037411 (-0.001033) | 0.143470 / 0.014526 (0.128944) | 0.148096 / 0.176557 (-0.028461) | 0.225877 / 0.737135 (-0.511258) | 0.147072 / 0.296338 (-0.149266) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.723119 / 0.215209 (0.507910) | 6.824981 / 2.077655 (4.747326) | 2.883840 / 1.504120 (1.379720) | 2.468707 / 1.541195 (0.927513) | 2.525549 / 1.468490 (1.057059) | 1.426640 / 4.584777 (-3.158137) | 5.816045 / 3.745712 (2.070333) | 5.727037 / 5.269862 (0.457175) | 2.650307 / 4.565676 (-1.915369) | 0.160306 / 0.424275 (-0.263970) | 0.015371 / 0.007607 (0.007764) | 0.835778 / 0.226044 (0.609733) | 8.622836 / 2.268929 (6.353907) | 3.616338 / 55.444624 (-51.828287) | 2.974243 / 6.876477 (-3.902234) | 2.884557 / 2.142072 (0.742485) | 1.734874 / 4.805227 (-3.070353) | 0.277474 / 6.500664 (-6.223190) | 0.094189 / 0.075469 (0.018720) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.785728 / 1.841788 (-0.056059) | 19.376490 / 8.074308 (11.302182) | 24.560403 / 10.191392 (14.369011) | 0.250686 / 0.680424 (-0.429738) | 0.034333 / 0.534201 (-0.499868) | 0.557331 / 0.579283 (-0.021952) | 0.641007 / 0.434364 (0.206643) | 0.657138 / 0.540337 (0.116800) | 0.759023 / 1.386936 (-0.627913) |

</details>
</details>

![](https://cml.dev/watermark.png#06ae3f678651bfbb3ca7dd3274ee2f38e0e0237e "CML watermark")


I am editing the title and description of this PR:
- It should be "Lint code" instead of "Format code": formatting was still done with `black`
- This PR uses `ruff` instead of `isort` and `flake8` (not `black`): note that `black` was still used for formatting