Title: Avoid redundant warning when encoding NumPy array as `Image`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-11-02T16:37:58Z

Avoid a redundant warning in `encode_np_array` by removing the identity check as NumPy `dtype`s can be equal without having identical `id`s.

Additionally, fix "unreachable" checks in `encode_np_array`.

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008649 / 0.011353 (-0.002704) | 0.005754 / 0.011008 (-0.005254) | 0.101992 / 0.038508 (0.063484) | 0.084932 / 0.023109 (0.061823) | 0.393928 / 0.275898 (0.118030) | 0.414059 / 0.323480 (0.090579) | 0.006564 / 0.007986 (-0.001422) | 0.004746 / 0.004328 (0.000418) | 0.078624 / 0.004250 (0.074373) | 0.060465 / 0.037052 (0.023412) | 0.420767 / 0.258489 (0.162278) | 0.497797 / 0.293841 (0.203956) | 0.047031 / 0.128546 (-0.081516) | 0.014316 / 0.075646 (-0.061330) | 0.340347 / 0.419271 (-0.078925) | 0.067126 / 0.043533 (0.023593) | 0.390806 / 0.255139 (0.135667) | 0.413711 / 0.283200 (0.130512) | 0.037838 / 0.141683 (-0.103845) | 1.713547 / 1.452155 (0.261393) | 1.825591 / 1.492716 (0.332874) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316357 / 0.018006 (0.298350) | 0.594279 / 0.000490 (0.593789) | 0.013659 / 0.000200 (0.013459) | 0.000547 / 0.000054 (0.000492) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031310 / 0.037411 (-0.006101) | 0.090410 / 0.014526 (0.075884) | 0.114620 / 0.176557 (-0.061936) | 0.183036 / 0.737135 (-0.554099) | 0.112700 / 0.296338 (-0.183638) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.582424 / 0.215209 (0.367215) | 5.670424 / 2.077655 (3.592769) | 2.444326 / 1.504120 (0.940206) | 2.108555 / 1.541195 (0.567360) | 2.091594 / 1.468490 (0.623104) | 0.839067 / 4.584777 (-3.745710) | 5.280942 / 3.745712 (1.535230) | 4.611059 / 5.269862 (-0.658803) | 2.911145 / 4.565676 (-1.654531) | 0.091929 / 0.424275 (-0.332346) | 0.008774 / 0.007607 (0.001167) | 0.657948 / 0.226044 (0.431904) | 6.816300 / 2.268929 (4.547371) | 3.232260 / 55.444624 (-52.212364) | 2.479626 / 6.876477 (-4.396851) | 2.497886 / 2.142072 (0.355813) | 0.959160 / 4.805227 (-3.846068) | 0.222306 / 6.500664 (-6.278358) | 0.072962 / 0.075469 (-0.002507) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.580415 / 1.841788 (-0.261372) | 23.689597 / 8.074308 (15.615289) | 20.430709 / 10.191392 (10.239317) | 0.237891 / 0.680424 (-0.442533) | 0.028194 / 0.534201 (-0.506007) | 0.464915 / 0.579283 (-0.114368) | 0.611512 / 0.434364 (0.177148) | 0.556564 / 0.540337 (0.016227) | 0.811075 / 1.386936 (-0.575861) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008703 / 0.011353 (-0.002649) | 0.005030 / 0.011008 (-0.005978) | 0.079251 / 0.038508 (0.040743) | 0.079054 / 0.023109 (0.055945) | 0.440220 / 0.275898 (0.164322) | 0.479824 / 0.323480 (0.156344) | 0.006312 / 0.007986 (-0.001673) | 0.004506 / 0.004328 (0.000177) | 0.078454 / 0.004250 (0.074203) | 0.061041 / 0.037052 (0.023989) | 0.490104 / 0.258489 (0.231615) | 0.480925 / 0.293841 (0.187084) | 0.049601 / 0.128546 (-0.078945) | 0.013114 / 0.075646 (-0.062532) | 0.092576 / 0.419271 (-0.326696) | 0.059516 / 0.043533 (0.015983) | 0.433728 / 0.255139 (0.178589) | 0.490039 / 0.283200 (0.206839) | 0.035359 / 0.141683 (-0.106324) | 1.823618 / 1.452155 (0.371463) | 1.980894 / 1.492716 (0.488178) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284679 / 0.018006 (0.266673) | 0.606623 / 0.000490 (0.606133) | 0.007531 / 0.000200 (0.007331) | 0.000109 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033261 / 0.037411 (-0.004150) | 0.102908 / 0.014526 (0.088382) | 0.123912 / 0.176557 (-0.052644) | 0.169893 / 0.737135 (-0.567242) | 0.115366 / 0.296338 (-0.180973) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.598239 / 0.215209 (0.383030) | 6.003464 / 2.077655 (3.925809) | 2.828483 / 1.504120 (1.324363) | 2.485996 / 1.541195 (0.944802) | 2.434986 / 1.468490 (0.966496) | 0.832718 / 4.584777 (-3.752058) | 5.327407 / 3.745712 (1.581694) | 4.732271 / 5.269862 (-0.537590) | 3.047555 / 4.565676 (-1.518121) | 0.103576 / 0.424275 (-0.320699) | 0.009795 / 0.007607 (0.002188) | 0.755443 / 0.226044 (0.529399) | 7.465857 / 2.268929 (5.196928) | 3.564923 / 55.444624 (-51.879701) | 2.740483 / 6.876477 (-4.135994) | 3.044993 / 2.142072 (0.902920) | 1.012925 / 4.805227 (-3.792302) | 0.207498 / 6.500664 (-6.293167) | 0.073361 / 0.075469 (-0.002108) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.704988 / 1.841788 (-0.136800) | 24.669992 / 8.074308 (16.595684) | 21.103096 / 10.191392 (10.911704) | 0.253759 / 0.680424 (-0.426665) | 0.040109 / 0.534201 (-0.494092) | 0.465646 / 0.579283 (-0.113637) | 0.619696 / 0.434364 (0.185332) | 0.552228 / 0.540337 (0.011890) | 0.794907 / 1.386936 (-0.592029) |

</details>
</details>

![](https://cml.dev/watermark.png#85bba8991f6a2d9ed9fd4769d945eeaf318d3aa6 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006347 / 0.011353 (-0.005006) | 0.003725 / 0.011008 (-0.007283) | 0.080233 / 0.038508 (0.041725) | 0.061013 / 0.023109 (0.037904) | 0.390046 / 0.275898 (0.114148) | 0.420526 / 0.323480 (0.097046) | 0.003579 / 0.007986 (-0.004407) | 0.002837 / 0.004328 (-0.001491) | 0.062929 / 0.004250 (0.058678) | 0.048781 / 0.037052 (0.011729) | 0.400722 / 0.258489 (0.142233) | 0.435022 / 0.293841 (0.141182) | 0.027560 / 0.128546 (-0.100986) | 0.007981 / 0.075646 (-0.067666) | 0.262838 / 0.419271 (-0.156433) | 0.045480 / 0.043533 (0.001947) | 0.394443 / 0.255139 (0.139304) | 0.413828 / 0.283200 (0.130628) | 0.023375 / 0.141683 (-0.118307) | 1.412865 / 1.452155 (-0.039290) | 1.495761 / 1.492716 (0.003044) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224876 / 0.018006 (0.206870) | 0.424234 / 0.000490 (0.423745) | 0.007502 / 0.000200 (0.007302) | 0.000220 / 0.000054 (0.000166) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024246 / 0.037411 (-0.013165) | 0.073982 / 0.014526 (0.059456) | 0.082704 / 0.176557 (-0.093852) | 0.143137 / 0.737135 (-0.593998) | 0.083398 / 0.296338 (-0.212941) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400220 / 0.215209 (0.185010) | 3.973037 / 2.077655 (1.895382) | 2.025903 / 1.504120 (0.521783) | 1.912888 / 1.541195 (0.371693) | 1.999578 / 1.468490 (0.531088) | 0.499378 / 4.584777 (-4.085399) | 3.025715 / 3.745712 (-0.719997) | 2.992338 / 5.269862 (-2.277524) | 1.851155 / 4.565676 (-2.714522) | 0.057528 / 0.424275 (-0.366747) | 0.006802 / 0.007607 (-0.000805) | 0.469516 / 0.226044 (0.243471) | 4.675630 / 2.268929 (2.406702) | 2.472166 / 55.444624 (-52.972458) | 2.238052 / 6.876477 (-4.638424) | 2.288255 / 2.142072 (0.146183) | 0.584906 / 4.805227 (-4.220321) | 0.125902 / 6.500664 (-6.374762) | 0.060681 / 0.075469 (-0.014788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.236383 / 1.841788 (-0.605404) | 17.554238 / 8.074308 (9.479930) | 13.749298 / 10.191392 (3.557906) | 0.144715 / 0.680424 (-0.535708) | 0.017449 / 0.534201 (-0.516752) | 0.334831 / 0.579283 (-0.244452) | 0.362660 / 0.434364 (-0.071704) | 0.385295 / 0.540337 (-0.155043) | 0.541173 / 1.386936 (-0.845763) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006118 / 0.011353 (-0.005235) | 0.003660 / 0.011008 (-0.007348) | 0.062373 / 0.038508 (0.023865) | 0.063404 / 0.023109 (0.040295) | 0.354149 / 0.275898 (0.078251) | 0.410324 / 0.323480 (0.086844) | 0.004826 / 0.007986 (-0.003160) | 0.002881 / 0.004328 (-0.001448) | 0.061631 / 0.004250 (0.057381) | 0.048052 / 0.037052 (0.010999) | 0.352905 / 0.258489 (0.094416) | 0.400096 / 0.293841 (0.106255) | 0.028472 / 0.128546 (-0.100075) | 0.008076 / 0.075646 (-0.067571) | 0.067910 / 0.419271 (-0.351362) | 0.040671 / 0.043533 (-0.002862) | 0.352131 / 0.255139 (0.096992) | 0.402140 / 0.283200 (0.118940) | 0.020065 / 0.141683 (-0.121618) | 1.456938 / 1.452155 (0.004783) | 1.506484 / 1.492716 (0.013767) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222295 / 0.018006 (0.204288) | 0.416672 / 0.000490 (0.416183) | 0.003015 / 0.000200 (0.002815) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026428 / 0.037411 (-0.010983) | 0.080072 / 0.014526 (0.065547) | 0.089992 / 0.176557 (-0.086564) | 0.141739 / 0.737135 (-0.595397) | 0.092281 / 0.296338 (-0.204058) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417758 / 0.215209 (0.202549) | 4.175673 / 2.077655 (2.098018) | 2.262369 / 1.504120 (0.758249) | 2.100440 / 1.541195 (0.559246) | 2.075827 / 1.468490 (0.607337) | 0.505673 / 4.584777 (-4.079104) | 3.129020 / 3.745712 (-0.616692) | 2.843255 / 5.269862 (-2.426607) | 1.853288 / 4.565676 (-2.712389) | 0.058337 / 0.424275 (-0.365938) | 0.006461 / 0.007607 (-0.001147) | 0.491797 / 0.226044 (0.265753) | 4.933327 / 2.268929 (2.664399) | 2.675374 / 55.444624 (-52.769250) | 2.358103 / 6.876477 (-4.518374) | 2.540436 / 2.142072 (0.398363) | 0.591550 / 4.805227 (-4.213677) | 0.121572 / 6.500664 (-6.379092) | 0.057311 / 0.075469 (-0.018158) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.365368 / 1.841788 (-0.476419) | 17.763413 / 8.074308 (9.689105) | 14.368754 / 10.191392 (4.177362) | 0.132979 / 0.680424 (-0.547445) | 0.017957 / 0.534201 (-0.516244) | 0.334035 / 0.579283 (-0.245248) | 0.385349 / 0.434364 (-0.049015) | 0.392636 / 0.540337 (-0.147702) | 0.537957 / 1.386936 (-0.848979) |

</details>
</details>

![](https://cml.dev/watermark.png#92503c94839b31125b4d5288d0a49d81b9b9b3cc "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008053 / 0.011353 (-0.003300) | 0.004966 / 0.011008 (-0.006043) | 0.102219 / 0.038508 (0.063711) | 0.099319 / 0.023109 (0.076210) | 0.418458 / 0.275898 (0.142559) | 0.459344 / 0.323480 (0.135864) | 0.004756 / 0.007986 (-0.003229) | 0.003940 / 0.004328 (-0.000388) | 0.076824 / 0.004250 (0.072573) | 0.068090 / 0.037052 (0.031038) | 0.428689 / 0.258489 (0.170200) | 0.476153 / 0.293841 (0.182312) | 0.036927 / 0.128546 (-0.091619) | 0.010232 / 0.075646 (-0.065414) | 0.345126 / 0.419271 (-0.074145) | 0.063182 / 0.043533 (0.019649) | 0.416633 / 0.255139 (0.161494) | 0.437418 / 0.283200 (0.154218) | 0.028192 / 0.141683 (-0.113491) | 1.768869 / 1.452155 (0.316715) | 1.847022 / 1.492716 (0.354306) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269997 / 0.018006 (0.251991) | 0.544246 / 0.000490 (0.543756) | 0.012940 / 0.000200 (0.012740) | 0.000754 / 0.000054 (0.000699) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035570 / 0.037411 (-0.001842) | 0.104318 / 0.014526 (0.089792) | 0.115263 / 0.176557 (-0.061294) | 0.184693 / 0.737135 (-0.552442) | 0.116023 / 0.296338 (-0.180315) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.472361 / 0.215209 (0.257152) | 4.714327 / 2.077655 (2.636673) | 2.405434 / 1.504120 (0.901314) | 2.197871 / 1.541195 (0.656677) | 2.312901 / 1.468490 (0.844411) | 0.569736 / 4.584777 (-4.015041) | 4.600008 / 3.745712 (0.854296) | 4.127967 / 5.269862 (-1.141895) | 2.462232 / 4.565676 (-2.103445) | 0.067759 / 0.424275 (-0.356516) | 0.009277 / 0.007607 (0.001670) | 0.569658 / 0.226044 (0.343614) | 5.694050 / 2.268929 (3.425121) | 3.041495 / 55.444624 (-52.403129) | 2.688418 / 6.876477 (-4.188059) | 2.762175 / 2.142072 (0.620102) | 0.683250 / 4.805227 (-4.121977) | 0.158772 / 6.500664 (-6.341892) | 0.073364 / 0.075469 (-0.002105) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.627241 / 1.841788 (-0.214547) | 23.054465 / 8.074308 (14.980157) | 17.122451 / 10.191392 (6.931059) | 0.170272 / 0.680424 (-0.510152) | 0.021678 / 0.534201 (-0.512523) | 0.467301 / 0.579283 (-0.111982) | 0.509480 / 0.434364 (0.075116) | 0.555077 / 0.540337 (0.014740) | 0.816199 / 1.386936 (-0.570737) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008499 / 0.011353 (-0.002854) | 0.004724 / 0.011008 (-0.006284) | 0.077519 / 0.038508 (0.039011) | 0.103237 / 0.023109 (0.080127) | 0.447470 / 0.275898 (0.171572) | 0.484778 / 0.323480 (0.161298) | 0.006475 / 0.007986 (-0.001511) | 0.003946 / 0.004328 (-0.000383) | 0.075596 / 0.004250 (0.071346) | 0.069265 / 0.037052 (0.032213) | 0.454185 / 0.258489 (0.195696) | 0.491039 / 0.293841 (0.197198) | 0.038611 / 0.128546 (-0.089935) | 0.009889 / 0.075646 (-0.065758) | 0.084012 / 0.419271 (-0.335260) | 0.057265 / 0.043533 (0.013732) | 0.448622 / 0.255139 (0.193483) | 0.470961 / 0.283200 (0.187762) | 0.029220 / 0.141683 (-0.112463) | 1.773347 / 1.452155 (0.321192) | 1.872669 / 1.492716 (0.379953) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.272429 / 0.018006 (0.254423) | 0.569907 / 0.000490 (0.569418) | 0.013359 / 0.000200 (0.013159) | 0.000187 / 0.000054 (0.000133) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038784 / 0.037411 (0.001373) | 0.114958 / 0.014526 (0.100432) | 0.132745 / 0.176557 (-0.043811) | 0.186283 / 0.737135 (-0.550852) | 0.126652 / 0.296338 (-0.169686) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.482753 / 0.215209 (0.267544) | 4.827287 / 2.077655 (2.749633) | 2.539959 / 1.504120 (1.035839) | 2.348483 / 1.541195 (0.807288) | 2.421739 / 1.468490 (0.953249) | 0.586064 / 4.584777 (-3.998713) | 4.579865 / 3.745712 (0.834152) | 3.950617 / 5.269862 (-1.319244) | 2.528447 / 4.565676 (-2.037229) | 0.070280 / 0.424275 (-0.353995) | 0.008801 / 0.007607 (0.001194) | 0.568857 / 0.226044 (0.342812) | 5.692739 / 2.268929 (3.423810) | 3.192045 / 55.444624 (-52.252579) | 2.768092 / 6.876477 (-4.108384) | 3.002934 / 2.142072 (0.860862) | 0.701887 / 4.805227 (-4.103340) | 0.155563 / 6.500664 (-6.345102) | 0.069397 / 0.075469 (-0.006072) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.607991 / 1.841788 (-0.233796) | 24.658060 / 8.074308 (16.583752) | 17.616229 / 10.191392 (7.424837) | 0.209730 / 0.680424 (-0.470693) | 0.024052 / 0.534201 (-0.510149) | 0.476648 / 0.579283 (-0.102635) | 0.534452 / 0.434364 (0.100089) | 0.567702 / 0.540337 (0.027365) | 0.772933 / 1.386936 (-0.614003) |

</details>
</details>

![](https://cml.dev/watermark.png#a49e78ede85c2a680adddacbb6b9638cba4062f3 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004684 / 0.011353 (-0.006669) | 0.002944 / 0.011008 (-0.008064) | 0.063065 / 0.038508 (0.024557) | 0.051627 / 0.023109 (0.028518) | 0.243485 / 0.275898 (-0.032413) | 0.275144 / 0.323480 (-0.048336) | 0.002934 / 0.007986 (-0.005052) | 0.002395 / 0.004328 (-0.001934) | 0.048579 / 0.004250 (0.044328) | 0.038940 / 0.037052 (0.001887) | 0.250244 / 0.258489 (-0.008245) | 0.287404 / 0.293841 (-0.006437) | 0.022958 / 0.128546 (-0.105588) | 0.007189 / 0.075646 (-0.068458) | 0.202483 / 0.419271 (-0.216788) | 0.035477 / 0.043533 (-0.008056) | 0.243793 / 0.255139 (-0.011346) | 0.265990 / 0.283200 (-0.017209) | 0.019675 / 0.141683 (-0.122008) | 1.119127 / 1.452155 (-0.333028) | 1.183230 / 1.492716 (-0.309486) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097090 / 0.018006 (0.079084) | 0.305815 / 0.000490 (0.305325) | 0.000228 / 0.000200 (0.000028) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019233 / 0.037411 (-0.018178) | 0.061743 / 0.014526 (0.047217) | 0.077033 / 0.176557 (-0.099524) | 0.119786 / 0.737135 (-0.617349) | 0.074740 / 0.296338 (-0.221598) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284361 / 0.215209 (0.069152) | 2.761501 / 2.077655 (0.683846) | 1.464980 / 1.504120 (-0.039140) | 1.348026 / 1.541195 (-0.193169) | 1.362690 / 1.468490 (-0.105800) | 0.392022 / 4.584777 (-4.192755) | 2.401330 / 3.745712 (-1.344382) | 2.618999 / 5.269862 (-2.650863) | 1.599526 / 4.565676 (-2.966150) | 0.045621 / 0.424275 (-0.378654) | 0.005153 / 0.007607 (-0.002454) | 0.337279 / 0.226044 (0.111234) | 3.330135 / 2.268929 (1.061206) | 1.803544 / 55.444624 (-53.641081) | 1.515545 / 6.876477 (-5.360932) | 1.561745 / 2.142072 (-0.580327) | 0.468735 / 4.805227 (-4.336492) | 0.098882 / 6.500664 (-6.401782) | 0.042923 / 0.075469 (-0.032546) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.961106 / 1.841788 (-0.880682) | 12.030489 / 8.074308 (3.956181) | 10.824166 / 10.191392 (0.632774) | 0.132135 / 0.680424 (-0.548289) | 0.015320 / 0.534201 (-0.518881) | 0.269691 / 0.579283 (-0.309592) | 0.270700 / 0.434364 (-0.163664) | 0.308317 / 0.540337 (-0.232020) | 0.397871 / 1.386936 (-0.989065) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004859 / 0.011353 (-0.006494) | 0.003400 / 0.011008 (-0.007609) | 0.048095 / 0.038508 (0.009587) | 0.054885 / 0.023109 (0.031776) | 0.276976 / 0.275898 (0.001078) | 0.302298 / 0.323480 (-0.021182) | 0.004084 / 0.007986 (-0.003902) | 0.002647 / 0.004328 (-0.001681) | 0.048570 / 0.004250 (0.044319) | 0.040683 / 0.037052 (0.003631) | 0.279828 / 0.258489 (0.021339) | 0.306037 / 0.293841 (0.012196) | 0.024263 / 0.128546 (-0.104283) | 0.007336 / 0.075646 (-0.068310) | 0.053768 / 0.419271 (-0.365503) | 0.032284 / 0.043533 (-0.011248) | 0.276706 / 0.255139 (0.021567) | 0.294706 / 0.283200 (0.011506) | 0.018092 / 0.141683 (-0.123591) | 1.153430 / 1.452155 (-0.298725) | 1.208783 / 1.492716 (-0.283933) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096946 / 0.018006 (0.078939) | 0.308118 / 0.000490 (0.307628) | 0.000234 / 0.000200 (0.000034) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021834 / 0.037411 (-0.015577) | 0.070934 / 0.014526 (0.056408) | 0.080310 / 0.176557 (-0.096247) | 0.123299 / 0.737135 (-0.613836) | 0.081591 / 0.296338 (-0.214748) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302242 / 0.215209 (0.087033) | 2.934477 / 2.077655 (0.856822) | 1.623768 / 1.504120 (0.119648) | 1.493868 / 1.541195 (-0.047326) | 1.516553 / 1.468490 (0.048063) | 0.410319 / 4.584777 (-4.174458) | 2.471346 / 3.745712 (-1.274366) | 2.667371 / 5.269862 (-2.602491) | 1.625390 / 4.565676 (-2.940286) | 0.046465 / 0.424275 (-0.377810) | 0.004867 / 0.007607 (-0.002740) | 0.355516 / 0.226044 (0.129471) | 3.442294 / 2.268929 (1.173365) | 1.973859 / 55.444624 (-53.470765) | 1.682089 / 6.876477 (-5.194388) | 1.865253 / 2.142072 (-0.276819) | 0.475750 / 4.805227 (-4.329477) | 0.098298 / 6.500664 (-6.402366) | 0.041025 / 0.075469 (-0.034445) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969864 / 1.841788 (-0.871924) | 12.437806 / 8.074308 (4.363498) | 10.461262 / 10.191392 (0.269870) | 0.131051 / 0.680424 (-0.549373) | 0.016232 / 0.534201 (-0.517969) | 0.273968 / 0.579283 (-0.305315) | 0.285369 / 0.434364 (-0.148995) | 0.309046 / 0.540337 (-0.231291) | 0.398776 / 1.386936 (-0.988160) |

</details>
</details>

![](https://cml.dev/watermark.png#a49e78ede85c2a680adddacbb6b9638cba4062f3 "CML watermark")