Title: Remove token value from warnings
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2023-11-14T17:34:06Z

Fix #6412

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005135 / 0.011353 (-0.006218) | 0.002950 / 0.011008 (-0.008058) | 0.062316 / 0.038508 (0.023808) | 0.030068 / 0.023109 (0.006959) | 0.251998 / 0.275898 (-0.023900) | 0.274806 / 0.323480 (-0.048674) | 0.003067 / 0.007986 (-0.004919) | 0.003082 / 0.004328 (-0.001247) | 0.048503 / 0.004250 (0.044253) | 0.045167 / 0.037052 (0.008114) | 0.254277 / 0.258489 (-0.004212) | 0.290528 / 0.293841 (-0.003313) | 0.023666 / 0.128546 (-0.104880) | 0.007049 / 0.075646 (-0.068597) | 0.202367 / 0.419271 (-0.216905) | 0.056291 / 0.043533 (0.012758) | 0.251923 / 0.255139 (-0.003216) | 0.273595 / 0.283200 (-0.009605) | 0.019065 / 0.141683 (-0.122618) | 1.100832 / 1.452155 (-0.351322) | 1.266758 / 1.492716 (-0.225959) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094311 / 0.018006 (0.076305) | 0.303199 / 0.000490 (0.302709) | 0.000238 / 0.000200 (0.000039) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019413 / 0.037411 (-0.017999) | 0.062618 / 0.014526 (0.048092) | 0.072850 / 0.176557 (-0.103707) | 0.119124 / 0.737135 (-0.618012) | 0.074044 / 0.296338 (-0.222294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.273660 / 0.215209 (0.058451) | 2.682371 / 2.077655 (0.604716) | 1.426041 / 1.504120 (-0.078079) | 1.317186 / 1.541195 (-0.224009) | 1.332385 / 1.468490 (-0.136106) | 0.394599 / 4.584777 (-4.190178) | 2.368167 / 3.745712 (-1.377545) | 2.683728 / 5.269862 (-2.586134) | 1.668348 / 4.565676 (-2.897329) | 0.046177 / 0.424275 (-0.378098) | 0.004833 / 0.007607 (-0.002774) | 0.331413 / 0.226044 (0.105369) | 3.278984 / 2.268929 (1.010055) | 1.797600 / 55.444624 (-53.647024) | 1.492202 / 6.876477 (-5.384274) | 1.536039 / 2.142072 (-0.606034) | 0.470601 / 4.805227 (-4.334626) | 0.100833 / 6.500664 (-6.399831) | 0.042787 / 0.075469 (-0.032682) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.959036 / 1.841788 (-0.882752) | 11.632956 / 8.074308 (3.558648) | 10.384574 / 10.191392 (0.193182) | 0.127477 / 0.680424 (-0.552946) | 0.014072 / 0.534201 (-0.520129) | 0.269534 / 0.579283 (-0.309749) | 0.259753 / 0.434364 (-0.174611) | 0.313450 / 0.540337 (-0.226888) | 0.431799 / 1.386936 (-0.955137) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004964 / 0.011353 (-0.006389) | 0.002906 / 0.011008 (-0.008102) | 0.048145 / 0.038508 (0.009637) | 0.056457 / 0.023109 (0.033348) | 0.274131 / 0.275898 (-0.001767) | 0.298534 / 0.323480 (-0.024946) | 0.004145 / 0.007986 (-0.003841) | 0.002415 / 0.004328 (-0.001913) | 0.048558 / 0.004250 (0.044308) | 0.039031 / 0.037052 (0.001978) | 0.278948 / 0.258489 (0.020459) | 0.312358 / 0.293841 (0.018517) | 0.024902 / 0.128546 (-0.103645) | 0.007286 / 0.075646 (-0.068360) | 0.053839 / 0.419271 (-0.365433) | 0.032510 / 0.043533 (-0.011023) | 0.272023 / 0.255139 (0.016884) | 0.293420 / 0.283200 (0.010221) | 0.018932 / 0.141683 (-0.122750) | 1.122792 / 1.452155 (-0.329362) | 1.167385 / 1.492716 (-0.325331) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094574 / 0.018006 (0.076567) | 0.303810 / 0.000490 (0.303321) | 0.000227 / 0.000200 (0.000027) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021675 / 0.037411 (-0.015737) | 0.070289 / 0.014526 (0.055763) | 0.080345 / 0.176557 (-0.096211) | 0.120220 / 0.737135 (-0.616915) | 0.084080 / 0.296338 (-0.212259) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300134 / 0.215209 (0.084925) | 2.945831 / 2.077655 (0.868176) | 1.605303 / 1.504120 (0.101183) | 1.480135 / 1.541195 (-0.061059) | 1.526039 / 1.468490 (0.057549) | 0.398264 / 4.584777 (-4.186512) | 2.461391 / 3.745712 (-1.284321) | 2.559929 / 5.269862 (-2.709933) | 1.541391 / 4.565676 (-3.024286) | 0.045319 / 0.424275 (-0.378957) | 0.004834 / 0.007607 (-0.002773) | 0.352186 / 0.226044 (0.126141) | 3.500108 / 2.268929 (1.231180) | 1.966394 / 55.444624 (-53.478230) | 1.675500 / 6.876477 (-5.200977) | 1.683134 / 2.142072 (-0.458938) | 0.465085 / 4.805227 (-4.340142) | 0.097235 / 6.500664 (-6.403429) | 0.040764 / 0.075469 (-0.034705) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982813 / 1.841788 (-0.858975) | 12.382529 / 8.074308 (4.308221) | 11.082660 / 10.191392 (0.891268) | 0.129113 / 0.680424 (-0.551310) | 0.015718 / 0.534201 (-0.518483) | 0.272776 / 0.579283 (-0.306507) | 0.275513 / 0.434364 (-0.158850) | 0.304933 / 0.540337 (-0.235404) | 0.414591 / 1.386936 (-0.972345) |

</details>
</details>

![](https://cml.dev/watermark.png#8723b129a64928eba40baf70ffd462060ade9f97 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004400 / 0.011353 (-0.006953) | 0.002580 / 0.011008 (-0.008428) | 0.060975 / 0.038508 (0.022467) | 0.029337 / 0.023109 (0.006228) | 0.248643 / 0.275898 (-0.027255) | 0.274476 / 0.323480 (-0.049004) | 0.003925 / 0.007986 (-0.004061) | 0.002332 / 0.004328 (-0.001997) | 0.049501 / 0.004250 (0.045251) | 0.042730 / 0.037052 (0.005678) | 0.255823 / 0.258489 (-0.002666) | 0.281748 / 0.293841 (-0.012093) | 0.023118 / 0.128546 (-0.105428) | 0.006957 / 0.075646 (-0.068690) | 0.201630 / 0.419271 (-0.217641) | 0.054258 / 0.043533 (0.010725) | 0.252289 / 0.255139 (-0.002850) | 0.267561 / 0.283200 (-0.015639) | 0.016903 / 0.141683 (-0.124780) | 1.104322 / 1.452155 (-0.347833) | 1.160027 / 1.492716 (-0.332689) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096340 / 0.018006 (0.078333) | 0.305187 / 0.000490 (0.304697) | 0.000222 / 0.000200 (0.000022) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018733 / 0.037411 (-0.018678) | 0.062382 / 0.014526 (0.047856) | 0.072309 / 0.176557 (-0.104248) | 0.119772 / 0.737135 (-0.617364) | 0.074655 / 0.296338 (-0.221683) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286150 / 0.215209 (0.070941) | 2.770328 / 2.077655 (0.692673) | 1.494593 / 1.504120 (-0.009527) | 1.358611 / 1.541195 (-0.182583) | 1.396308 / 1.468490 (-0.072182) | 0.394806 / 4.584777 (-4.189971) | 2.349100 / 3.745712 (-1.396613) | 2.600541 / 5.269862 (-2.669321) | 1.568975 / 4.565676 (-2.996701) | 0.046212 / 0.424275 (-0.378063) | 0.004821 / 0.007607 (-0.002786) | 0.332286 / 0.226044 (0.106242) | 3.302643 / 2.268929 (1.033714) | 1.838992 / 55.444624 (-53.605633) | 1.571919 / 6.876477 (-5.304557) | 1.574956 / 2.142072 (-0.567117) | 0.464156 / 4.805227 (-4.341071) | 0.097983 / 6.500664 (-6.402681) | 0.042243 / 0.075469 (-0.033226) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941675 / 1.841788 (-0.900113) | 11.450326 / 8.074308 (3.376017) | 10.169943 / 10.191392 (-0.021449) | 0.137879 / 0.680424 (-0.542545) | 0.013765 / 0.534201 (-0.520436) | 0.268633 / 0.579283 (-0.310650) | 0.265083 / 0.434364 (-0.169281) | 0.302099 / 0.540337 (-0.238238) | 0.423033 / 1.386936 (-0.963903) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004998 / 0.011353 (-0.006355) | 0.003174 / 0.011008 (-0.007834) | 0.047924 / 0.038508 (0.009416) | 0.057598 / 0.023109 (0.034489) | 0.278823 / 0.275898 (0.002925) | 0.334349 / 0.323480 (0.010869) | 0.004053 / 0.007986 (-0.003932) | 0.002554 / 0.004328 (-0.001774) | 0.047797 / 0.004250 (0.043547) | 0.039802 / 0.037052 (0.002749) | 0.278295 / 0.258489 (0.019806) | 0.319597 / 0.293841 (0.025757) | 0.024802 / 0.128546 (-0.103744) | 0.007362 / 0.075646 (-0.068284) | 0.066983 / 0.419271 (-0.352288) | 0.032707 / 0.043533 (-0.010826) | 0.277350 / 0.255139 (0.022211) | 0.296829 / 0.283200 (0.013629) | 0.017902 / 0.141683 (-0.123781) | 1.129765 / 1.452155 (-0.322390) | 1.201940 / 1.492716 (-0.290777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095631 / 0.018006 (0.077625) | 0.296999 / 0.000490 (0.296510) | 0.000234 / 0.000200 (0.000034) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021547 / 0.037411 (-0.015865) | 0.070003 / 0.014526 (0.055477) | 0.083173 / 0.176557 (-0.093384) | 0.121676 / 0.737135 (-0.615459) | 0.082974 / 0.296338 (-0.213364) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298982 / 0.215209 (0.083773) | 2.918666 / 2.077655 (0.841011) | 1.582054 / 1.504120 (0.077934) | 1.463804 / 1.541195 (-0.077391) | 1.484384 / 1.468490 (0.015893) | 0.399443 / 4.584777 (-4.185334) | 2.393515 / 3.745712 (-1.352197) | 2.533004 / 5.269862 (-2.736858) | 1.490411 / 4.565676 (-3.075266) | 0.045274 / 0.424275 (-0.379002) | 0.004783 / 0.007607 (-0.002824) | 0.350510 / 0.226044 (0.124465) | 3.437927 / 2.268929 (1.168998) | 1.940115 / 55.444624 (-53.504509) | 1.662025 / 6.876477 (-5.214452) | 1.640621 / 2.142072 (-0.501452) | 0.464014 / 4.805227 (-4.341214) | 0.095506 / 6.500664 (-6.405158) | 0.040172 / 0.075469 (-0.035297) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975618 / 1.841788 (-0.866169) | 12.561067 / 8.074308 (4.486759) | 11.408037 / 10.191392 (1.216645) | 0.130699 / 0.680424 (-0.549725) | 0.016796 / 0.534201 (-0.517405) | 0.271130 / 0.579283 (-0.308153) | 0.283506 / 0.434364 (-0.150857) | 0.304482 / 0.540337 (-0.235856) | 0.413673 / 1.386936 (-0.973263) |

</details>
</details>

![](https://cml.dev/watermark.png#723038a73248dd12dc0673d2b341e9295c441ea3 "CML watermark")