Title: Don't require trust_remote_code in inspect_dataset
Author: lhoestq (MEMBER)
State: closed
Created: 2023-11-28T19:47:07Z

don't require `trust_remote_code` in (deprecated) `inspect_dataset` (it defeats its purpose)

(not super important but we might as well keep it until the next major release)

this is needed to fix the tests in https://github.com/huggingface/datasets/pull/6448

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005705 / 0.011353 (-0.005648) | 0.003536 / 0.011008 (-0.007473) | 0.062852 / 0.038508 (0.024343) | 0.053902 / 0.023109 (0.030793) | 0.239465 / 0.275898 (-0.036433) | 0.270829 / 0.323480 (-0.052651) | 0.004052 / 0.007986 (-0.003934) | 0.002775 / 0.004328 (-0.001554) | 0.048475 / 0.004250 (0.044225) | 0.039430 / 0.037052 (0.002377) | 0.244318 / 0.258489 (-0.014171) | 0.277539 / 0.293841 (-0.016302) | 0.027637 / 0.128546 (-0.100909) | 0.010875 / 0.075646 (-0.064771) | 0.208839 / 0.419271 (-0.210432) | 0.036984 / 0.043533 (-0.006549) | 0.246355 / 0.255139 (-0.008784) | 0.271200 / 0.283200 (-0.011999) | 0.020636 / 0.141683 (-0.121047) | 1.078472 / 1.452155 (-0.373683) | 1.155701 / 1.492716 (-0.337015) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100971 / 0.018006 (0.082965) | 0.310996 / 0.000490 (0.310507) | 0.000218 / 0.000200 (0.000018) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019300 / 0.037411 (-0.018111) | 0.060625 / 0.014526 (0.046099) | 0.073778 / 0.176557 (-0.102778) | 0.120280 / 0.737135 (-0.616855) | 0.075288 / 0.296338 (-0.221051) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289838 / 0.215209 (0.074629) | 2.859492 / 2.077655 (0.781837) | 1.528478 / 1.504120 (0.024358) | 1.417911 / 1.541195 (-0.123283) | 1.444227 / 1.468490 (-0.024263) | 0.566799 / 4.584777 (-4.017978) | 2.402526 / 3.745712 (-1.343186) | 2.805241 / 5.269862 (-2.464620) | 1.798572 / 4.565676 (-2.767104) | 0.062920 / 0.424275 (-0.361355) | 0.004995 / 0.007607 (-0.002612) | 0.340688 / 0.226044 (0.114644) | 3.347967 / 2.268929 (1.079039) | 1.898464 / 55.444624 (-53.546160) | 1.604784 / 6.876477 (-5.271693) | 1.648864 / 2.142072 (-0.493209) | 0.642242 / 4.805227 (-4.162985) | 0.117567 / 6.500664 (-6.383097) | 0.041911 / 0.075469 (-0.033558) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949099 / 1.841788 (-0.892689) | 12.367323 / 8.074308 (4.293015) | 10.694238 / 10.191392 (0.502846) | 0.143424 / 0.680424 (-0.537000) | 0.014569 / 0.534201 (-0.519632) | 0.289127 / 0.579283 (-0.290156) | 0.270490 / 0.434364 (-0.163874) | 0.326470 / 0.540337 (-0.213867) | 0.432223 / 1.386936 (-0.954713) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005380 / 0.011353 (-0.005973) | 0.003582 / 0.011008 (-0.007426) | 0.049341 / 0.038508 (0.010833) | 0.053274 / 0.023109 (0.030165) | 0.284319 / 0.275898 (0.008421) | 0.334248 / 0.323480 (0.010768) | 0.004032 / 0.007986 (-0.003953) | 0.002682 / 0.004328 (-0.001646) | 0.048317 / 0.004250 (0.044067) | 0.040157 / 0.037052 (0.003105) | 0.284594 / 0.258489 (0.026105) | 0.341567 / 0.293841 (0.047726) | 0.029639 / 0.128546 (-0.098908) | 0.010780 / 0.075646 (-0.064867) | 0.057990 / 0.419271 (-0.361282) | 0.032730 / 0.043533 (-0.010803) | 0.290328 / 0.255139 (0.035189) | 0.298563 / 0.283200 (0.015363) | 0.018546 / 0.141683 (-0.123137) | 1.143157 / 1.452155 (-0.308998) | 1.191391 / 1.492716 (-0.301326) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093802 / 0.018006 (0.075796) | 0.312771 / 0.000490 (0.312282) | 0.000221 / 0.000200 (0.000021) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021867 / 0.037411 (-0.015544) | 0.069064 / 0.014526 (0.054538) | 0.082270 / 0.176557 (-0.094287) | 0.120222 / 0.737135 (-0.616913) | 0.084628 / 0.296338 (-0.211710) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295505 / 0.215209 (0.080296) | 2.891105 / 2.077655 (0.813450) | 1.619480 / 1.504120 (0.115360) | 1.498290 / 1.541195 (-0.042905) | 1.547896 / 1.468490 (0.079406) | 0.575188 / 4.584777 (-4.009589) | 2.434426 / 3.745712 (-1.311286) | 2.899286 / 5.269862 (-2.370576) | 1.806085 / 4.565676 (-2.759591) | 0.063660 / 0.424275 (-0.360616) | 0.004933 / 0.007607 (-0.002674) | 0.348274 / 0.226044 (0.122229) | 3.447900 / 2.268929 (1.178971) | 1.956237 / 55.444624 (-53.488387) | 1.680416 / 6.876477 (-5.196061) | 1.732307 / 2.142072 (-0.409766) | 0.668428 / 4.805227 (-4.136799) | 0.119161 / 6.500664 (-6.381503) | 0.041694 / 0.075469 (-0.033775) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.973730 / 1.841788 (-0.868058) | 12.082452 / 8.074308 (4.008144) | 10.624836 / 10.191392 (0.433444) | 0.144027 / 0.680424 (-0.536397) | 0.014830 / 0.534201 (-0.519370) | 0.289946 / 0.579283 (-0.289337) | 0.281939 / 0.434364 (-0.152424) | 0.325639 / 0.540337 (-0.214699) | 0.551690 / 1.386936 (-0.835246) |

</details>
</details>

![](https://cml.dev/watermark.png#9e1cf8526c9216b08b5431695d9f8e0eec64cc5f "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005279 / 0.011353 (-0.006074) | 0.003506 / 0.011008 (-0.007502) | 0.062579 / 0.038508 (0.024071) | 0.052809 / 0.023109 (0.029700) | 0.274693 / 0.275898 (-0.001205) | 0.283917 / 0.323480 (-0.039563) | 0.003950 / 0.007986 (-0.004036) | 0.002772 / 0.004328 (-0.001557) | 0.048127 / 0.004250 (0.043877) | 0.037771 / 0.037052 (0.000719) | 0.280595 / 0.258489 (0.022106) | 0.292310 / 0.293841 (-0.001531) | 0.027890 / 0.128546 (-0.100656) | 0.010771 / 0.075646 (-0.064875) | 0.207285 / 0.419271 (-0.211987) | 0.036179 / 0.043533 (-0.007354) | 0.253617 / 0.255139 (-0.001522) | 0.276107 / 0.283200 (-0.007093) | 0.018253 / 0.141683 (-0.123430) | 1.112219 / 1.452155 (-0.339936) | 1.166756 / 1.492716 (-0.325960) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095159 / 0.018006 (0.077152) | 0.306097 / 0.000490 (0.305608) | 0.000219 / 0.000200 (0.000019) | 0.000042 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019056 / 0.037411 (-0.018355) | 0.060445 / 0.014526 (0.045919) | 0.073553 / 0.176557 (-0.103004) | 0.120306 / 0.737135 (-0.616829) | 0.075613 / 0.296338 (-0.220725) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277839 / 0.215209 (0.062630) | 2.761037 / 2.077655 (0.683382) | 1.508524 / 1.504120 (0.004404) | 1.368994 / 1.541195 (-0.172201) | 1.415961 / 1.468490 (-0.052529) | 0.570490 / 4.584777 (-4.014287) | 2.356355 / 3.745712 (-1.389357) | 2.806626 / 5.269862 (-2.463235) | 1.757849 / 4.565676 (-2.807827) | 0.063504 / 0.424275 (-0.360771) | 0.005021 / 0.007607 (-0.002586) | 0.338880 / 0.226044 (0.112836) | 3.290947 / 2.268929 (1.022018) | 1.818238 / 55.444624 (-53.626386) | 1.529970 / 6.876477 (-5.346507) | 1.557085 / 2.142072 (-0.584987) | 0.645352 / 4.805227 (-4.159876) | 0.123066 / 6.500664 (-6.377598) | 0.043387 / 0.075469 (-0.032082) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974512 / 1.841788 (-0.867276) | 11.976411 / 8.074308 (3.902103) | 10.361084 / 10.191392 (0.169692) | 0.127171 / 0.680424 (-0.553253) | 0.014091 / 0.534201 (-0.520110) | 0.288608 / 0.579283 (-0.290675) | 0.261886 / 0.434364 (-0.172478) | 0.331632 / 0.540337 (-0.208705) | 0.437002 / 1.386936 (-0.949934) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005129 / 0.011353 (-0.006224) | 0.003490 / 0.011008 (-0.007518) | 0.049005 / 0.038508 (0.010497) | 0.054077 / 0.023109 (0.030968) | 0.276653 / 0.275898 (0.000755) | 0.298752 / 0.323480 (-0.024728) | 0.003979 / 0.007986 (-0.004007) | 0.002625 / 0.004328 (-0.001703) | 0.047951 / 0.004250 (0.043701) | 0.040969 / 0.037052 (0.003916) | 0.279879 / 0.258489 (0.021390) | 0.306244 / 0.293841 (0.012403) | 0.029025 / 0.128546 (-0.099522) | 0.010450 / 0.075646 (-0.065197) | 0.056846 / 0.419271 (-0.362426) | 0.033476 / 0.043533 (-0.010057) | 0.273340 / 0.255139 (0.018201) | 0.294783 / 0.283200 (0.011584) | 0.019105 / 0.141683 (-0.122578) | 1.126389 / 1.452155 (-0.325766) | 1.183369 / 1.492716 (-0.309348) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094995 / 0.018006 (0.076989) | 0.306984 / 0.000490 (0.306495) | 0.000224 / 0.000200 (0.000024) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021880 / 0.037411 (-0.015532) | 0.069674 / 0.014526 (0.055148) | 0.082191 / 0.176557 (-0.094366) | 0.120956 / 0.737135 (-0.616179) | 0.083843 / 0.296338 (-0.212495) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295139 / 0.215209 (0.079929) | 2.860520 / 2.077655 (0.782865) | 1.578892 / 1.504120 (0.074772) | 1.451003 / 1.541195 (-0.090192) | 1.483099 / 1.468490 (0.014609) | 0.550491 / 4.584777 (-4.034286) | 2.430352 / 3.745712 (-1.315360) | 2.874468 / 5.269862 (-2.395393) | 1.741474 / 4.565676 (-2.824202) | 0.062563 / 0.424275 (-0.361712) | 0.004962 / 0.007607 (-0.002645) | 0.343747 / 0.226044 (0.117703) | 3.419046 / 2.268929 (1.150118) | 1.943774 / 55.444624 (-53.500851) | 1.650989 / 6.876477 (-5.225488) | 1.704083 / 2.142072 (-0.437990) | 0.645447 / 4.805227 (-4.159780) | 0.125105 / 6.500664 (-6.375559) | 0.041319 / 0.075469 (-0.034150) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.959708 / 1.841788 (-0.882079) | 12.235906 / 8.074308 (4.161598) | 10.575402 / 10.191392 (0.384010) | 0.143619 / 0.680424 (-0.536805) | 0.015517 / 0.534201 (-0.518684) | 0.285231 / 0.579283 (-0.294052) | 0.281549 / 0.434364 (-0.152815) | 0.326649 / 0.540337 (-0.213689) | 0.565706 / 1.386936 (-0.821230) |

</details>
</details>

![](https://cml.dev/watermark.png#fb6985bc33277a3ece7f28c74ca742ba84655b0c "CML watermark")