Repository: UKGovernmentBEIS/inspect_ai
Issue: [WARNING] [3.93 onward] Abrupt change in behavior for vllm backend
Root cause: Configuration Error

Hi there,

I observe with the release 0.3.93 / 0.3.94 a strong variation in both the results and the timings on the execution of the same benchmark.
The benchmark runs with vllm, for which the usage has been strongly revised in the aforementioned release.

To reproduce, ideally on 1 node with 4 x A100 GPUs:

```
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals --force
```

 _pip show inspect_evals
Name: inspect_evals
Version: 0.3.dev1254+gdac86bc
Summary: Collection of large language model evaluations
Home-page:
Author: UK AI Security Institute
Author-email:
License: MIT License
Location: conda/envs/inspect/lib/python3.12/site-packages
Requires: datasets, pillow, pydantic, requests, tiktoken
Required-by:_

_pip show vllm
Name: vllm
Version: 0.8.5.post1
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email:
License-Expression: Apache-2.0
Location: conda/envs/inspect/lib/python3.12/site-packages
Requires: aiohttp, blake3, cachetools, cloudpickle, compressed-tensors, depyf, einops, fastapi, filelock, gguf, huggingface-hub, importlib_metadata, lark, llguidance, lm-format-enforcer, mistral_common, msgspec, ninja, numba, numpy, openai, opencv-python-headless, opentelemetry-api, opentelemetry-exporter-otlp, opentelemetry-sdk, opentelemetry-semantic-conventions-ai, outlines, partial-json-parser, pillow, prometheus-fastapi-instrumentator, prometheus_client, protobuf, psutil, py-cpuinfo, pydantic, python-json-logger, pyyaml, pyzmq, ray, requests, scipy, sentencepiece, setuptools, six, tiktoken, tokenizers, torch, torchaudio, torchvision, tqdm, transformers, typing_extensions, watchfiles, xformers, xgrammar
Required-by:_

(inspect_evals and vllm version remain the same through all experiments)

### 0.3.92
```
pip install inspect_ai==0.3.92 --force
inspect eval inspect_evals/ifeval --model vllm/meta-llama/Llama-3.2-1B-Instruct
```

```
╭─ ifeval (541 samples): vllm/meta-llama/Llama-3.2-1B-Instruct ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                   dataset: google/IFEval │
│                                                                                                                                                                                                                          │
│ total time:                                                                                               0:05:01                                                                                                        │
│ vllm/meta-llama/Llama-3.2-1B-Instruct                                                                     423,034 tokens [I: 44,321, O: 378,713]                                                                         │
│                                                                                                                                                                                                                          │
│ prompt_strict_acc: 0.327  prompt_strict_stderr: 0.0202  prompt_loose_acc: 0.364  prompt_loose_stderr: 0.0207  inst_strict_acc: 0.471  inst_strict_stderr: 0.0167  inst_loose_acc: 0.507  inst_loose_stderr: 0.0166       │
│ final_acc: 0.417  final_stderr: 0.0179                                                                                                                                                                                   │
│                                                                                                                                                                                                                          │
│ Log: logs/2025-05-13T15-56-28+02-00_ifeval_hWoZ5zw74RToKRj6Wya9eL.eval                                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
### 0.3.93
```
pip install inspect_ai==0.3.93 --force
inspect eval inspect_evals/ifeval --model vllm/meta-llama/Llama-3.2-1B-Instruct
```
vllm server does not start:

```
inspect_ai._util.error.PrerequisiteError: ERROR: Unable to initialise vLLM client

No [bold][blue]VLLM_API_KEY[/blue][/bold] defined in the environment.
```
### 0.3.94
```
pip install inspect_ai==0.3.94 --force
inspect eval inspect_evals/ifeval --model vllm/meta-llama/Llama-3.2-1B-Instruct
```
```
╭─ ifeval (541 samples): vllm/meta-llama/Llama-3.2-1B-Instruct ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                   dataset: google/IFEval │
│                                                                                                                                                                                                                          │
│ total time:                                                                                               1:02:34                                                                                                        │
│ vllm/meta-llama/Llama-3.2-1B-Instruct                                                                     299,548 tokens [I: 43,780, O: 255,768]                                                                         │
│                                                                                                                                                                                                                          │
│ prompt_strict_acc: 0.512  prompt_strict_stderr: 0.0215  prompt_loose_acc: 0.562  prompt_loose_stderr: 0.0214  inst_strict_acc: 0.632  inst_strict_stderr: 0.017  inst_loose_acc: 0.673  inst_loose_stderr: 0.0165        │
│ final_acc: 0.595  final_stderr: 0.0194                                                                                                                                                                                   │
│                                                                                                                                                                                                                          │
│ Log: logs/2025-05-14T08-56-32+02-00_ifeval_BndD4neNfoGkcJ8BiZDtnc.eval                                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

The following run (that appears to solve the timing issues) was obtained after going through the messages below, and modifying the cmd launch of the vllm server to:

```
        cmd = ["vllm", "serve", model_path, "--host", host, "--api-key", api_key, "--generation-config", "vllm"]
```

```
─ ifeval (541 samples): vllm/meta-llama/Llama-3.2-1B-Instruct ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                    dataset: google/IFEval │
│                                                                                                                                                                                                                           │
│ total time:                                                                                                0:05:36                                                                                                        │
│ vllm/meta-llama/Llama-3.2-1B-Instruct                                                                      691,401 tokens [I: 43,780, O: 647,621]                                                                         │
│                                                                                                                                                                                                                           │
│ prompt_strict_acc: 0.436  prompt_strict_stderr: 0.0213  prompt_loose_acc: 0.46  prompt_loose_stderr: 0.0214  inst_strict_acc: 0.56  inst_strict_stderr: 0.0174  inst_loose_acc: 0.584  inst_loose_stderr: 0.0172          │
│ final_acc: 0.51  final_stderr: 0.0194                                                                                                                                                                                     │
│                                                                                                                                                                                                                           │
│ Log: logs/2025-05-15T15-58-54+02-00_ifeval_72yiwc6Qe8riKdBVgSMraG.eval                                                                                                                                                    │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

To obtain this last run, I hacked the vllm prompt_token_ids to duplicate the first token:

```
self.prompt_token_ids = [prompt_token_ids[0]] + prompt_token_ids
```

```
╭─ ifeval (541 samples): vllm/meta-llama/Llama-3.2-1B-Instruct ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                    dataset: google/IFEval │
│                                                                                                                                                                                                                           │
│ total time:                                                                                                0:05:58                                                                                                        │
│ vllm/meta-llama/Llama-3.2-1B-Instruct                                                                      711,048 tokens [I: 43,780, O: 667,268]                                                                         │
│                                                                                                                                                                                                                           │
│ prompt_strict_acc: 0.349  prompt_strict_stderr: 0.0205  prompt_loose_acc: 0.386  prompt_loose_stderr: 0.021  inst_strict_acc: 0.488  inst_strict_stderr: 0.0167  inst_loose_acc: 0.529  inst_loose_stderr: 0.0166         │
│ final_acc: 0.438  final_stderr: 0.0183                                                                                                                                                                                    │
│                                                                                                                                                                                                                           │
│ Log: logs/2025-05-15T16-16-54+02-00_ifeval_E8FAgeqCak9hBENkDZQ3Fr.eval                                                                                                                                                    │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

While these two changes explain well the timings, and the scores, there is still a mismatch at the number of generated tokens...

Addressing the "chunked prefill" difference was tricky, but is explained by vllm's code :

```
        # NOTE(Kuntai): Setting large `max_num_batched_tokens` for A100 reduces
        # throughput, see PR #17885 for more details.
        # So here we do an extra device name check to prevent such regression.
        if device_memory >= 70 * GiB_bytes and "a100" not in device_name:
            # For GPUs like H100 and MI300x, use larger default values.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 8192,
            }
            default_max_num_seqs = 1024
        else:
            # TODO(woosuk): Tune the default values for other hardware.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 8192,
                UsageContext.OPENAI_API_SERVER: 2048,
            }
            default_max_num_seqs = 256
```

since from 0.3.92 to 0.3.94 LLM_CLASS usage was replaced by OPENAI_API_SERVER.

Unfortunately, forcing 8192 does not lead to the 0.3.92 number of output tokens:

```
╭─ ifeval (541 samples): vllm/meta-llama/Llama-3.2-1B-Instruct ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                 dataset: google/IFEval │
│                                                                                                                                                                                                                        │
│ total time:                                                                                              0:05:43                                                                                                       │
│ vllm/meta-llama/Llama-3.2-1B-Instruct                                                                    694,815 tokens [I: 43,780, O: 651,035]                                                                        │
│                                                                                                                                                                                                                        │
│ prompt_strict_acc: 0.372  prompt_strict_stderr: 0.0208  prompt_loose_acc: 0.403  prompt_loose_stderr: 0.0211  inst_strict_acc: 0.49  inst_strict_stderr: 0.0175  inst_loose_acc: 0.526  inst_loose_stderr: 0.0172      │
│ final_acc: 0.448  final_stderr: 0.0189                                                                                                                                                                                 │
│                                                                                                                                                                                                                        │
│ Log: logs/2025-05-16T10-32-21+02-00_ifeval_A9bYgAoP3cdPQ3Hzhf6du2.eval                                                                                                                                                 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

I notice a change in the number of input tokens (+1) for all input prompts.

It could mean that there is a change in the tokenizer used, or that this tokenizer has been modified, or its usage updated.

Here a comparison between vllm configurations retrieved from the logs:

# vllm configuration with inspect_ai 0.3.92:
```
[core.py:58] Initializing a V1 LLM engine (v0.8.5.post1) with config: model='meta-llama/Llama-3.2-1B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.2-1B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=meta-llama/Llama-3.2-1B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
```
# vllm configuration with inspect_ai 0.3.94:

```
INFO core.py:58: Initializing a V1 LLM engine (v0.8.5.post1) with config: model='meta-llama/Llama-3.2-1B-Instruct', speculative_config=None,           local_server.py:108
                             tokenizer='meta-llama/Llama-3.2-1B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None,
                             trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1,
                             disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda,
                             decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False,
                             otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=meta-llama/Llama-3.2-1B-Instruct,
                             num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True,
                             disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None,
                             compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"
                             use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,
                             320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size"
                             :512}
```

The configurations seem identical...

Nevertheless, there is an additional warning message that is triggerd by vllm using inspect_ai 0.3.94:
```
WARNING:vllm.config:Default sampling parameters have been overridden by the model's Hugging Face generation config recommended   local_server.py:108
                             from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
```

This is generated in the vllm_config.py file, function get_diff_sampling_params of vllm, with docstring:

```
This method returns a dictionary containing the parameters
        that differ from the default sampling parameters. If
        `generation_config` is `"vllm"`, an empty dictionary is returned.

        Returns:
            dict[str, Any]: A dictionary with the differing sampling
            parameters, if `generation_config` is `"vllm"` an empty dictionary.
```

The variables differing are temperature and top_p...

Also, in 0.3.92 this message appears:
```
[config.py:2006] Chunked prefill is enabled with max_num_batched_tokens=8192.
```
while not in 0.3.94

It would seem that apply_hf_chat_template is only called on vllm's side in the version 0.3.94 (which is expected given the changes brought by 0.3.93).

My best guess at the current state of investigation was that the tokenizer retrieved by vllm differs from the one retrieved by inspect... but they are identical (see comment below)

It would be nice to have a sample filter on timings in the evals view (some few prompts took about 45 minutes to generate, and it is not straightforward to identity them in the inspect view).

It would be nice to have the information about which tokenizer was applied within the inspect view.

It would be nice to have which version of vllm was used to execute the benchmark (within the Task section ?).

It could be interesting (but might clutter the logs) to have the tokenized version of the prompt.

# 0.3.94 (vllm's side)
```
vllm.entrypoints.chat_utils:CachedPreTrainedTokenizerFast(name_or_path='meta-llama/Llama-3.2-1B-Instruct', vocab_size=128000, model_max_length=131072, is_fast=True, 
                             padding_side='right', truncation_side='left', special_tokens={'bos_token': '
', 'eos_token': '
'}, clean_up_tokenization_spaces=True,
                             added_tokens_decoder={
128000: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128001: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128002: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128003: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128004: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128005: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128006: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128007: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128008: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128009: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128010: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128011: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128012: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128013: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128014: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128015: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128016: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128017: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128018: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128019: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128020: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128021: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128022: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128023: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128024: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128025: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128026: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128027: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128028: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128029: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128030: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128031: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128032: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128033: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128034: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128035: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128036: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128037: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128038: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128039: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128040: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128041: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128042: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128043: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128044: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128045: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128046: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128047: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128048: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128049: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128050: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128051: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128052: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128053: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128054: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128055: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128056: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128057: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128058: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128059: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128060: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128061: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128062: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128063: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128064: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128065: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128066: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128067: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128068: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128069: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128070: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128071: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128072: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128073: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128074: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128075: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128076: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128077: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128078: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128079: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128080: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128081: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128082: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128083: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128084: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128085: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128086: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128087: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128088: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128089: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128090: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128091: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128092: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128093: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128094: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128095: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128096: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128097: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128098: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128099: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128100: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128101: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128102: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128103: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128104: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128105: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128106: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128107: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128108: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128109: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128110: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128111: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128112: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128113: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128114: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128115: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128116: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128117: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128118: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128119: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128120: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128121: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128122: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128123: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128124: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128125: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128126: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128127: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128128: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128129: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128130: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128131: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128132: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128133: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128134: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128135: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128136: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128137: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128138: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128139: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128140: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128141: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128142: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128143: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128144: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128145: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128146: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128147: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128148: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128149: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128150: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128151: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128152: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128153: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128154: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128155: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128156: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128157: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128158: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128159: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128160: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128161: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128162: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128163: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128164: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128165: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128166: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128167: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128168: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128169: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128170: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128171: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128172: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128173: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128174: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128175: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128176: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128177: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128178: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128179: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128180: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128181: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128182: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128183: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128184: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128185: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128186: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128187: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128188: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128189: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128190: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128191: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128192: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128193: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128194: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128195: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128196: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128197: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128198: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128199: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128200: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128201: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128202: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128203: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128204: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128205: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128206: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128207: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128208: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128209: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128210: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128211: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128212: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128213: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128214: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128215: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128216: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128217: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128218: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128219: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128220: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128221: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128222: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128223: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128224: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128225: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128226: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128227: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128228: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128229: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128230: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128231: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128232: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128233: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128234: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128235: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128236: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128237: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128238: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128239: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128240: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128241: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128242: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128243: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128244: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128245: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128246: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128247: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128248: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128249: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128250: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128251: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128252: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128253: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128254: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128255: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
```

# 0.3.92 (inspect_ai's side)
```
    CachedPreTrainedTokenizerFast(name_or_path='meta-llama/Llama-3.2-1B-Instruct', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right',  
                             truncation_side='left', special_tokens={'bos_token': '
', 'eos_token': '
'}, clean_up_tokenization_spaces=True, added_tokens_decoder={
                                     128000: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128001: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128002: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128003: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128004: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128005: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128006: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128007: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128008: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128009: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128010: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128011: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128012: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128013: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128014: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128015: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128016: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128017: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128018: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128019: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128020: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128021: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128022: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128023: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128024: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128025: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128026: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128027: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128028: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128029: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128030: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128031: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128032: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128033: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128034: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128035: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128036: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128037: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128038: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128039: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128040: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128041: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128042: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128043: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128044: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128045: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128046: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128047: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128048: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128049: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128050: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128051: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128052: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128053: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128054: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128055: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128056: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128057: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128058: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128059: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128060: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128061: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128062: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128063: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128064: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128065: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128066: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128067: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128068: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128069: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128070: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128071: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128072: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128073: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128074: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128075: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128076: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128077: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128078: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128079: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128080: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128081: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128082: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128083: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128084: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128085: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128086: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128087: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128088: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128089: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128090: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128091: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128092: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128093: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128094: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128095: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128096: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128097: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128098: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128099: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128100: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128101: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128102: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128103: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128104: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128105: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128106: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128107: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128108: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128109: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128110: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128111: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128112: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128113: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128114: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128115: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128116: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128117: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128118: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128119: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128120: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128121: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128122: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128123: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128124: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128125: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128126: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128127: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128128: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128129: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128130: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128131: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128132: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128133: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128134: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128135: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128136: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128137: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128138: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128139: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128140: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128141: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128142: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128143: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128144: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128145: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128146: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128147: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128148: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128149: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128150: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128151: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128152: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128153: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128154: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128155: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128156: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128157: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128158: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128159: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128160: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128161: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128162: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128163: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128164: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128165: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128166: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128167: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128168: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128169: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128170: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128171: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128172: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128173: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128174: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128175: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128176: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128177: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128178: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128179: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128180: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128181: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128182: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128183: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128184: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128185: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128186: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128187: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128188: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128189: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128190: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128191: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128192: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128193: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128194: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128195: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128196: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128197: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128198: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128199: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128200: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128201: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128202: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128203: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128204: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128205: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128206: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128207: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128208: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128209: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128210: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128211: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128212: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128213: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128214: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128215: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128216: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128217: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128218: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128219: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128220: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128221: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128222: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128223: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128224: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128225: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128226: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128227: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128228: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128229: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128230: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128231: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128232: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128233: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128234: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128235: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128236: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128237: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128238: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128239: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128240: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128241: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128242: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128243: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128244: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128245: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128246: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128247: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128248: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128249: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128250: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128251: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128252: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128253: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128254: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                                     128255: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
                             }
                             )
```

google/IFEval first prompt
# 0.3.94 
```
[128000, 128006, 9125, 128007, 271, 38766, 1303, 33025, 2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696, 25, 220, 868, 3297,  220, 2366, 20, 271, 128009, 128006, 882, 128007, 271, 8144, 264, 220, 3101, 10, 3492, 12399, 315, 279, 59318, 2199, 330, 2485, 1129, 268, 34466, 2726, 26583, 19945, 352, 12669, 62, 23440, 21025, 2568, 3659, 1159, 4664, 14559, 3343, 3234, 539, 1005, 904, 77702, 323, 11415, 520, 3325, 220, 18, 14491, 430, 706, 15671, 304, 51594, 3645, 11, 369, 3187, 353, 36298, 291, 3857, 961, 220, 16, 12594, 353, 36298, 291, 3857, 961, 220, 17, 12594, 353, 36298, 291, 3857, 961, 220, 18, 20517, 128009, 128006, 78191, 128007, 271]
```
in Request::__init__

# 0.3.92
```
[128000, 128000, 128006, 9125, 128007, 271, 38766, 1303, 33025, 2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696, 25, 220, 868, 3297, 220, 2366, 20, 271, 128009, 128006, 882, 128007, 271, 8144, 264, 220, 3101, 10, 3492, 12399, 315, 279, 59318, 2199, 330, 2485, 1129, 268, 34466, 2726, 26583, 19945, 352, 12669, 62, 23440, 21025, 2568, 3659, 1159, 4664, 14559, 3343, 3234, 539, 1005, 904, 77702, 323, 11415, 520, 3325, 220, 18, 14491, 430, 706, 15671, 304, 51594, 3645, 11, 369, 3187, 353, 36298, 291, 3857, 961, 220, 16, 12594, 353, 36298, 291, 3857, 961, 220, 17, 12594, 353, 36298, 291, 3857, 961, 220, 18, 20517, 128009, 128006, 78191, 128007, 271]
```

The first token seem to have been duplicated in the old version

It would be nice to have the sampling parameters used for inference in inspect view.

It would be great if all logs from vllm could pass through inspect_ai in its latest implementation.

Currently, by default, when using `--log-level debug` only vllm warnings are displayed in the console (no debug, nor info).

Moreover, the "max_tokens" (such as used in the multichoice solver) argument might be ignored  by vllm while parsing OpenAIAPI requests (and is actually not taken into account since moving from inspect_ai 0.3.92 to 0.3.93/0.3.94):

```
    # TODO(#9845): remove max_tokens when field is removed from OpenAI API
    max_tokens: Optional[int] = Field(
        default=None,
        deprecated=
        'max_tokens is deprecated in favor of the max_completion_tokens field')
    max_completion_tokens: Optional[int] = None
```

dirty patching the model/_openai.py:

```
-        params["max_tokens"] = config.max_tokens
+        params["max_completion_tokens"] = config.max_tokens
```

bypasses the issue.

Hi @jjallaire , would you have any input regarding this issue ?

@kaifronsdal If you have any time to weigh in on these differences, I think you're the most knowledgeable about the vLLM implementation...

I think the main differences come down to differences in the sampling params used between versions (I could be wrong about this though).

Here are the differences in tokenization and sampling params when running:
```
inspect eval examples/hello_world.py --model vllm/meta-llama/Llama-3.2-1B-Instruct
```

### Version 0.3.92
'

system
\n\nCutting Knowledge Date: December 2023\nToday Date: 13 Jun 2025\n\n

user
\n\nJust reply with Hello World

assistant
\n\n'

SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[128000, 128009], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=2048, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None)


### Version 0.3.94

'

system
\n\nCutting Knowledge Date: December 2023\nToday Date: 13 Jun 2025\n\n

user
\n\nJust reply with Hello World

assistant
\n\n'

SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.9, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131032, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None)


The tokenization looks the same between them.
The main differences between the two are that in 0.3.92 we use a default `DEFAULT_MAX_TOKENS = 2048`. This was set somewhat arbitrarily by me when I first made the old implementation and so I decided to remove it in the new version. There are also different `temperature=1.0` and `top_p=1.0.` This is because before when we were using the offline inference api, we had to create a `SamplingParams` object to pass to vllm which has some global defaults. When using the online server api, vllm uses default params on an individual model level. The `stop_token_ids` are also set in 0.3.92 to be end of sequence tokens by inspect since there were some issues with vllm correctly ending generation when using the offline api.

I think the new behavior of using the model's default params is much better than the old behavior of global defaults. If you want the old behavior back, all you need to do is set `max_tokens`, `temperature`, and `top_p` yourself (and any other params your models might set different defaults for).

As for the +1 extra input token, it seems that either there was a subtle bug in inspect the implementation always (i.e. vllm always added the bos token and we added an extra one when we applied the chat template), or perhaps there was a change in vllm when they recently moved to the `llm.chat` interface for offline chat generation rather than having to apply the chat template yourself.

@kaifronsdal Thanks for the input. 

The changelog might benefit from that.