Repository: stanford-crfm/helm
Issue: add_generation_prompt default breaks chat generation
Root cause: Configuration Error

Hi team,

As I see that in the huggingface client the default value of **`add_generation_prompt` in `apply_chat_template` is `False`**:

```python
# src/helm/clients/huggingface_client.py  (L≈296-300)
prompt = tokenizer.apply_chat_template(request.messages, tokenize=False)
```
[Reference file](https://raw.githubusercontent.com/stanford-crfm/helm/refs/heads/main/src/helm/clients/huggingface_client.py)

If a caller forgets to override this, `model.generate()` keeps writing as the **user** instead of beginning an **assistant** turn, so the output is scrambled.

Could we please:

1. Set the default to `True`, 

```python
# src/helm/clients/huggingface_client.py  (L≈296-300)
prompt = tokenizer.apply_chat_template(request.messages, tokenize=False, add_generation_prompt=True)
```
**or**
2. Emit a clear warning/error when `tokenize=True` and `add_generation_prompt` is left `False`.
Relevant docs:
[https://huggingface.co/docs/transformers/en/chat\_templating#applychattemplate](https://huggingface.co/docs/transformers/en/chat_templating#applychattemplate)

Thanks!


Thanks. I think it makes sense to set `add_generation_prompt` to `True` by default. Could you open a pull request for this?

Could you also provide an example of a model for which the output is scrambled when `add_generation_prompt` is `False`? So far, I haven't run into any issues with chat models that I've tried, because the end of message token is already applied by default. 

Hi, thanks for your response.

Let me illustrate this using the Qwen model as an example. Suppose we start with the following user message:

```python
prompt = 'Hello, who are you?'
messages = [
    {"role": "user", "content": prompt}
]
```

### 1. Without Generation Prompt (`add_generation_prompt=False`)

When we call:

```python
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
```

The resulting formatted text is:

```

system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.


user
Hello, who are you?

```

This format treats the input more like a language modeling task rather than a structured chat. As a result, generation may degrade in quality and include redundant or misplaced special tokens, such as:

```

user
I'm curious about the history of your creation and how it came to be. Can
```

### 2. With Generation Prompt (`add_generation_prompt=True`)

Now, if we instead call:

```python
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
```

We get:

```

system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.


user
Hello, who are you?


assistant
```

This format preserves the conversational structure and guides the model to generate an appropriate assistant response. The output is cleaner and aligned with expected assistant behavior:

```
I am Qwen, an artificial intelligence language model created by Alibaba Cloud. I'm here to assist
```

### Example Code

Here's a toy example you can try out:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-0.5B-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = 'Hello, who are you?'
messages = [
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True  # Toggle this to see the effect
)

print(text)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=20,
    temperature=1e-7
)

# Strip input tokens to get only the generated part
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids)[0]
print(response)
```

Thanks!

Hi, I have created PR #3703