Repository: EleutherAI/lm-evaluation-harness
Issue: HuggingFace model prompt formatting
Root cause: Unimplemented Feature Gap

Is there an existing feature request for tokenizer prompt formatting for HuggingFace models? For example, the new mixtral instruct tokenizers come with a prompt formatting option to best capitalize on the sft. 

I'm experimenting with adding the option since I don't see it in the codebase at present. 

Hi! We don’t support this yet but would very much like to.

I planned to get to this after the holidays at a pretty high priority. If it’s helpful, I can sketch how I think the implementation would be best laid out for extensibility/compatibility?

Sounds great. Yes, please feel free to sketch it out for me! 

I was going to hardcode it into `lm_eval/models/huggingface/tok_encode` as a v0 for my experiment - but admittedly I wasn't thinking much about extensibility/compatibility yet.

Actually I think it can be as simple as—write a method in HFLM, wrap_chat_template(self, requests: List[Instance]) -> List[Instance] that takes in the requests and applies the chat template to replace each req.args with req.args (still strings) formatted within chat template.

This method should be called prior to anything else in the `loglikelihood()` and `generate_until()` methods but should not be applied for `_loglikelihood_tokens` or for `loglikelihood_rolling` (and should log a warning that perplexity /loglikelihood rolling tasks do not get chat templates applied.)

There should be flags `—model_args use_chat_template=True,system_prompt={string}` and one should not be able to pass a system prompt without turning on chat templating.


Hope this helps as a sketch!

Beyond the concerns in #1098 there are some other ones:
- we want this to be compatible with CachingLM and to log the chat template-wrapped text when calling with —log_samples so one can observe the inputs + outputs to the model in their final true form.
- Some things, like “Let’s think step by step”, should ideally be fed to the model *after the last “Assistant: “ turn, before having the model generate output. Still unsure how to deal with this.
- The string loglikelihood uses as target shouldn’t change, though we should still have the last “Assistant: “ turn prefixed that shouldn’t be part of the targets.

these concerns can be dealt with once there is something to work with and test out though.

I can take a stab at a v0 of support for chat templates as per your above note (on my fork). Will take into account concerns in https://github.com/EleutherAI/lm-evaluation-harness/issues/1098 and above. 

I agree some of these "extra" functionalities might be better suited to adding after a baseline is working and tested (for instance, Mistral/Mixtral chat templates don't support a system message and throw an error when given one).

@haileyschoelkopf is there a preferred way to update the req.args? 

Per your note "applies the chat template to replace each req.args with req.args (still strings) formatted within chat template"

* I was able to format each req.args easily enough and confirm that it's working with some print statements. 
* However, updating the formatted strings back into the requests list is throwing me an error along the lines of "requests doesn't support item assignment". 
    * Am I missing a helper to update the req.args perhaps? Or were you thinking construct a new requirements list instead of updating the existing one?

@daniel-furman sorry for not getting back to you on this until now!

updating request args is not something we've had need for in the library yet. it should be fine to add a method `Instance.update_args(new_args)` that overrides them! 

@haileyschoelkopf cycling back here, had some free time over this weekend...

Initial results are mixed. 

I will do some more testing before adding in the CLI controls (as you can find on my [fork](https://github.com/daniel-furman/lm-evaluation-harness)). 

lm_eval/api/instance.py:

```
@args.setter
def args(self, new_arguments: tuple) -> None:
    """
    Update the arguments of this instance with a new one
    """
    if isinstance(new_arguments, tuple):
        self.arguments = new_arguments
    else:
        print("Please enter a valid arguments tuple")
```

lm_eval/models/huggingface.py:

**WIP**