mlabonne

/

Marcoro14-7B-slerp

like 29

Hi Eric, like NeuralMarcoro14 (https://huggingface.co/spaces/mlabonne/NeuralMarcoro14-7B-GGUF-Chat/blob/main/Dockerfile#L7), it should use the chatml template.

However, the ChatML tokens are not added (ie </s> is the EOS token)

https://huggingface.co/mlabonne/Marcoro14-7B-slerp/blob/main/tokenizer_config.json

So, you trained it to expect / generate <|im_start|> and <|im_end|> but not as tokens?

Yes, I had a discussion on this topic about another model. For NeuralMarcoro14, I added the chatml template as "chat_template" (https://huggingface.co/mlabonne/NeuralMarcoro14-7B/blob/main/tokenizer_config.json#L44). This quick fix worked so I didn't try to change the tokens.

Do you think it's better to modify the tokenizer config to have a proper chatml template or just stick with Llama's chat template?

So, you trained it to expect / generate <|im_start|> and <|im_end|> but not as tokens?

This one is untrained, but this is exactly what happened with the neural version during DPO (this is mildly infuriating).