template problem,

#1
by 2dts - opened

Hi, I trying to implement chat "continue" using api with this model, so I found that sometime continue "not good" after short debug, i found that it is because api send </s> token after request, which breaks previous line, so may be I could ask you to update the template here, to allow enduser to provide additional payload with a message, to prevent closing token

like change this part with additional condition on if add_generation_prompt property as in some other templates

{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}

or may be you know where this template from, so we can ask to update him too :)

great model btw !

I'm encountering a common issue with the model ending with wrong variations of <|im_end|> since it's technically multiple tokens instead of one, so it has a higher chance of outputting wrong end of turn token(s).

<|im_end|>
Tokens: [523, 28766, 321, 28730, 416, 28766, 28767]

which is honestly terrible mistake since it sometimes outputs wrong tokens like: <|user|>, <||, <|\n<user>

Edit: I should have opened another issue instead.

@Bakanayatsu model has another themplate - <s>[INST]</s> look into tokenizer.config

@2dts isn't the template ChatML as per the model card? I know this model is initialized from Mistral-7B-Instruct-v0.2 so that template remained in tokenizer.config.

Ok, for ppl who meet same problem - i just placed "not loop.last" condition to eos token, as there is no reason set </s> when there was last reply from bot ("continue" is only case when it happens)

Sign up or log in to comment