How to drop the stop token from the response?
#31
by
mattma1970
- opened
I'm using TGI to get responses from a chat using OpenOrca. The model response always contains the stop token <|im_end|> in the decoded text. How do I stop this string representation of the stop token from showing up in the decoded text?
(I'm using an agent framework that takes the model response and adds it directly to the conversation history so I'd have to hack the framework to pre-process it before saving to memory. Doable but not ideal and it seems like there should be an option to suppress it).
Thanks
Matt