Recursive loop when implement model
Currently, we tried to implement the model with OpenAI API standard. The setup is all good but the model's output having recursive loop issue. This happen on both Completion API and Chat Completion API, we don't know what causing this because other models we implement before didn't met this issue. Is there any solution for this?
-----Issue Example-----
Chat Completion Input
"{messages": [{"role": "user", "content": "Hi"}]}
Chat Completion Output
{ "message": { "role": "assistant","content": "浈鳙ichern����������"}
Completion Input
"{"prompt": "San Francisco is a"}
Completion Output
{"text": " Growinggo\n[ xu/CompleteiabilityIONfidfCoderocomnDidLoadlify iable hf iphabit hoypoendiff"}
Hi @ann-lab52 , i believe the tokenizer is not updated to use with Chat Completion.
@ann-lab52 updated, now you should be able to use it with Completion API . however, i think this model is having some issue doing inference on VLLM + TGI at the same time, not sure why. for now please use vanilla transformers if you run into any problem
Hi @qnguyen3 , thanks for the fast reply. Unfortunately as you expected, the tokenizer is not the only cause and this issue persisted even though we used the latest model update. Hopefully this will be fixed in your future works. Also, if possible, we would be happy to have your permission to quantize your models and make them available to the community follows your model's license. Please let us know if you have any other requirements, we will be happy to follow your instructions.
HIi @ann-lab52 . That is absolutely fine! No worries about that
btw, if you are still looking for a working alternative, here is one: vilm/Quyen-Mini-v0.1
This new model is actually an update of this current model. It uses Qwen 1.5 (1.8B) as the backbone and finetune on the same dataset + DPO. The score is also better than this model.
Thanks @qnguyen3 , model vilm/Quyen-Mini-v0.1 works excellent. Also, quan-1.8B-chat-AWQ was uploaded and other AWQ version of your models will be release soon. Thanks again for your permission.
Thank you you much @ann-lab52 , i will add AWQ!