hugging-quants
/

Meta-Llama-3.1-405B-Instruct-AWQ-INT4

Text Generation

text-generation-inference

Inference Endpoints

4-bit precision

Model card Files Files and versions Community

End Token fix

#9

by Lasdem - opened Jul 25

Lasdem

Jul 25

•

I had the issue that the generation did not stop at the end token, because the end token is wrongly configured.
in special_tokens_map.json change line 10 to "content": "<|eot_id|>",
in tokenizer_config.json change line 2055 to "eos_token": "<|eot_id|>",

PS: I am using the vllm server to run the model: vllm/vllm-openai:latest

Hugging Quants org Jul 30

Already fixed within #11, thanks for opening the issue @Lasdem !

alvarobartt changed discussion status to closed Jul 30

jmesout

Jul 30

@alvarobartt still experiencing this issue with the latest model on vLLM.

Lasdem

Jul 31

I have not tested, but I can see that here in tokenizer_config.json line 2055, still contains the wrong token.

Sep 13

This still needs to be fixed :) Maybe just accept the open commit? https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4/discussions/18

In the meantime, is there a workaround to make this run locally properly?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment