Error using with Text-Generation-Inference RuntimeError: weight model.layers.0.self_attn.rotary_emb.inv_freq does not exist
Hi,
I'm getting the following error when trying to run this model with TGI:
RuntimeError: weight model.layers.0.self_attn.rotary_emb.inv_freq does not exist
I don't see weight model.layers.0.self_attn.rotary_emb.inv_freq in pytorch_model.bin.index.json even though it is present in the llama-2 index (and in the other l2 airoboros models). Not sure if it is an issue with the model or with something I'm doing, but if anyone has successfully used with TGI and could point me in the right direction I'd appreciate it!
Thanks for your awesome work on this model!!
same issue
Not sure why, but it seems the merge -> save_pretrained saved everything except the *rotary_emb.inv_freq data to the index. I uploaded a new version: https://huggingface.co/jondurbin/airoboros-l2-70b-gpt4-1.4.1/blob/main/pytorch_model.bin.index.json
Try downloading that again and let me know.
thank you. Let me try again !
Thanks for uploading the new file. I am still getting the same error, trying to figure out how TGI is loading the model and debug to see if it is on their end. Its not clear to me if the bin.index is causing the issue or the actual files.
Sorry I don't have an answer for you, but I don't use TGI. I usually do one-off tests with FastChat, "real" work directly with python/transformers library.
Just a thought: have you considered vllm? With the TGI license change, and better inference speed of vllm, it might be worth a look, although I haven't tested it with my model yet.
https://github.com/vllm-project/vllm
no it dint work RuntimeError: weight model.layers.40.self_attn.rotary_emb.inv_freq does not exist
This is happening because of this PR: https://github.com/huggingface/transformers/pull/24998.
Try to load with a newer version of transformers (install from source) or save from older version.
were you able to resolve this?