Error running with TGI using docker image
I'm trying to run this model using the HF TGI docker image: ghcr.io/huggingface/text-generation-inference
I'm running it with these options: --model-id mattshumer/mistral-8x7b-chat --max-input-length 3072 --max-total-tokens 4096
However TGI fails to start with these errors:
ERROR text_generation_launcher: Error when initializing model
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/weights.py", line 63, in get_filename
raise RuntimeError(f"weight {tensor_name} does not exist")
RuntimeError: weight model.layers.0.mlp.gate_proj.weight does not exist
Has this model been tested with TGI? I see it has a text-generation-interface label. If so, any advice on what I'm doing wrong?
FWIW, with those same settings I'm able to start the mistralai/Mixtral-8x7B-Instruct-v0.1 model with no issues.