enghwa/lab

Model Card for Mistral-7B-Instruct-v0.3 for inf2.xlarge

The Mistral-7B-Instruct-v0.3 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.3. neuron compiled these are the shapes currently cached. don't ask me why 8196.. it was a typo.

python -m vllm.entrypoints.openai.api_server --model ./  --max-model-len 8196 --device neuron --tensor-parallel-size 2 --max-num-seqs 2
python -m vllm.entrypoints.openai.api_server --model ./  --max-model-len 8196 --device neuron --tensor-parallel-size 2 --max-num-seqs 4
python -m vllm.entrypoints.openai.api_server --model ./  --max-model-len 10240 --device neuron --tensor-parallel-size 2 --max-num-seqs 4