mistralai

/

Mixtral-8x7B-Instruct-v0.1

like 3.89k

model = AutoModelForCausalLM.from_pretrained(DEFAULT_CKPT_PATH, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping)

when the prompt is only test, it will cause the errror:

File "/home/devbrain/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 2897, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: probability tensor contains either inf, nan or element < 0