mistralai

/

Mixtral-8x22B-Instruct-v0.1

like 590

Textgen webui supports it, I haven't tried the original safetensors but it works with a exl2 and gguf quant I made myself. LLama.cpp works too

how big is your gguf? With it being so huge, i didnt even bother trying.

You can use https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator to check the different sizes for gguf quants

Textgen webui supports it, I haven't tried the original safetensors but it works with a exl2 and gguf quant I made myself. LLama.cpp works too

Textgen webui supports it, I haven't tried the original safetensors but it works with a exl2 and gguf quant I made myself. LLama.cpp works too

How does text generation web ui supports it? Which plugin uses mistral defined tool usage format?

I used 8x7 in oobas text gen 'as is' nothing special was needed.

I used 8x7 in oobas text gen 'as is' nothing special was needed.

How did it use tools?