macadeliccc
/

laser-dolphin-mixtral-4x7b-dpo-AWQ

Text Generation Transformers Safetensors mixtral Inference Endpoints text-generation-inference 4-bit precision

Model card Files Files and versions Community

Edit model card

OpenAI compatible endpoint using VLLM

Runs well on 4090

python -m vllm.entrypoints.openai.api_server --model macadeliccc/laser-dolphin-mixtral-4x7b-dpo-AWQ  --max-model-len 25000

Downloads last month: 8

Safetensors

Model size

3.46B params

Tensor type

I32

·

FP16

·

Quantized from