mgoin

/

Meta-Llama-3-70B-Instruct-Marlin

like 5

Hey @Samvanity Marlin is a 4bit weight quantized format for performant inference. You can read the latest summary in this blog here https://neuralmagic.com/blog/pushing-the-boundaries-of-mixed-precision-llm-inference-with-marlin/

thank you for make this great model as marlin