Spaces:

CoderCowMoo
/

Llama-3-70b-Instruct-Orthoganalized

Runtime error

CoderCowMoo commited on May 5

Commit

5be4bff

•

1 Parent(s): d23fc4b

problem with marlin_kernel step

Files changed (1) hide show

app.py CHANGED Viewed

@@ -23,7 +23,6 @@ model = AutoGPTQForCausalLM.from_quantized(
         device="cuda:0",
         use_safetensors=True,
         disable_exllamav2=True,
-        use_marlin=True,
         quantize_config=quantize_config).eval()

         device="cuda:0",
         use_safetensors=True,
         disable_exllamav2=True,
         quantize_config=quantize_config).eval()