oleksandrfluxon
/

mpt-7b-instruct-evaluate

Text Generation

text-generation-inference

Model card Files Files and versions Community

oleksandrfluxon commited on Jul 23, 2023

Commit

fb80f65

•

1 Parent(s): d1aa94f

Update pipeline.py

Files changed (1) hide show

pipeline.py +1 -1

pipeline.py CHANGED Viewed

@@ -26,7 +26,7 @@ class PreTrainedPipeline():
               torch_dtype=torch.float16,
               trust_remote_code=True,
               device_map="auto",
-              load_in_4bit=True # Load model in the lowest 4-bit precision quantization
             )
             model.to('cuda')
             print("===> model loaded")

               torch_dtype=torch.float16,
               trust_remote_code=True,
               device_map="auto",
+              load_in_8bit=True # Load model in the lowest 4-bit precision quantization
             )
             model.to('cuda')
             print("===> model loaded")