oleksandrfluxon
/

mpt-7b-chat-4bit

Text Generation

text-generation-inference

Model card Files Files and versions Community

oleksandrfluxon commited on Jul 20, 2023

Commit

64acb26

•

1 Parent(s): 6ee6fc5

Update pipeline.py

Files changed (1) hide show

pipeline.py +1 -2

pipeline.py CHANGED Viewed

@@ -25,8 +25,7 @@ class PreTrainedPipeline():
               # torch_dtype=torch.bfloat16, # Load model weights in bfloat16
               torch_dtype=torch.float16,
               trust_remote_code=True,
-              device_map="auto",
-              revision="pr/47",
               load_in_8bit=True # Load model in the lowest 4-bit precision quantization
             )
             # model.to('cuda')

               # torch_dtype=torch.bfloat16, # Load model weights in bfloat16
               torch_dtype=torch.float16,
               trust_remote_code=True,
+              device_map="auto",
               load_in_8bit=True # Load model in the lowest 4-bit precision quantization
             )
             # model.to('cuda')