teknium
/

OpenHermes-2.5-Mistral-7B

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

teknium commited on Nov 3, 2023

Commit

9ae7036

•

1 Parent(s): 93f2ad1

Update transformers_inference.py

Files changed (1) hide show

transformers_inference.py +1 -1

transformers_inference.py CHANGED Viewed

@@ -10,7 +10,7 @@ tokenizer = LlamaTokenizer.from_pretrained('teknium/OpenHermes-2.5-Mistral-7B',
 model = MistralForCausalLM.from_pretrained(
     "teknium/OpenHermes-2.5-Mistral-7B",
     torch_dtype=torch.float16,
-    device_map=device_map="auto",#{'': 'cuda:0'},
     load_in_8bit=False,
     load_in_4bit=True,
     use_flash_attention_2=True

 model = MistralForCausalLM.from_pretrained(
     "teknium/OpenHermes-2.5-Mistral-7B",
     torch_dtype=torch.float16,
+    device_map="auto",#{'': 'cuda:0'},
     load_in_8bit=False,
     load_in_4bit=True,
     use_flash_attention_2=True