michaelfeil
/

ct2fast-flan-alpaca-large

Inference Endpoints

Model card Files Files and versions Community

michaelfeil commited on May 3, 2023

Commit

7df365d

•

1 Parent(s): 1881f31

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -24,7 +24,7 @@ model = TranslatorCT2fromHfHub(
         # load in int8 on CUDA
         model_name_or_path=model_name,
         device="cuda",
-        compute_type="cpu"
 )
 outputs = model.generate(
     text=["How do you call a fast Flan-ingo?", "Translate to german: How are you doing?"],

         # load in int8 on CUDA
         model_name_or_path=model_name,
         device="cuda",
+        compute_type="int8_float16"
 )
 outputs = model.generate(
     text=["How do you call a fast Flan-ingo?", "Translate to german: How are you doing?"],