ibm-fms
/

llama-13b-accelerator

Inference Endpoints

Model card Files Files and versions Community

JRosenkranz commited on Apr 19

Commit

8c8a362

•

1 Parent(s): a68ddb6

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -46,7 +46,7 @@ docker run -d --rm --gpus all \
     -p 8033:8033 \
     -v /path/to/all/models:/models \
     -e MODEL_NAME=/models/model_weights/llama/13B-F \
-    -e SPECULATOR_NAME=ibm-fms/llama-13b-accelerator \
     -e FLASH_ATTENTION=true \
     -e PAGED_ATTENTION=true \
     -e DTYPE_STR=float16 \

     -p 8033:8033 \
     -v /path/to/all/models:/models \
     -e MODEL_NAME=/models/model_weights/llama/13B-F \
+    -e SPECULATOR_NAME=/models/speculator_weights/llama/llama-13b-accelerator \
     -e FLASH_ATTENTION=true \
     -e PAGED_ATTENTION=true \
     -e DTYPE_STR=float16 \