Text Generation
Transformers
PyTorch
mpt
Composer
MosaicML
llm-foundry
custom_code
text-generation-inference

error when trying inference locally

#8
by hussainwali1 - opened

RuntimeError: Please install flash-attn==1.0.3.post0 and triton==2.0.0.dev20221202

If you would like to use the triton implementation, you will need to do as the message says and install flash attention and triton. Otherwise you can stick with the torch implementation without issue.

daking changed discussion status to closed

Sign up or log in to comment