mistralai

/

Mixtral-8x7B-Instruct-v0.1

like 3.9k

idk if my problem is related to this.. but from the last hour the model was acting so weird and generating random numbers.. here are some examples

We manually reverted in hf.co/chat the version to fix the issue.

Seems like sliding window: None causes catastrophic failure in flash attention (potentially not respecting requests boundaries, or just outputting garbage) . We'll figure out out to fix this upstream tomorrow.

ok we are waiting for the changes tomorrow I'm working on a project

Hi @issa130 and @medmac01 ! Which tools do you use to run inference? Is it TGI? transformers?

I'm using inference API it was working very fine but when I saw that someone set the sliding_window to null the model started acting weird