Sliding window is null in config file

#16
by RonanMcGovern - opened

I notice in config.json that the sliding window is set to null. Does this mean that the defacto sliding window is the full embedding input length of 32k?

FWIW, running Mistral v0.1 there always seemed to be issues running contexts above the sliding window length (e.g. see this issue), so maybe that's related?

The hugging face docs for Mistral say that if set to None, it will default to 4096: https://huggingface.co/docs/transformers/main/en/model_doc/mistral
I am not sure what is the intention here

Hi @DreamGenX . Can you tell me which part of the doc specifies that?

@KurtGD1915 It says under sliding window that it defaults to 4096, but looking at the code, it is just default param value (so if you set it to None explicitly, it will stay none). I have fixed the link above that actually does the inference, where you can see that sliding window is disabled if it's None or larger than content length.

Sign up or log in to comment