Increase `sliding_window` to 32k
#24
by
alpindale
- opened
Some backends, such as vLLM, do not dynamically adjust the sliding window param based on the max context length provided. Modifying it here seems to fix that.
Some backends, such as vLLM, do not dynamically adjust the sliding window param based on the max context length provided. Modifying it here seems to fix that.
Though the sliding window is indeed 4k context length? Seems like a vLLM issue to me. I think this would break loaders that actually support SWA