Compression script limits context length to 4098?
#1
by
Kayvane
- opened
Why did you decide to limit the context length in this way, is it possible to release another version (versions) with other context lengths?
The context length is still 32k for this model https://huggingface.co/neuralmagic/Mistral-7B-Instruct-v0.3-FP8/blob/3d03cee39c9d23f9d8409bc73a0881c58cf721f4/config.json#L13. The compression script just controls the size of calibration samples.
mgoin
changed discussion status to
closed