Token indices sequence length
When trying to make an AWQ quant, I am getting this error:
Token indices sequence length is longer than the specified maximum sequence length for this model (385 > 255). Running this sequence through the model will result in indexing errors.
Never seen this before. Not sure what to suggest about it.
I'm not sure what this error could mean. This is a simple merge. I know that it is compatible with GGUF conversion and models of similar lineage are compatible with exl2 conversion. Sorry I can't be of more help.
I was able not able resolve this by updating the tokenizer_config.json, line 39 - and change from 255 to 385. The error just tells gives me a higher value, and if I increase it further, then I am in an infinite loop. Lol, got up to 77697 and gave up on it. Must be an issue with my AWQ, thank-you for looking into it.
I can see other people with this issue, and they just put a massive number here, like this:
https://huggingface.co/Crystalcareai/CrystalMistral-14b/blob/main/tokenizer_config.json#L33
This doesn't seem like a logical solution to me, but I was able to make an AWQ quant from this model, by making this change and it appears to perform very well.
AWQ quant is up: https://huggingface.co/solidrust/Mewthree-7B-AWQ