Small max new token limit
#3
by
mwgupta
- opened
max_new_token parameter upper bound seems to be 512 (couldn't find documentation online about upper bound, bigger than 512 returns an index out of range error). Any suggestions on workarounds?
The maximum token limit is indeed 2048 tokens. If you encounter an "index out of range" error, it could be due to the text being cut off before reaching that limit. Make sure to count both input and output tokens, and if the total exceeds 2048, you'll need to truncate or omit parts of the text. Keep in mind that very long conversations may lead to incomplete replies.