mistralai

/

Mistral-7B-Instruct-v0.1

like 1.44k

Hi,
Can you please clarify how you use the max_position_embeddings hyperparameter? The config.json file specifies max_position_embeddings=32768 while the paper claims an attention span of 131K tokens (see Section 2 on "Architectural details" → "Sliding Window Attention").
Thanks!