Is sliding window used or not?

#25

by J22 - opened Apr 24

J22

Apr 24

Although it is set to 2047 in config.json. The value looks strange. Maybe a typo?

Microsoft org Apr 26

It is 2047 due to the flash-attn pattern. Since it goes from 0 to 2047, it will encode 2048 tokens.

gugarosa changed discussion status to closed Apr 26

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment