mistralai

/

Mistral-7B-v0.1

like 3.22k

I have two questions about sliding window attention (SWA):

1: Is the SWA used during pertaining all the time? I mean, in every pretraining step.
2: If not, is the SWA used during pertaining in a certain stage? For example, pretrain the model with the full-attention for sometime then use SWA to do pretraining.

Thanks!