Architectural difference with Llama
#20
by
imone
- opened
What is the architectural difference between Mistral and Llama? HF Mistral seems the same as Llama except for sliding window attention.
Also, there’s gqa which is a big difference. Llama 7b, 13b do not contain gqa which mistral does