What is the expected ratio of GPU memory and KV cache tokens for this model?
#44
by h01-10w - opened
In my comparison with the Qwen's tokenizers via vLLM inference, it seems that this model tokenizer (Tekkenizer) uses more memory for the smaller token budget. Is is expectable?
In numbers, this model gives about 870-900 tokens, compared to 5000+ tokens of the Qwen models.
Using fp8 cache quantization, this model gives about 1750 tokens.
Some package versions for reference:
vLLM version: 0.24.0 (latest, docker image)
mistral-common: 1.11.5
soundfile: 0.14.0
soxr : 1.1.0
librosa: 0.11.0
transformers: 5.12.1