Repository: ollama
Issue: IBM Granite MoE & Dense-2b is very slow when KV Cache quantization is enabled
Labels: bug

### What is the issue?

I found all Granite MoE models + dense:2b runs extremely slow when KV Cache is enabled, there didn't seems to be any hit on models response quality, just speed, kinda strange

I'm using Windows 11 + RTX 4090

Here is an example using model: granite3.1-moe:3b-instruct-q8_0

### `set OLLAMA_FLASH_ATTENTION=1 && set OLLAMA_KV_CACHE_TYPE=q8_0 && ollama serve`

```
>>> how far is the moon
The distance from Earth to the Moon can vary due to the elliptical shape of its orbit around our planet. On
average, it's about 238,855 miles (384,400 kilometers) away from Earth. However, this is approximately 238,855
miles (384,400 kilometers) at its closest approach and can range up to 252,088 miles (405,696 kilometers) during
its farthest point in its elliptical path.

total duration:       8.3218603s
load duration:        15.7633ms
prompt eval count:    49 token(s)
prompt eval duration: 242ms
prompt eval rate:     202.48 tokens/s
eval count:           130 token(s)
eval duration:        8.005s
eval rate:            16.24 tokens/s
```

### `ollama serve`

```
>>> how far is the moon
The average distance from Earth to the Moon is approximately 238,855 miles (384,400 kilometers). However, it's
important to note that this can fluctuate slightly due to the elliptical nature of its orbit around our planet. At
its closest point, known as perigee, it's about 225,623 miles (363,104 kilometers) away from Earth, while at its
farthest point, called apogee, it can reach up to 252,088 miles (405,696 kilometers).

total duration:       4.2702016s
load duration:        805.8374ms
prompt eval count:    193 token(s)
prompt eval duration: 287ms
prompt eval rate:     672.47 tokens/s
eval count:           142 token(s)
eval duration:        3.115s
eval rate:            45.59 tokens/s
```

---

granite3.1-dense:2b also have the same issue

`ollama run granite3.1-dense:2b-instruct-q8_0 --verbose`

### `set OLLAMA_FLASH_ATTENTION=1 && set OLLAMA_KV_CACHE_TYPE=q8_0 && ollama serve`

```
>>> how far is the moon
As previously mentioned, the average distance from the Earth to the Moon is approximately 238,855 miles (384,400
kilometers). This value remains constant throughout their orbital motion around each other.

total duration:       3.7165709s
load duration:        847.2622ms
prompt eval count:    124 token(s)
prompt eval duration: 93ms
prompt eval rate:     1333.33 tokens/s
eval count:           52 token(s)
eval duration:        2.717s
eval rate:            19.14 tokens/s
```

### `ollama serve`

```
>>> how far is the moon
The average distance from the Earth to the Moon is about 238,855 miles (384,400 kilometers). This distance is
often referred to as the semi-major axis of the Moon's elliptical orbit around the Earth.

total duration:       815.8894ms
load duration:        16.3291ms
prompt eval count:    49 token(s)
prompt eval duration: 286ms
prompt eval rate:     171.33 tokens/s
eval count:           61 token(s)
eval duration:        458ms
eval rate:            133.19 tokens/s
```


### OS

Windows

### GPU

Nvidia

### CPU

AMD

### Ollama version

0.5.4