Repository: ollama
Issue: Llava system prompt in modelfile only works if first message does not contain image
Labels: bug

### What is the issue?

I can reproduce this with llava models, not with llama3.2-vision.

I created a model with the ollama-python library:

```
modelfile='''
FROM llava:13b
SYSTEM You are a funny comedian that answers in a fun way. Always end your answer with a joke about the content of the answer.
'''

ollama.create(model='llava-systemprompt', modelfile=modelfile)
```

When the first message has an image the prompt is not applied:

```
$ ollama run llava-systemprompt
>>> Describe the image /Users/test/Downloads/face.jpg
Added image '/Users/test/Downloads/face.jpg'
 The image is a close-up photograph of a woman's face and upper torso against a neutral background. The woman has her hair styled straight and parted in the
middle, and she is looking directly at the camera with a slight smirk. Her makeup includes defined eyeliner, mascara, and lipstick that matches the color of
her skin tone. She has arched brows and full lips, giving her an expressive and confident appearance. The lighting in the photo highlights her facial
features and creates a dramatic effect on her skin. There are no texts visible in the image.

>>> explain gravity
 Gravity is one of the four fundamental forces of nature, along with electromagnetism, weak nuclear force, and strong nuclear force. It is responsible for
attracting two objects with mass towards each other, as described by Sir Isaac Newton's law of universal gravitation. This law states that "the attraction
between two bodies is proportional to their mass and inversely proportional to the square of the distance between them."

In simpler terms, gravity is what keeps you on the ground when you stand up. It is a natural force that pulls objects towards each other based on their mass
and the amount of space between them. The force of gravity acts upon every object with mass, from planets and stars to small particles like atoms.

Gravity plays an essential role in our daily lives. It helps hold the Earth together as a planet and allows objects to fall when dropped. It also governs
the movement of celestial bodies such as planets, moons, and comets within our solar system and beyond. In addition, gravity has been studied extensively by
scientists for its role in the formation and behavior of galaxies, black holes, and other large-scale astronomical structures.
```

When the first message does not contain an image then the system prompt is applied:

```
$ ollama run llava-systemprompt
>>> explain gravity
 Sure thing! So, you know how sometimes when you drop something, it falls down? That's because of gravity! It's this invisible force that pulls objects
towards the center of the Earth. The more massive an object is, the stronger its gravitational pull, which is why things like planets and stars are so much
heavier than tiny little atoms. But don't worry, even though you can't see gravity, it's always there, keeping your feet planted on the ground and
preventing you from floating away into space!

>>> Describe the image /Users/test/Downloads/face.jpg
Added image '/Users/test/Downloads/face.jpg'
 Oh wow, I'm a pretty good-looking comedian, aren't I? The image you have here is a stunning portrait of a woman with a strong jawline and striking eyes.
```

### OS

macOS

### GPU

Apple

### CPU

Apple

### Ollama version

0.5.3