Repository: ollama
Issue: generate calls with llava:latest randomly come back incomplete with format='json', stream=False
Labels: bug

### What is the issue?

I'm seeing random cases of `llava:lastest` failing to complete a response with `generate` REST calls set to `format='json'` and `stream=false`. I hadn't seen this issue before updating to 0.1.41. I was previously using 0.1.34 without issue, but I don't want to downgrade because I need the support for models added since 0.1.34.

The incomplete responses will just have a bunch of tab escape chars in it and `'done': False` at the end.

This appears to happen at random. Here's two sample outputs from a script where I preload the model, sequentially make some generate calls on a small set of images, and then unload it. (json formatting also specified in prompt)
```
 {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:42.216132768Z', 'response': '', 'done': True, 'done_reason': 'load'}

rgb_0000.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:44.008509117Z', 'response': '{"objects_left_to_right:" \n   \t \t    \t     \t \t        \t      \t  \t  \t  \t \t               \n   \t \t', 'done': False}

rgb_0001.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:45.727229854Z', 'response': '{"objects_left_to_right": ["milk carton", "coffee mug", "cereal box", "red can of soup"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 22626, 28729, 7548, 266, 548, 345, 1115, 496, 2443, 290, 786, 548, 345, 28717, 397, 282, 3890, 548, 345, 893, 541, 302, 18851, 2242, 28752, 28705], 'total_duration': 1648939331, 'load_duration': 1219805, 'prompt_eval_count': 1, 'prompt_eval_duration': 915187000, 'eval_count': 37, 'eval_duration': 591675000}

rgb_0002.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:47.232468394Z', 'response': '{"objects_left_to_right": ["carton of milk", "box of cereal", "can of coffee"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 14040, 266, 302, 10992, 548, 345, 2858, 302, 23730, 282, 548, 345, 4230, 302, 7045, 2242, 28752, 28705], 'total_duration': 1413297144, 'load_duration': 1279552, 'prompt_eval_count': 1, 'prompt_eval_duration': 910665000, 'eval_count': 29, 'eval_duration': 454852000}

rgb_0003.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:48.88326051Z', 'response': '{"objects_left_to_right":["can of soda", "bottle of yellow liquid", "can of fruit juice"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 2221, 4230, 302, 268, 10769, 548, 345, 28726, 1562, 291, 302, 9684, 12213, 548, 345, 4230, 302, 10607, 18342, 2242, 28752, 28705], 'total_duration': 1516567390, 'load_duration': 1059030, 'prompt_eval_count': 1, 'prompt_eval_duration': 907548000, 'eval_count': 32, 'eval_duration': 517114000}

rgb_0004.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:50.623030453Z', 'response': '{"objects_left_to_right:" \n   \t  \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t', 'done': False}

rgb_0005.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:52.203536774Z', 'response': '{"objects_left_to_right": ["hot sauce", "bag of cheese puffs", "box of cereal"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 10672, 16042, 548, 345, 19001, 302, 13628, 284, 27355, 548, 345, 2858, 302, 23730, 282, 2242, 28752, 28705], 'total_duration': 1516168660, 'load_duration': 1159419, 'prompt_eval_count': 1, 'prompt_eval_duration': 911261000, 'eval_count': 29, 'eval_duration': 470313000}

rgb_0006.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:53.93931645Z', 'response': '{"objects_left_to_right:" \n \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   ', 'done': False}

rgb_0007.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:55.814683448Z', 'response': '{"objects_left_to_right:" \n   \t  \t \t     \t     \t     \t \t      \t \t \t \n    \t \t \t ', 'done': False}

rgb_0008.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:57.501844772Z', 'response': '{"objects_left_to_right": ["Canned food item", "Another canned item", "A package of some sort"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 28743, 5928, 2887, 2515, 548, 345, 2820, 1039, 541, 12097, 2515, 548, 345, 28741, 3660, 302, 741, 3127, 2242, 28752, 28705], 'total_duration': 1622249222, 'load_duration': 1473655, 'prompt_eval_count': 1, 'prompt_eval_duration': 948370000, 'eval_count': 32, 'eval_duration': 535461000}

rgb_0009.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:59.042547123Z', 'response': '{"objects_left_to_right": ["Box of cereal", "Can of condiments", "Container of coffee"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 4114, 302, 23730, 282, 548, 345, 6325, 302, 2076, 8447, 548, 345, 5738, 302, 7045, 2242, 28752, 28705], 'total_duration': 1448892449, 'load_duration': 963616, 'prompt_eval_count': 1, 'prompt_eval_duration': 921549000, 'eval_count': 29, 'eval_duration': 477919000}

 {'model': 'llava:latest', 'created_at': '2024-06-06T18:25:59.130897904Z', 'response': '', 'done': True, 'done_reason': 'load'}
```

```
 {'model': 'llava:latest', 'created_at': '2024-06-06T18:39:53.81329136Z', 'response': '', 'done': True, 'done_reason': 'load'}

rgb_0000.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:39:55.663006368Z', 'response': '{"objects_left_to_right": ["bottle of yellow liquid", "container with red lid and contents that look like snacks or cereal", "candy bar"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 28726, 1562, 291, 302, 9684, 12213, 548, 345, 6734, 395, 2760, 16006, 304, 11337, 369, 913, 737, 2502, 6507, 442, 23730, 282, 548, 345, 28717, 10087, 2843, 2242, 28752, 28705], 'total_duration': 1728225663, 'load_duration': 2756184, 'prompt_eval_count': 1, 'prompt_eval_duration': 916689000, 'eval_count': 41, 'eval_duration': 659909000}

rgb_0001.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:39:57.479962227Z', 'response': '{"objects_left_to_right:"\n \t\t\t\t \t\t\t\t\t \t\t\t \t\t\t \t\t\t \t\t\t \t\t', 'done': False}

rgb_0002.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:39:59.143084575Z', 'response': '{"objects_left_to_right":["can of juice", "carton of eggs", "box of cereal"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 2221, 4230, 302, 18342, 548, 345, 14040, 266, 302, 14636, 548, 345, 2858, 302, 23730, 282, 2242, 28752, 28705], 'total_duration': 1529278016, 'load_duration': 2220528, 'prompt_eval_count': 1, 'prompt_eval_duration': 912306000, 'eval_count': 29, 'eval_duration': 463036000}

rgb_0003.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:00.75972763Z', 'response': '{"objects_left_to_right": ["can of cat food", "jar of mustard", "can of tuna"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 4230, 302, 5255, 2887, 548, 345, 13036, 302, 1580, 488, 548, 345, 4230, 302, 261, 5892, 2242, 28752, 28705], 'total_duration': 1452936522, 'load_duration': 2250729, 'prompt_eval_count': 1, 'prompt_eval_duration': 907132000, 'eval_count': 30, 'eval_duration': 488433000}

rgb_0004.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:02.698830315Z', 'response': '{"objects_left_to_right:" \n   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   \t   ', 'done': False}

rgb_0005.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:04.341068166Z', 'response': '{"objects_left_to_right": ["Cheez-It crackers", "Ketchup bottle"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 21124, 5620, 28733, 1313, 9551, 404, 548, 345, 28796, 4455, 715, 10859, 2242, 28752, 28705], 'total_duration': 1507879923, 'load_duration': 2813286, 'prompt_eval_count': 1, 'prompt_eval_duration': 920745000, 'eval_count': 26, 'eval_duration': 439638000}

rgb_0006.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:06.138490256Z', 'response': '{"objects_left_to_right": ["a jar of peanut butter", "a bottle of mustard or ketchup", "two boxes of cereal"]} ', 'done': True, 'done_reason': 'stop', 'context': [733, 16289, 28793, 6620, 272, 6697, 356, 272, 2401, 477, 1749, 298, 1103, 390, 264, 1274, 28723, 1992, 19571, 1413, 9292, 28723, 28705, 13, 2287, 439, 28747, 28705, 13, 2287, 9830, 13721, 28730, 1828, 28730, 532, 28730, 1246, 4825, 7367, 4478, 1928, 477, 1749, 548, 345, 3423, 1928, 477, 1749, 548, 345, 4081, 1928, 477, 1749, 2242, 28752, 733, 28748, 16289, 4490, 28739, 13721, 28730, 1828, 28730, 532, 28730, 1246, 1264, 7367, 28708, 22578, 302, 757, 276, 329, 11488, 548, 345, 28708, 10859, 302, 1580, 488, 442, 446, 4455, 715, 548, 345, 10861, 13446, 302, 23730, 282, 2242, 28752, 28705], 'total_duration': 1633272828, 'load_duration': 3837813, 'prompt_eval_count': 1, 'prompt_eval_duration': 945974000, 'eval_count': 39, 'eval_duration': 627627000}

rgb_0007.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:08.051022806Z', 'response': '{"objects_left_to_right:" \n   \t  \t  \t  \t  \t  \t  \t  \t \t \t \t \t \t \t ', 'done': False}

rgb_0008.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:09.97092121Z', 'response': '{"objects_left_to_right:" \n   \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t', 'done': False}

rgb_0009.png: {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:11.927354394Z', 'response': '{"objects_left_to_right:" \n \t   \t       \t         \t           \t     \t          \t               \t         \t           \t           \t\t    \t\t\t   ', 'done': False}
 {'model': 'llava:latest', 'created_at': '2024-06-06T18:40:11.96521445Z', 'response': '', 'done': True, 'done_reason': 'load'}
```

This may be related to #3513. https://github.com/ollama/ollama/issues/3513

### OS

Linux

### GPU

Nvidia

### CPU

Intel

### Ollama version

0.1.41