Repository: ollama
Issue: "CUDA error: an illegal memory access was encountered" during image processing via minicpm-v:latest model
Labels: bug

### What is the issue?

The crash happens while processing the _png_ image via minicpm-v:latest (1862d7d5fee50b69f6e3007ec999145ab38f17688251495f87669eb81e9dd97c) model. It occurs only on specific _png_ image. Other images are processed without any issues.
The same image was processed without issue via llava:latest (8dd30f6b0cb19f555f2c7a7ebda861449ea2cc76bf1f44e262931f45fc81d081)

Example of request:

POST: http://OLLAMA_IP:11434/api/generate

Body:
```
{
  "model": "minicpm-v:latest",
  "prompt": "Extract all visible text to markdown blocks.",
  "images": [
    "iVBORw0KGgoAAAANSUhEUgAAA2oAAACUCAYAAADiW9r/AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAB2tSURBVHhe7d1Na9xWuMDxp5e7KBS6VBICl5JNuxmXhHZxoYULDnjo5kKXpba/Q0GfQtDvELt02WUZgw0X2l1DQj2bdlO6CYm1LBS66z3P0TnS0cvMHI3ksez8f+BkRiNppPMmPTpHmnf+NQQAAAAAMBn/4f4HAAAAAEwEgRoAAAAATAyBGgAAAABMDIEaAAAAAEwMgRoAAAAATAyBGgAAAABMDIEaAAAAAEwMgRoAAAAATAyBGgAAAABMDIEaAAAAAEwMgRoAAAAATAyBGgAAAABMzDv/Gu71KJanqSwepJI+TdwUL5eLLJNF7t4mc0nTfWnOBQAAAABvu1F71PLzTE4u3ZsaF6TdP5LM/J9lqcxlIdnp0n0OAAAAAPBGCtQ0EEslO/PdZQ2XC1nkicwPZm5CIvvHc0nM9IsViwAAAADA22qEQM0PaZzJUXYkPhQL5fmVic2eyCwc55jM5EmSy4tLIjUAAAAACI0QqCWyn+pwxu4gTV29NsHY/aTzfrT8tQniAAAAAAAlnvoIAAAAABNDoAYAAAAAE0OgBgAAAAATs5NA7d6DRORNLl2PDUke3HOvAAAAAABqJ4FakphgLH8hyzBSy5fyIk/kyR4/eQ0AAAAAod0MfdybyzzJZfHswvWq5XLxbCG5mb5PnAYAAAAANTu6R00f4Z/KXBaSmf/TNDOv5pIernqgPwAAAAC8vd7513CvAQAAAAATwFMfAQAAAGBiCNQAAAAAYGII1AAAAABgYgjUAAAAAGBiCNQAAAAAYGII1AAAAABgYgjUAAAAAGBiCNQAAAAAYGII1AAAAABgYgjUAAAAAGBi3vnXcK+3c3ki6elSJJlLmu5LYifmcpFlssjtm8Zn05GfZ5KdmY2c6Patotv9TI4lfepS+xr3Y8i6m9tp5ReSPRM5vrH0XspJemL+FZkdZnK0V0wdYnmaysuP/brGXz+67Dadr6OOddYPjKaZvlpPTy7Ni70jyQ5ndhr6Gzsd6+3nFPLpLrXh1blYcpCaunDl9i2ReZrKvlYNfw63zt7/yOzy/+5ImnS7De1x8zh05epKkbcd263nW9nClIIgv/30Gz0P2+QOnketyosI19Kjlp8/q4I0dT+ZaGG4ffQgZivqxHVupx4QbEG9K/Qg6E4qgB5uSz2+rUjf24D2E9PwVrUXd+487O673qGPekUsy7h6iQmZyZGWSfN3F68KAsB10ivcHNdvi0SS+8Wre8mKy+X+PM3+HZkjZKHMZ5vX/+Wm4iYlyb3iRWznR7JvgjLNw349OJiWkYc+zmRpCkWtN81odV2u7AKsuunDZfRqR3jVrdnN2z0saXPXafdywXSna/nmNlXr6N6HVftcX8+6LtFqvZVi/tllsB/HIs/KqyUd6yu3w9kwvKR/GnVv52efifz8c21ikI9VXlmrviuZmYPIUpa6Gj9Pc3+69rmmXS5q+3jwSrJyGMi6dTW22dIg8LG8DNb/+NcgfzvSOj7/vfb39iufyq9Dv28urzK/Pv/94Xd0bFPPMuRtqsfl53smny9NPutEv+7Gd84Oj0ROq3Ru1zOnsW1rv2ONsIwcPX4hJ2XZ1zyvTm6q9QfrLLdJ09JW0M563Mr3YLmZ2b/lpS5Uzbu6DnoddXHFdtXXX5WJmUmnpUknVa6/NVQq3P6wfqXy8Cz4/lY617dPy8P8ddZOP6Nedlakl7W6nUzOgrz5+GWwD/U8VJvTtq25TLMdC8vGkZxU+9OYT9X31+iYp5UPrfSN2I9mfVmbtoVmGQ/rxvD2U+v19eVTXB6EZXhNG76hfm7Ow3YadG1/fT2b86epSCNxy/n60U7LQn3fq22JTBNn7TaX6Wa2IX0oi7L8uW2qlclV2xloluGO7VndZq1uL3R7N9XplZrf11huuzpjuH0Vd+z061lZ7oMyqustzxUDK4dNqog2Zm05r31/eK6x4Xtr5a061quu5frWkXL+zvOARr3syvO1x0CnNo/5/PCJvDit8mLd9jXdzMNETJQ/t4Uqlxe2cXPypbywb2fy2H3eNTQiNwf/eiKNRzOwWZDttPNqWqtgqnzhKptptB8XObD8tdrG/PKF2RsjeSIz+3HXvuWm0epYdyzdhrDBsusLCpwWnNrnhp12UZ+2QUwaRdOK3DhY2f1oTlOmfNggTekVpWYDbek+Z3KxxaZUeeg10q+nVjkxaV2l0Tb53z6wq+b3rC+foeb+mfcmiKinfWOercpQz3rsG04jeXDPzNjO52XQcJf6bFvzO2KZdKyCNNWdJ+PK3Umguid6YXxzHew6ATFqZdBrr9/zQZq2aw/1yrym58ZyVFieNr6/8d3Nz7U8tMv+NbSTrX0weRiUkc1p29Y6oVO2Pe4qe0GAoBrzxdRf/b5WPjT2a+N+TLz9vI58Km3IA6+VF2a5mPrTWk7V0iemLR+n7CdPTZqYZYoTw0T2U1NWNwU/a6xPkz7brPkZlj99r9vWnNZRhzwtI5vafH3frCvm001ls1edDnTWTV2u85ymZ51xPWQ+UCnydv3FiW3FtjGtfG3tk2rvV3d739Y81uty25W3Dl3nGs18auZdTHlqzWM+t0HadkYO1IpGID1wR3qNUFcUotnHRTORv1yWG++DmeRgbhuR6l43jUC1Aps/E5XbtZuEiMqIPkwmLew69SpO/fvys4XLhKW8bMxT7u+bvNj+vSduG1+6ZUxD/rLYy9mBi8wvF8W+abTuv8tdqViedTUERdr6tNSrCl3d2XplpZZO5faaAn1WbE05j1l+rjOZgljsd4SNadS9nf/7v+Z/fyXG7bM2NkuzjE0GO1+xvmLZpSw6Drjltuu63rwq0qlMQ7c/ZmrtAkAPtfVbPv2adP/99/nlGge/IG99epTlvXf+G3kuV/ZFlfZ+vdVFgc3ls8bV0XJ/zXfkzWnmW81kY7sy1L8eV/NpGSkvcnTtU6nvttW/o49y/UEd6yqr3eLqcYvPE3fFeWM75S96dZXB10UpqgnX7yZZQd3SbfTlrJ0GvowEWvXSfLcv/+U+dK0rcC3tZFAOy3L/orgIFHUMaLt6XWxJ2Y75ffHrrVnz/VH115Q3dwLZ+j6TXjbIitmPqbef69Jpy3yqrMuDQFf9Cc5ZSrX6E5GHMW35NseIXViXJj232ZffMn20EWlNe+XSqimuzV/fZq1uL/rVaceUy2fNulmWcxN0tk7wg+3aWGeGs4Gd/x6XR93Hv4g2JqachzqOMWFnxkpd5c3X8cF1pH4eEHM+GnMMLPerY9u38Z/u/93be2wbtKUt9PumYvhgJpEne7rbVXCTHBxXB1rbG7ewJ3c2MfZ8lg9XnRDq1a7Uvqq4TEiKQmmZStm+ImmYeZ4kWoCKgjy73+wpDDLSRuuL4rXnGia/y9FMoSh6KvV1ItpPUG5b2Vtpvvs0ldbeFTtXvFkjLo2Kd5tVDZLtXTkrXnvFCWWwsnD/1P2H5lNzgPBpqJXCVLp993Fv4fp9+XRvt1EG5ca9B+ZVcPKzVf6XeerT3jWQ9kMvonwG/AWTcH+7pllblaEt6vHePDixDpcvLuDY10/N67PgClbfbat9Rw+1OlbtQ6usjsocTA6q9Imrg3rl1dWEziuAofr6Q2EZVvbgZF/plcyOHrtAtWwxysAf+K0gSOhKT+9a2skw730b4t5u2775+u3bMT3Ql+nfFH6/eT1PzEmAWbEGR/tPI+qvmVf326SwzP1JVpjfRn4esR8Tbz+vI59Ka/PATTfWteGVZv2JOUfY3JZfS9kfwXjHNX+uZ17pxW17ktw1bYXINr9PmxXqVaedslyaunBcBkAmGDTlY6FtsL14P7N5b41dZ8YS0cZElfNSWEeK8+Bl0M6vE5a3mTkHSExwplun2Tu4joTtgEn5mPPRzeWpWk/3tvd3g7+jVg1vtFfvysjYDw2sNG+CtRVo53J59ab4X7taU9O4rn5yTn34Y1l5bUWM4Q40O9R5lb03n0Yj6boqEzINx7G/gqNsZS3ypt1Lc5usyn/TMJZX3VRxkNf9bQ4FWF8+PTecraZrWpxNZWjcenxPHvZYfJzybTRu4r6ZtmgdXwersrE+SOvHDoex640/4emS59350S89d91Orm7fkqfHZe+Qsgd6m06NYTRGfaht9bCHQkT99UHu1tx+3Nn2c/NxaH0eDBXTBse05evs/hxhuOY214dZF7qmbce3+du2WX3qdEvzYR/2QsMtEtXG9DnX2FbjfMRd4Igzch1x56NbHwN7bXvdDQZqpqkKhj9euMg4eTxrFejiykjFd0lfG72yqFe3Gn+269IHlIbv+iy7ewPh8MeF296ypyJUdgeHfxFDobZWdfXW/moHjQjr0mgLZTdy+Bdx064fo23/gn240aEhffTN/3D+sstdDyKNoQDGuvI5zHZlaLt6vOok6kpedS4+UvlepXHx4NrbonXW1MH83A//qtJjyNCLQjUcpiyHQRkcw8r03HU72bt9K4ZQteerhsx49YsGpjyFQUVM/e1z0rdhP259+7nlcWhtHgwV2wZvasu9XZf9Mex0m9e1+UParPg63dK8yDz44sqOxbQxOznXaFx0MecQnZf4Ri5vq89HY8rTiovIq7Y9wo0GamUPU24y3F7BC7pZza77Xqn87Fl1c3Me3NvQDHxcV6d9WZ6oxCuDq3J7DB02FFxFKa8EmwNE0WVdDc2qscN49MXSdfFWwx5Vue2msPt9KyP1DTerqt49BHY4pr7ITUHzKVNdxYu9ghqTRqHO7SzzKRgKWp4YVFdpNl1ZbKWXVhx/shGUhV1oBiGbbJX/zXTWoQhlI1FcPYoun9vYqgxtUY8bfC9LeALTqt8jle+NwnK/bh+CA7Uf996ldz02YupgGfCUQzuq4RhbKw805sTIDWOphqD107kPQXp6u24n+7ZvhXabpQd6f2LX2oZgX6qTnWLIV1T9La/MhieM9bIedSy7xe3ndvkUWJMHQ0XlYURbPrTs34SdbnNMm9+zzarqas867YTl8llZN826/PZFj6ragXV1PKKNudZzjUB40ag8lrqRd+OWt4jz0ajyVF1c7tz2LdzcPWqWCcxMBL4sI9R6Ia7uQzEV0SRWbQSqOaiUVwW1cthEKArS1mxw5e6baYx5Lu+Psb9joRnWMSa2wRYi/9S0ZgUNxsU39y0c17qSbejNvKbxmLtJ64XjpItlS2VFixCTRqFgO4/cJLNkkU96YuDH7bbSMwzau5XloysvdtwgFmOa9cpeXG5slf/lMh3l3DVcxe+sxJXP/rYrQ7H1eJVq+XX1e0D51vl1OTNfzKOX2/dDVA18Wedj0z+sHyPWwVyDWzukvJEWQ7iDt4agrXzsq3MfzAFOEz88mo3YTj52k9bq275ZVdnrurehPJkoddUDF1BH1d/quNn6vrKsR+zHJNvPdHOeqq3yKbQmDwaKaoMj2nKblxvLvi6vbWORdtfTY9XD0PraS0yb37PNCtvjXnXaCcple7nmcNfNNOiwT54ML6KMJjgPa617cxuTXF7nuUagY/3lyLuRy1t5H1nrO/35aFx5Wr2e7dxsj5pRXoEw2oW/uFmxeQJjn8gSFixTOdJGQdPfXtimWHedMNmbSH3QYAp1rXtXT+78U2yaTwMKDnbtfTONjKkk4RhotemEbdv9sroqpG5/xMlpaGMaGZ3b2XVSrnnX6jrWfI/Zz+75WuXj2pg8PG5ue6xt8r97GZuvPg/7lM9tbFWGIuvxSu18XlW+xijfa5n1HYXp29y2Zvqbb54ftsvIoHpsbKqDOqSt9rktI247yqfR9mX2tVZX9eTQb0f/JwXq76yFZdm+bw1z3X07GdO+tXSVvVr6BMy8rbzxyzbLz4r6W3t6m9co65v34za3n1vmk7cuD4aKysOItnzFPJvK/s3a8TZ35VutHsS1WdHHk9ry3XRfO7epo67dCLNfMXmxsY2JKufDNfOmXsdHLm8bz0cjj4Gt9Zj5Os4DYg3/weuh9AqGXhGpJQYA7JC2Q2cPaye6uEbBU8KqA68OMylu0I4+4b5FtOdHr7R3nwBiF+5mHmjPyEIemhPWG+9Rw+hsr9rrOW3GW+zGetTKcaQ2SNMDc8xwBQAYn33Mb/NJXbg+5f0l5ligQ2v0WJD6p2iNc68Q8FawPeTjPS0RU1Lc91V/SineNjcWqBXjuB3tQr1jV08B3BKXJ3LyZt4aPo3rtGLol5k+iXttgFthKSenV6bOMBrpLsrPn8niPufHb7ubH/oIAAAAAKi58YeJAAAAAADqCNQAAAAAYGII1AAAAABgYgjUAAAAAGBiCNQAAAAAYGII1AAAAABgYm7R4/n11/dPzL8is8NMju5fSJYtJJeZHGX6GyK5XGT+B1NXSeSzz0R+/tnMlMwlTffv5A/cLk9TefmxSaM9N2FyNuUlAAAA8HajR+1O0WA1lZNL9xYAAADArXSLArV78tB2fyXy8L6d0JDIfppJlhV/6YHrK9OeMzctM0HMf79XTMZN2pSXAAAAwNttnKGPuR+6VrFD2oKhd/m5CZTOgjm2GHqoQ/pOLv3wODd8bsV6yu9rfF6bfizyrNzuROZpKvvhipr7tXck2WE1MK/YHrPkQSrz11nZk6Xv06dJ+Xk4rVIN/7Oi0qNjeGe5TY31WauHEsblR/v7mvla7qPZjvTBIljn+mGMffISAAAAeNsMD9Q6grRCFfi0ggLvGk/MNwZqnYLg4vJE0tN62GMF6wsDsbpEkiSXvPE1ZZCzMs3WBTer78ErgsCr6EAtLj+6Ar9CGHSuTgODwAsAAADYyuChj/nlCxtw6Ml7MbzQBwa5vHpjX8jV6yIoKOdJ58XJe/5Clh3xwq5o4FTbHhOWvLRBhwmKzooQpZwnS2WuM+ULWbQCEw2IdJ5q3/O8Oc2kg4vclmdFkFalme+lWsrifEWC5Et5YRcywU9tGfPR6yvzr36f20Z9Z7e7O+iLyY/lqQ/S/H6YP9ebmJ89k4vWZmpgXsxXDju94fwFAAAAbqvBgVrytDjZ90P90o5emHsPihP3/MzMl5r5L2cu2GgMNdwlE/DM/RC+JJF77qXlgyKj2Cf9q3qzfMBV2nvsAqKZPPbr7Jpm+WCwSg/9871SRdDVIdkv0kx7qLS3L1imr835UW3j7DAI9vZMmtnPc3lx2UyDeZmXyd4TetEAAACAAYY/TMQFDesCh+TpcdnTo6oApXto3dQ1g6nkQS3Ms7qmRXmT2962Nh2K6ILGriGZPcTnR/NhH4kkPPwDAAAAuHYDA7VqiGA1JK9ruF39iYzVwyjWDPWbhGo4X+0veKDIENWwyuBvxT1d+fnCBVHVNoUP9egnNj+q4asFE0TW3gMAAAC4DgMDtSt55c7rZwcuwLh82eiV0YdgFD1BmQsCyodqGCuH+t2kZCZP7M7ksvCBqNkr36O17ZDDQjUUcnl2Yb5BtdOoyd9XVg0xrIYndmkNzyzF5EewjeW9asblwg3/TOTJXlc4CQAAAGAMAwM1/3tYekJfnPy3h+Ulsn9Q9EB13ZM1+9h9du4+GzisbxzVNldDO13AEt7btqXZgX94x0Iyu25//9tM5rVH+Ff8fWWt7VmhSOus86EfMflRbmMQoPq8SQ6Oe95bWAWHw4JcAAAA4O0wMFDTIXT1oY7aO+Of+rf81YUSjd8fKxRD+LYfvnfNurZ5rMfN64NByidNet2P0vf0oS21tNLt8+soezFNfhw319shJj/sw0uqp0h6Nn9XBJMAAAAAxjHOD16PxP6+1+v5aPeAAQAAAMBtNPypj6PJZfky3/5piQAAAABwR0wmUMvPn8ni/hHD6gAAAAC89SY19BEAAAAAMKmhjwAAAAAARaAGAAAAABNDoAYAAAAAE0OgBgAAAAATQ6AGAAAAABNDoAYAAAAAE0OgBgAAAAATQ6AGAAAAABNDoAYAAAAAE0OgBgAAAAATQ6AGAAAAABPzzr+Gez3Yb7/9Js+fP5c//vhD/v77b3nvvffk0aNH8sknn8hHH33k5gIAAAAArDNaoPbDDz/In3/+KZ9//rl8+OGH8v7778tff/0lv//+u/z000/ywQcfyJdffunmBgAAAACsMkqgpkHaP//8I1999ZWb0vb999/Lu+++S7AGAAAAABsMDtR0uOOPP/4o33zzjZuy2rfffitffPFF1DDI5WkqJ5fuTUsi8zSV/cS9BQAAAIA7ZHCg9t1339mhjp9++qmbstovv/xih0J+/fXXbkqkyxNJzx5Kmu6bEA0AAAAA7rbBT33UB4dooBZD59P5x7GUk/RELs4zE8ClkmYXkucXkqWZXORuFkN75tLTpXtnaNCn87u/1b12AAAAAHAzBgdq+nRHfXBIDJ1P5x/PUhav55JlmQnQInrbNEg7FTnS+e0yc7k6rQd2AAAAAHDTBgdq+gh+fbpjDJ1P5x/T7OOZe7VJLhdnS5kdHkm5RLIvxwciCzMdAAAAAKZicKCmv5Om953F0Pl0/ptxJa9yNxQyGPqYndGdBgAAAGBaBgdq+mPW+jtpMXQ+nf8mzQ7dsMfw7zC2Vw4AAAAArt/gQE0fta8/Zq2/k7aOfq7zxTya/3rck4eJyFVODxoAAACAaRscqCn9EWv9MWv9nTR9BL+/Z03/1/c6fSc/dp0kJhzL5cWlC8byC1mUT3VMZP9gJvnZs+DhIblcZKlk5wRvAAAAAKZj8O+ohfTHr58/f24fwa9Pd9QHh+g9aTrccVBPWufvqBWP55fDTI723CRln+zoHg6SzOXo8Qs50SdDuuGN+XlWuy8tOUglfcqvswEAAACYjlEDNQAAAADAcKMMfQQAAAAAjIdADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACbmnX8N93qw3377TZ4/fy5//PGH/P333/Lee+/Jo0eP5JNPPpGPPvrIzQUAAAAAWGe0QO2HH36QP//8Uz7//HP58MMP5f3335e//vpLfv/9d/npp5/kgw8+kC+//NLNDQAAAABYZZRATYO0f/75R7766is3pe3777+Xd999l2ANAAAAADYYHKjpcMcff/xRvvnmGzdltW+//Va++OKLyGGQuVxkmSxy97ZmJkfZkfl3veVpKidyJNnhpjkBAAAAYDoGB2rfffedHer46aefuimr/fLLL3Yo5Ndff+2mrFMEaq8OMjnac5N6IlADAAAAcBsNfuqjPjhEA7UYOp/OPyYNxtI0+Dtduk+alnKybr7Lk9p6Ti7ddAAAAADYscGBmj7dUR8cEkPn0/nHYnvM3swlzTLJ9E97zkzA1Q6ytHfuRK4O0mK+LJX5mxPJzt24Sg3STkWO/HrSuVydZnLROewSAAAAAK7X4EBNH8GvT3eMofPp/H20eszSKoCaHWpQtS9J8VZk7/HG+9YKieyb9aRPdUkTxJ0tzbqCe96SfTk+EFmY6QAAAACwa3fiHrX83ARsZ1X3lwZwukztHjXba+YCr2RuAj4f4OmQyBPzb4c97m8DAAAAsHuDe9T0x6z1d9Ji6Hw6/1g0QNNetuzsnhu2uOZJkBp0uWGNSb4w/6e1+9Rs75wf+uj/CNIAAAAA3IDBgZo+al9/zFp/J20d/Vzni3s0f4xcli9zF2BtflR/Kdkv7mmzvWwvZSn35GEicpVzQxoAAACAaRgcqCn9EWv9MWv9nTQd3ujvWdP/9b1Ov64fu64CrOKBId13lelnafXwEGP5q5nT3tOWyP7BTPKzZ8HDQ9rzAwAAAMCuDL5HLaQ/fv38+XP7CH59uqM+OOTRo0d2uGP/nrSIe9TyC8myhZmzMDtM5eFZJov7xb1ltXvUGvPW71MzHzfuc0sOUvewEQAAAADYrVEDNQAAAADAcKMMfQQAAAAAjIdADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmhkANAAAAACaGQA0AAAAAJoZADQAAAAAmReT/AYCE+fr/UmbjAAAAAElFTkSuQmCC"
  ],
  "stream": false,
  "keep_alive": 0
}
```

Ollama log:
```
ollama                       | [GIN] 2024/10/04 - 08:41:03 | 200 |  31.22349636s |  192.168.100.20 | POST     "/api/generate"
ollama                       | time=2024-10-04T08:41:23.009Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
ollama                       | time=2024-10-04T08:41:23.168Z level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-5d1301f5-9e77-83c1-9e2f-eff9c34008d5 library=cuda total="7.4 GiB" available="1.2 GiB"
ollama                       | time=2024-10-04T08:41:23.808Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0 gpu=GPU-5d1301f5-9e77-83c1-9e2f-eff9c34008d5 parallel=1 available=7857242112 required="5.8 GiB"
ollama                       | time=2024-10-04T08:41:23.808Z level=INFO source=server.go:103 msg="system memory" total="15.6 GiB" free="3.1 GiB" free_swap="0 B"
ollama                       | time=2024-10-04T08:41:23.810Z level=INFO source=memory.go:326 msg="offload to cuda" layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[7.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.8 GiB" memory.required.partial="5.8 GiB" memory.required.kv="112.0 MiB" memory.required.allocations="[5.8 GiB]" memory.weights.total="3.5 GiB" memory.weights.repeating="3.1 GiB" memory.weights.nonrepeating="425.3 MiB" memory.graph.full="303.2 MiB" memory.graph.partial="728.5 MiB"
ollama                       | time=2024-10-04T08:41:23.812Z level=INFO source=server.go:388 msg="starting llama server" cmd="/usr/lib/ollama/runners/cuda_v12/ollama_llama_server --model /root/.ollama/models/blobs/sha256-262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 29 --mmproj /root/.ollama/models/blobs/sha256-f8a805e9e62085805c69c427287acefc284932eb4abfe6e1b1ce431d27e2f4e0 --no-mmap --parallel 1 --port 33077"
ollama                       | time=2024-10-04T08:41:23.813Z level=INFO source=sched.go:449 msg="loaded runners" count=1
ollama                       | time=2024-10-04T08:41:23.813Z level=INFO source=server.go:587 msg="waiting for llama runner to start responding"
ollama                       | time=2024-10-04T08:41:23.813Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
ollama                       | INFO [main] build info | build=10 commit="3f6ec33" tid="140630251470848" timestamp=1728031283
ollama                       | INFO [main] system info | n_threads=4 n_threads_batch=4 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140630251470848" timestamp=1728031283 total_threads=8
ollama                       | INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="33077" tid="140630251470848" timestamp=1728031283
ollama                       | ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ollama                       | ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ollama                       | ggml_cuda_init: found 1 CUDA devices:
ollama                       |   Device 0: Tesla P4, compute capability 6.1, VMM: yes
ollama                       | time=2024-10-04T08:41:24.065Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
ollama                       | llama_model_loader: loaded meta data with 22 key-value pairs and 339 tensors from /root/.ollama/models/blobs/sha256-262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0 (version GGUF V3 (latest))
ollama                       | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama                       | llama_model_loader: - kv   0:                       general.architecture str              = qwen2
ollama                       | llama_model_loader: - kv   1:                               general.name str              = model
ollama                       | llama_model_loader: - kv   2:                          qwen2.block_count u32              = 28
ollama                       | llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
ollama                       | llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 3584
ollama                       | llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 18944
ollama                       | llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 28
ollama                       | llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 4
ollama                       | llama_model_loader: - kv   8:                       qwen2.rope.freq_base f32              = 1000000.000000
ollama                       | llama_model_loader: - kv   9:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
ollama                       | llama_model_loader: - kv  10:                          general.file_type u32              = 2
ollama                       | llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
ollama                       | llama_model_loader: - kv  12:                         tokenizer.ggml.pre str              = qwen2
ollama                       | llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,151666]  = ["!", "\"", "#", "$", "%", "&", "'", ...
ollama                       | llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,151666]  = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
ollama                       | llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
ollama                       | llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 151644
ollama                       | llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 151645
ollama                       | llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 128244
ollama                       | llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 0
ollama                       | llama_model_loader: - kv  20:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
ollama                       | llama_model_loader: - kv  21:               general.quantization_version u32              = 2
ollama                       | llama_model_loader: - type  f32:  141 tensors
ollama                       | llama_model_loader: - type q4_0:  197 tensors
ollama                       | llama_model_loader: - type q6_K:    1 tensors
ollama                       | llm_load_vocab: special tokens cache size = 25
ollama                       | llm_load_vocab: token to piece cache size = 0.9309 MB
ollama                       | llm_load_print_meta: format           = GGUF V3 (latest)
ollama                       | llm_load_print_meta: arch             = qwen2
ollama                       | llm_load_print_meta: vocab type       = BPE
ollama                       | llm_load_print_meta: n_vocab          = 151666
ollama                       | llm_load_print_meta: n_merges         = 151387
ollama                       | llm_load_print_meta: vocab_only       = 0
ollama                       | llm_load_print_meta: n_ctx_train      = 32768
ollama                       | llm_load_print_meta: n_embd           = 3584
ollama                       | llm_load_print_meta: n_layer          = 28
ollama                       | llm_load_print_meta: n_head           = 28
ollama                       | llm_load_print_meta: n_head_kv        = 4
ollama                       | llm_load_print_meta: n_rot            = 128
ollama                       | llm_load_print_meta: n_swa            = 0
ollama                       | llm_load_print_meta: n_embd_head_k    = 128
ollama                       | llm_load_print_meta: n_embd_head_v    = 128
ollama                       | llm_load_print_meta: n_gqa            = 7
ollama                       | llm_load_print_meta: n_embd_k_gqa     = 512
ollama                       | llm_load_print_meta: n_embd_v_gqa     = 512
ollama                       | llm_load_print_meta: f_norm_eps       = 0.0e+00
ollama                       | llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
ollama                       | llm_load_print_meta: f_clamp_kqv      = 0.0e+00
ollama                       | llm_load_print_meta: f_max_alibi_bias = 0.0e+00
ollama                       | llm_load_print_meta: f_logit_scale    = 0.0e+00
ollama                       | llm_load_print_meta: n_ff             = 18944
ollama                       | llm_load_print_meta: n_expert         = 0
ollama                       | llm_load_print_meta: n_expert_used    = 0
ollama                       | llm_load_print_meta: causal attn      = 1
ollama                       | llm_load_print_meta: pooling type     = 0
ollama                       | llm_load_print_meta: rope type        = 2
ollama                       | llm_load_print_meta: rope scaling     = linear
ollama                       | llm_load_print_meta: freq_base_train  = 1000000.0
ollama                       | llm_load_print_meta: freq_scale_train = 1
ollama                       | llm_load_print_meta: n_ctx_orig_yarn  = 32768
ollama                       | llm_load_print_meta: rope_finetuned   = unknown
ollama                       | llm_load_print_meta: ssm_d_conv       = 0
ollama                       | llm_load_print_meta: ssm_d_inner      = 0
ollama                       | llm_load_print_meta: ssm_d_state      = 0
ollama                       | llm_load_print_meta: ssm_dt_rank      = 0
ollama                       | llm_load_print_meta: ssm_dt_b_c_rms   = 0
ollama                       | llm_load_print_meta: model type       = ?B
ollama                       | llm_load_print_meta: model ftype      = Q4_0
ollama                       | llm_load_print_meta: model params     = 7.61 B
ollama                       | llm_load_print_meta: model size       = 4.12 GiB (4.65 BPW) 
ollama                       | llm_load_print_meta: general.name     = model
ollama                       | llm_load_print_meta: BOS token        = 151644 '
'
ollama                       | llm_load_print_meta: EOS token        = 151645 '
'
ollama                       | llm_load_print_meta: UNK token        = 128244 '<unk>'
ollama                       | llm_load_print_meta: PAD token        = 0 '!'
ollama                       | llm_load_print_meta: LF token         = 148848 'ÄĬ'
ollama                       | llm_load_print_meta: EOT token        = 151645 '
'
ollama                       | llm_load_print_meta: max token length = 256
ollama                       | llm_load_tensors: ggml ctx size =    0.30 MiB
ollama                       | llm_load_tensors: offloading 28 repeating layers to GPU
ollama                       | llm_load_tensors: offloading non-repeating layers to GPU
ollama                       | llm_load_tensors: offloaded 29/29 layers to GPU
ollama                       | llm_load_tensors:  CUDA_Host buffer size =   291.59 MiB
ollama                       | llm_load_tensors:      CUDA0 buffer size =  3926.95 MiB
ollama                       | llama_new_context_with_model: n_ctx      = 2048
ollama                       | llama_new_context_with_model: n_batch    = 512
ollama                       | llama_new_context_with_model: n_ubatch   = 512
ollama                       | llama_new_context_with_model: flash_attn = 0
ollama                       | llama_new_context_with_model: freq_base  = 1000000.0
ollama                       | llama_new_context_with_model: freq_scale = 1
ollama                       | llama_kv_cache_init:      CUDA0 KV buffer size =   112.00 MiB
ollama                       | llama_new_context_with_model: KV self size  =  112.00 MiB, K (f16):   56.00 MiB, V (f16):   56.00 MiB
ollama                       | llama_new_context_with_model:  CUDA_Host  output buffer size =     0.59 MiB
ollama                       | llama_new_context_with_model:      CUDA0 compute buffer size =   303.22 MiB
ollama                       | llama_new_context_with_model:  CUDA_Host compute buffer size =    11.01 MiB
ollama                       | llama_new_context_with_model: graph nodes  = 986
ollama                       | llama_new_context_with_model: graph splits = 2
ollama                       | INFO [main] model loaded | tid="140630251470848" timestamp=1728031297
ollama                       | time=2024-10-04T08:41:37.378Z level=INFO source=server.go:626 msg="llama runner started in 13.57 seconds"
ollama                       | ggml_cuda_compute_forward: SCALE failed
ollama                       | CUDA error: an illegal memory access was encountered
ollama                       |   current device: 0, in function ggml_cuda_compute_forward at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:2326
ollama                       |   err
ollama                       | /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:102: CUDA error
ollama                       | [GIN] 2024/10/04 - 08:41:54 | 500 | 31.825048562s |  192.168.100.20 | POST     "/api/generate"
ollama                       | time=2024-10-04T08:41:59.776Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.001895153 model=/root/.ollama/models/blobs/sha256-262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0
ollama                       | time=2024-10-04T08:42:00.027Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.252338468 model=/root/.ollama/models/blobs/sha256-262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0
ollama                       | time=2024-10-04T08:42:00.276Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.501732338 model=/root/.ollama/models/blobs/sha256-262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0
```

### OS

Docker

### GPU

Nvidia

### CPU

Intel

### Ollama version

0.3.12