Repository: ollama
Issue: "Error loading llama server" when using a T5ForConditionalGeneration architucture model, converted to GGUF format
Labels: bug

### What is the issue?

With the help of https://huggingface.co/spaces/ggml-org/gguf-my-repo I made the https://huggingface.co/iG8R/t5_translate_en_ru_zh_large_1024_v2-Q8_0-GGUF model which was successfully imported into `ollama`.
But when I try to use it, I always get the following error, while all other models work almost perfectly:
```
GGML_ASSERT: C:\a\ollama\ollama\llm\llama.cpp\src\llama.cpp:14882: strcmp(embd->name, "result_norm") == 0
time=2024-07-26T23:33:10.669+03:00 level=INFO source=server.go:617 msg="waiting for server to become available" status="llm server error"
time=2024-07-26T23:33:10.934+03:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000409"
[GIN] 2024/07/26 - 23:33:10 | 500 |    9.3372066s |       127.0.0.1 | POST     "/v1/chat/completions"
```

Here is the full log:
```
time=2024-07-26T23:33:09.450+03:00 level=WARN source=memory.go:115 msg="model missing blk.0 layer size"
time=2024-07-26T23:33:09.451+03:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=H:\OllamaModels\blobs\sha256-cca50b43a8d0071238d9cb22864768dec5a8146f0b9969b83e69a076e267b17e gpu=GPU-60a344b3-0290-00b9-ed05-6b799407d228 parallel=4 available=10883338240 required="702.5 MiB"
time=2024-07-26T23:33:09.451+03:00 level=WARN source=memory.go:115 msg="model missing blk.0 layer size"
time=2024-07-26T23:33:09.451+03:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[10.1 GiB]" memory.required.full="702.5 MiB" memory.required.partial="702.5 MiB" memory.required.kv="48.0 MiB" memory.required.allocations="[702.5 MiB]" memory.weights.total="48.0 MiB" memory.weights.repeating="17179869184.0 GiB" memory.weights.nonrepeating="67.5 MiB" memory.graph.full="128.0 MiB" memory.graph.partial="128.0 MiB"
time=2024-07-26T23:33:09.455+03:00 level=INFO source=server.go:383 msg="starting llama server" cmd="f:\\Ollama\\ollama_runners\\cuda_v11.3\\ollama_llama_server.exe --model H:\\OllamaModels\\blobs\\sha256-cca50b43a8d0071238d9cb22864768dec5a8146f0b9969b83e69a076e267b17e --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 25 --no-mmap --parallel 4 --port 62423"
time=2024-07-26T23:33:09.459+03:00 level=INFO source=sched.go:437 msg="loaded runners" count=1
time=2024-07-26T23:33:09.459+03:00 level=INFO source=server.go:583 msg="waiting for llama runner to start responding"
time=2024-07-26T23:33:09.461+03:00 level=INFO source=server.go:617 msg="waiting for server to become available" status="llm server error"
INFO [wmain] build info | build=3440 commit="d94c6e0c" tid="17472" timestamp=1722025989
INFO [wmain] system info | n_threads=4 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="17472" timestamp=1722025989 total_threads=8
INFO [wmain] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="62423" tid="17472" timestamp=1722025989
llama_model_loader: loaded meta data with 33 key-value pairs and 558 tensors from H:\OllamaModels\blobs\sha256-cca50b43a8d0071238d9cb22864768dec5a8146f0b9969b83e69a076e267b17e (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = t5
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = T5_Translate_En_Ru_Zh_Large_1024_V2
llama_model_loader: - kv   3:                         general.size_label str              = 851M
llama_model_loader: - kv   4:                            general.license str              = apache-2.0
llama_model_loader: - kv   5:                               general.tags arr[str,1]       = ["translation"]
llama_model_loader: - kv   6:                          general.languages arr[str,3]       = ["ru", "zh", "en"]
llama_model_loader: - kv   7:                           general.datasets arr[str,1]       = ["ccmatrix"]
llama_model_loader: - kv   8:                          t5.context_length u32              = 512
llama_model_loader: - kv   9:                        t5.embedding_length u32              = 1024
llama_model_loader: - kv  10:                     t5.feed_forward_length u32              = 2816
llama_model_loader: - kv  11:                             t5.block_count u32              = 24
llama_model_loader: - kv  12:                    t5.attention.head_count u32              = 16
llama_model_loader: - kv  13:                    t5.attention.key_length u32              = 64
llama_model_loader: - kv  14:                  t5.attention.value_length u32              = 64
llama_model_loader: - kv  15:            t5.attention.layer_norm_epsilon f32              = 0.000001
llama_model_loader: - kv  16:        t5.attention.relative_buckets_count u32              = 32
llama_model_loader: - kv  17:        t5.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  18:                  t5.decoder_start_token_id u32              = 0
llama_model_loader: - kv  19:                          general.file_type u32              = 7
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = t5
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,65100]   = ["<pad>", "
", "<unk>", ",", "▁"...
llama_model_loader: - kv  23:                      tokenizer.ggml.scores arr[f32,65100]   = [0.000000, 0.000000, 0.000000, -3.144...
llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,65100]   = [3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  25:            tokenizer.ggml.add_space_prefix bool             = true
llama_model_loader: - kv  26:    tokenizer.ggml.remove_extra_whitespaces bool             = true
llama_model_loader: - kv  27:        tokenizer.ggml.precompiled_charsmap arr[u8,237561]   = [0, 180, 2, 0, 0, 132, 0, 0, 0, 0, 0,...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = true
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  122 tensors
llama_model_loader: - type  f16:    2 tensors
llama_model_loader: - type q8_0:  434 tensors
time=2024-07-26T23:33:09.722+03:00 level=INFO source=server.go:617 msg="waiting for server to become available" status="llm server loading model"
llm_load_vocab: special tokens cache size = 103
llm_load_vocab: token to piece cache size = 0.5577 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = t5
llm_load_print_meta: vocab type       = UGM
llm_load_print_meta: n_vocab          = 65100
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 512
llm_load_print_meta: n_embd           = 1024
llm_load_print_meta: n_layer          = 24
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 2816
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = -1
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 512
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 780M
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 850.68 M
llm_load_print_meta: model size       = 862.32 MiB (8.50 BPW)
llm_load_print_meta: general.name     = T5_Translate_En_Ru_Zh_Large_1024_V2
llm_load_print_meta: EOS token        = 1 '
'
llm_load_print_meta: UNK token        = 2 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 4 '▁'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.44 MiB
llm_load_tensors: offloading 24 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 25/25 layers to GPU
llm_load_tensors:  CUDA_Host buffer size =    67.55 MiB
llm_load_tensors:      CUDA0 buffer size =   794.79 MiB
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   768.00 MiB
llama_new_context_with_model: KV self size  =  768.00 MiB, K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     1.01 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =  1046.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   291.00 MiB
llama_new_context_with_model: graph nodes  = 1350
llama_new_context_with_model: graph splits = 50
GGML_ASSERT: C:\a\ollama\ollama\llm\llama.cpp\src\llama.cpp:14882: strcmp(embd->name, "result_norm") == 0
time=2024-07-26T23:33:10.669+03:00 level=INFO source=server.go:617 msg="waiting for server to become available" status="llm server error"
time=2024-07-26T23:33:10.934+03:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000409"
[GIN] 2024/07/26 - 23:33:10 | 500 |    9.3372066s |       127.0.0.1 | POST     "/v1/chat/completions"
```


### OS

Windows

### GPU

Nvidia

### CPU

Intel

### Ollama version

0.3.0