Repository: EleutherAI/lm-evaluation-harness
Issue: Errors when trying to evaluate local T-MAC gguf model
Root cause: Configuration Error

I am having trouble evaluating a local gguf conversion of the T-MAC model using `lm_eval`.

I have built and installed the code from https://github.com/EleutherAI/lm-evaluation-harness. The basic example

`lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0  --batch_size 8`

runs successfully. I have also built and installed the code from https://github.com/microsoft/T-MAC, including submodules, and successfully run

`python tools/run_pipeline.py -o ./my_data/my_model`

to produce

`./my_data/my_model/ggml-model.i2.gguf`

The command

`3rdparty/llama.cpp/build/bin/main -m ./my_data/my_model/ggml-model.i2.gguf -n 128 -t 4 -p "Who was Vasco da Gama?" -b 1 -ngl 0 -c 2048`

produces sensible results (Portuguese, sailed to India, etc.). I start a llama.cpp server using the command

`3rdparty/llama.cpp/build/bin/server -m ./my_data/my_model/ggml-model.i2.gguf --port 8080 -b 1 -n 128 -t 4 -ngl 0 -c 2048 &`

The option `'-b 1'` is critical; any other setting gives the following error

```
terminate called after throwing an instance of 'dmlc::Error'
  what():  [13:48:27] /home/paulo/repos/T-MAC-1bit/3rdparty/llama.cpp/ggml-tmac.cpp:143: ne11=2, please set '-b 1' before T-MAC support batch size > 1
```

With the option `'-b 1'`, _it seems that_ the server starts successfully, issuing the message below:

```
{"tid":"139641752451264","timestamp":1722286349,"level":"WARN","function":"server_params_parse","line":2640,"msg":"Not compiled with GPU offload support, --n-gpu-layers option will be ignored. See main README.md for information on enabling GPU BLAS support","n_gpu_layers":-1}
[13:52:29] /home/my_user/repos/T-MAC-1bit/3rdparty/llama.cpp/ggml-tmac.cpp:38: ggml_tmac_init
{"tid":"139641752451264","timestamp":1722286349,"level":"INFO","function":"main","line":2918,"msg":"build info","build":2851,"commit":"d230bc6"}
{"tid":"139641752451264","timestamp":1722286349,"level":"INFO","function":"main","line":2925,"msg":"system info","n_threads":4,"n_threads_batch":-1,"total_threads":128,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "}
llama_model_loader: loaded meta data with 25 key-value pairs and 288 tensors from ./my_data/my_model/ggml-model.i2.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = bitnet
llama_model_loader: - kv   1:                               general.name str              = bitnet-b1_58-3B
llama_model_loader: - kv   2:                         bitnet.block_count u32              = 26
llama_model_loader: - kv   3:                      bitnet.context_length u32              = 2048
llama_model_loader: - kv   4:                    bitnet.embedding_length u32              = 3200
llama_model_loader: - kv   5:                 bitnet.feed_forward_length u32              = 8640
llama_model_loader: - kv   6:                bitnet.attention.head_count u32              = 32
llama_model_loader: - kv   7:             bitnet.attention.head_count_kv u32              = 32
llama_model_loader: - kv   8:                      bitnet.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv   9:    bitnet.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 32
llama_model_loader: - kv  11:                          bitnet.vocab_size u32              = 32002
llama_model_loader: - kv  12:                   bitnet.rope.scaling.type str              = linear
llama_model_loader: - kv  13:                 bitnet.rope.scaling.factor f32              = 1.000000
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,32002]   = ["<unk>", "
", "
", "<0x00>", "<...
llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 32000
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  24:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type  f16:    1 tensors
llama_model_loader: - type   i2:  182 tensors
llm_load_vocab: special tokens definition check successful ( 261/32002 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = bitnet
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32002
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 2048
llm_load_print_meta: n_embd           = 3200
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 26
llm_load_print_meta: n_rot            = 100
llm_load_print_meta: n_embd_head_k    = 100
llm_load_print_meta: n_embd_head_v    = 100
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 3200
llm_load_print_meta: n_embd_v_gqa     = 3200
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 8640
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 2048
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 3B
llm_load_print_meta: model ftype      = I2
llm_load_print_meta: model params     = 3.32 B
llm_load_print_meta: model size       = 965.21 MiB (2.44 BPW) 
llm_load_print_meta: general.name     = bitnet-b1_58-3B
llm_load_print_meta: BOS token        = 1 '
'
llm_load_print_meta: EOS token        = 2 '
'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 32000 '</line>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.14 MiB
llm_load_tensors:        CPU buffer size =   965.22 MiB
.................................................................................
llama_new_context_with_model: n_batch is less than GGML_KQ_MASK_PAD - increasing to 32
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 32
llama_new_context_with_model: n_ubatch   = 32
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =   650.00 MiB
llama_new_context_with_model: KV self size  =  650.00 MiB, K (f16):  325.00 MiB, V (f16):  325.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.24 MiB
llama_new_context_with_model:        CPU compute buffer size =     9.03 MiB
llama_new_context_with_model: graph nodes  = 942
llama_new_context_with_model: graph splits = 1
{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"init","line":708,"msg":"initializing slots","n_slots":1}
{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"init","line":720,"msg":"new slot","id_slot":0,"n_ctx_slot":2048}
{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"main","line":3015,"msg":"model loaded"}
{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"main","line":3040,"msg":"chat template","chat_example":"
system\nYou are a helpful assistant
\n
user\nHello
\n
assistant\nHi there
\n
user\nHow are you?
\n
assistant\n","built_in":true}
{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"main","line":3768,"msg":"HTTP server listening","n_threads_http":"127","port":"8080","hostname":"127.0.0.1"}
{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"update_slots","line":1807,"msg":"all slots are idle"}
```

I paid most attention to

`{"tid":"139641752451264","timestamp":1722286350,"level":"INFO","function":"main","line":3768,"msg":"HTTP server listening","n_threads_http":"127","port":"8080","hostname":"127.0.0.1"}`

which suggests to me that all is well. I then proceed with the command

`lm_eval --model gguf --model_args model=./my_data/my_model/ggml-model.i2.gguf,base_url=http://localhost:8080 -b 1 --tasks hellaswag --verbosity DEBUG`

which produces

```
2024-07-29:13:54:43,274 INFO     [__main__.py:272] Verbosity set to DEBUG
2024-07-29:13:54:43,403 INFO     [__init__.py:491] `group` and `group_alias` keys in tasks' configs will no longer be used in the next release of lm-eval. `tag` will be used to allow to call a collection of tasks just like `group`. `group` will be removed in order to not cause confusion with the new ConfigurableGroup which will be the offical way to create groups with addition of group-wide configuations.
2024-07-29:13:54:45,311 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,313 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,316 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,319 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,323 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,325 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,328 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,331 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,333 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,336 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,338 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,341 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,343 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,346 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,348 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,351 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,353 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,356 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,359 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,361 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,365 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,368 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,371 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,374 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,377 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,380 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,383 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,386 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,389 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,392 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,395 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,398 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,401 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,404 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,407 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,410 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,413 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,416 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,420 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,423 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,426 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,428 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,432 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,436 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,440 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,443 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,447 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,451 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,453 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,456 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,460 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,463 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,467 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,471 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,475 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,478 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,481 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,484 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,488 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,491 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,495 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,499 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,503 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,506 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,508 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,512 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,515 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,518 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,521 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,525 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,528 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,532 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,535 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,538 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,542 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,545 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,549 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,552 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,556 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,559 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,563 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,566 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,570 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,573 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,577 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,581 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,584 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:45,588 INFO     [__init__.py:512] The tag xnli is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:51,975 DEBUG    [__init__.py:520] File winogenerated.yaml in /home/my_user/repos/lm-evaluation-harness/lm_eval/tasks/model_written_evals/winogenerated could not be loaded
2024-07-29:13:54:51,979 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:51,982 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:51,986 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:51,989 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:51,993 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:51,997 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,000 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,004 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,007 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,011 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,015 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,030 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,040 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,044 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,054 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,058 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,062 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,072 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,083 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,087 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,091 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,095 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,098 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,102 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,105 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,109 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,113 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,117 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,121 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,124 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,128 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,132 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,136 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,140 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,144 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:52,149 INFO     [__init__.py:512] The tag mmlu is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-07-29:13:54:53,575 INFO     [__main__.py:376] Selected Tasks: ['hellaswag']
2024-07-29:13:54:53,581 INFO     [evaluator.py:158] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-07-29:13:54:53,582 INFO     [evaluator.py:195] Initializing gguf model, with arguments: {'model': '/public_data/bitnet_models/bitnet-b1_58-3B/ggml-model.i2.gguf', 'base_url': 'http://localhost:8080'}
2024-07-29:13:55:02,021 INFO     [evaluator.py:274] Setting fewshot random generator seed to 1234
2024-07-29:13:55:02,022 DEBUG    [cache.py:33] requests-hellaswag-0shot-rank0-world_size1-tokenizer is not cached, generating...
2024-07-29:13:55:02,022 INFO     [task.py:423] Building contexts for hellaswag on rank 0...
{"tid":"140505735203008","timestamp":1722286502,"level":"WARN","function":"launch_slot_with_task","line":887,"msg":"Max tokens to predict exceeds server configuration","params.n_predict":400,"slot.n_predict":128}
{"tid":"140505735203008","timestamp":1722286502,"level":"INFO","function":"launch_slot_with_task","line":1039,"msg":"slot is processing task","id_slot":0,"id_task":0}
{"tid":"140505735203008","timestamp":1722286502,"level":"INFO","function":"update_slots","line":2091,"msg":"kv cache rm [p0, end)","id_slot":0,"id_task":0,"p0":0}
terminate called after throwing an instance of 'dmlc::Error'
  what():  [13:55:02] /home/my_user/repos/T-MAC-1bit/3rdparty/llama.cpp/ggml-tmac.cpp:143: ne11=32, please set '-b 1' before T-MAC support batch size > 1
Stack trace:
  [bt] (0) 3rdparty/llama.cpp/build/bin/server(+0x303383) [0x560434faf383]
  [bt] (1) 3rdparty/llama.cpp/build/bin/server(+0x300dc1) [0x560434facdc1]
  [bt] (2) 3rdparty/llama.cpp/build/bin/server(+0x1e7494) [0x560434e93494]
  [bt] (3) 3rdparty/llama.cpp/build/bin/server(+0x297c3f) [0x560434f43c3f]
  [bt] (4) 3rdparty/llama.cpp/build/bin/server(+0x2969a8) [0x560434f429a8]
  [bt] (5) 3rdparty/llama.cpp/build/bin/server(+0x13687d) [0x560434de287d]
  [bt] (6) 3rdparty/llama.cpp/build/bin/server(+0x69871) [0x560434d15871]
  [bt] (7) 3rdparty/llama.cpp/build/bin/server(+0x6fcda) [0x560434d1bcda]
  [bt] (8) 3rdparty/llama.cpp/build/bin/server(+0x3482b) [0x560434ce082b]


100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10042/10042 [00:04<00:00, 2425.91it/s]
2024-07-29:13:55:07,278 DEBUG    [evaluator.py:428] Task: hellaswag; number of requests on this rank: 40168
2024-07-29:13:55:07,287 INFO     [evaluator.py:457] Running loglikelihood requests
  0%|                                                                                                                                                                                                                             | 0/40168 [00:00<?, ?it/s]{"tid":"139730564163776","timestamp":1722286507,"level":"INFO","function":"launch_slot_with_task","line":1039,"msg":"slot is processing task","id_slot":0,"id_task":0}
{"tid":"139730564163776","timestamp":1722286507,"level":"INFO","function":"update_slots","line":2091,"msg":"kv cache rm [p0, end)","id_slot":0,"id_task":0,"p0":0}
terminate called after throwing an instance of 'dmlc::Error'
  what():  [13:55:07] /home/my_user/repos/T-MAC-1bit/3rdparty/llama.cpp/ggml-tmac.cpp:143: ne11=28, please set '-b 1' before T-MAC support batch size > 1
Stack trace:
  [bt] (0) 3rdparty/llama.cpp/build/bin/server(+0x303383) [0x558a61951383]
  [bt] (1) 3rdparty/llama.cpp/build/bin/server(+0x300dc1) [0x558a6194edc1]
  [bt] (2) 3rdparty/llama.cpp/build/bin/server(+0x1e7494) [0x558a61835494]
  [bt] (3) 3rdparty/llama.cpp/build/bin/server(+0x297c3f) [0x558a618e5c3f]
  [bt] (4) 3rdparty/llama.cpp/build/bin/server(+0x2969a8) [0x558a618e49a8]
  [bt] (5) 3rdparty/llama.cpp/build/bin/server(+0x13687d) [0x558a6178487d]
  [bt] (6) 3rdparty/llama.cpp/build/bin/server(+0x69871) [0x558a616b7871]
  [bt] (7) 3rdparty/llama.cpp/build/bin/server(+0x6fcda) [0x558a616bdcda]
  [bt] (8) 3rdparty/llama.cpp/build/bin/server(+0x3482b) [0x558a6168282b]


2024-07-29:13:55:07,579 ERROR    [gguf.py:68] RequestException: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
{"tid":"139641752451264","timestamp":1722286512,"level":"INFO","function":"launch_slot_with_task","line":1039,"msg":"slot is processing task","id_slot":0,"id_task":0}
{"tid":"139641752451264","timestamp":1722286512,"level":"INFO","function":"update_slots","line":2091,"msg":"kv cache rm [p0, end)","id_slot":0,"id_task":0,"p0":0}
terminate called after throwing an instance of 'dmlc::Error'
  what():  [13:55:12] /home/my_user/repos/T-MAC-1bit/3rdparty/llama.cpp/ggml-tmac.cpp:143: ne11=28, please set '-b 1' before T-MAC support batch size > 1
Stack trace:
  [bt] (0) 3rdparty/llama.cpp/build/bin/server(+0x303383) [0x5650bc634383]
  [bt] (1) 3rdparty/llama.cpp/build/bin/server(+0x300dc1) [0x5650bc631dc1]
  [bt] (2) 3rdparty/llama.cpp/build/bin/server(+0x1e7494) [0x5650bc518494]
  [bt] (3) 3rdparty/llama.cpp/build/bin/server(+0x297c3f) [0x5650bc5c8c3f]
  [bt] (4) 3rdparty/llama.cpp/build/bin/server(+0x2969a8) [0x5650bc5c79a8]
  [bt] (5) 3rdparty/llama.cpp/build/bin/server(+0x13687d) [0x5650bc46787d]
  [bt] (6) 3rdparty/llama.cpp/build/bin/server(+0x69871) [0x5650bc39a871]
  [bt] (7) 3rdparty/llama.cpp/build/bin/server(+0x6fcda) [0x5650bc3a0cda]
  [bt] (8) 3rdparty/llama.cpp/build/bin/server(+0x3482b) [0x5650bc36582b]


2024-07-29:13:55:12,861 ERROR    [gguf.py:68] RequestException: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2024-07-29:13:55:17,866 ERROR    [gguf.py:68] RequestException: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /v1/completions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f404a4f2670>: Failed to establish a new connection: [Errno 111] Connection refused'))
  0%|                                                                                                                                                                                                                             | 0/40168 [00:15<?, ?it/s]
Traceback (most recent call last):
  File "/home/my_user/micromamba/envs/T-MAC-1BIT_env/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/evaluator.py", line 468, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/models/gguf.py", line 80, in loglikelihood
    response = self.gguf_completion(context=context, continuation=continuation)
  File "/home/my_user/repos/lm-evaluation-harness/lm_eval/models/gguf.py", line 71, in gguf_completion
    raise Exception(f"Failed to get a valid response after {retries} retries.")
Exception: Failed to get a valid response after 3 retries.
[1]   Aborted                 (core dumped) 3rdparty/llama.cpp/build/bin/server -m ./my_data/my_model/ggml-model.i2.gguf --port 8080 -b 1 -n 128 -t 4 -ngl 0 -c 2048
[2]-  Aborted                 (core dumped) 3rdparty/llama.cpp/build/bin/server -m ./my_data/my_model/bitnet-b1_58-3B/ggml-model.i2.gguf --port 8080 -b 1 -n 128 -t 4 -ngl 0 -c 2048
[3]+  Aborted                 (core dumped) 3rdparty/llama.cpp/build/bin/server -m ./my_data/my_model/bitnet-b1_58-3B/ggml-model.i2.gguf --port 8080 -b 1 -n 128 -t 4 -ngl 0 -c 2048
```

Besides the final connection failure, the error

`what():  [13:55:12] /home/my_user/repos/T-MAC-1bit/3rdparty/llama.cpp/ggml-tmac.cpp:143: ne11=28, please set '-b 1' before T-MAC support batch size > 1`

seems noteworthy. I tried variants such as `--batch_size 1` and also adding `batch_size=1` to the `--model_args` parameter, as in 

`lm_eval --model gguf --model_args model=./my_data/my_model/ggml-model.i2.gguf,base_url=http://localhost:8080,batch_size=1 --tasks hellaswag --verbosity DEBUG`

This produces a different error

```
[cut]
TypeError: ABCMeta object got multiple values for keyword argument 'batch_size'
```

A specific question is, therefore: How can I pass the `-b 1` option to my run of `lm_eval`? It seems that simply passing it at the command line does not work. Additionally, does anyone spot anything else fishy that I should look into?

You need to comment out a line in their code.

For example (check your Traceback), 
https://github.com/EleutherAI/lm-evaluation-harness/blob/1208afd34ce132e598fcd7e832762630a35d01c6/lm_eval/evaluator.py#L208C1-L209C1

It's actually a bug in their code, the `batch_size` arg is already in `model_args`.