Repository: EleutherAI/lm-evaluation-harness
Issue: Generation length is limited to 2048 tokens.Qwen3 model accuracy is low
Root cause: Interface Contract Mismatch

Hi Team,
I am using local-completions to evaluate my model deployed in local.
The generated response is limited to 2048 tokens,and not able to increase it throught model_args or gen_kwargs max_length=120000 also.
Did anyone face this issue?


Not with the bf16 or my own AWQ quants, but with Qwen's 32B-AWQ model, it seems I'm hitting some weird behavior:

```
~$ lm_eval --model local-completions --tasks gsm8k --model_args model=Qwen/Qwen3-32B-AWQ,base_url=http://127.0.0.1:11435/v1/completions,max_retries=5,timeout=1200,max_length=32768,num_concurrent=200 --num_fewshot 5  --limit 10 --log_samples --output_path=./results
INFO 05-03 17:04:42 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-03 17:04:42 [__init__.py:239] Automatically detected platform cuda.
2025-05-03:17:04:48,528 WARNING  [lm_eval.__main__:316]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2025-05-03:17:04:48,528 INFO     [lm_eval.__main__:379] Selected Tasks: ['gsm8k']
2025-05-03:17:04:48,529 INFO     [lm_eval.evaluator:169] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-05-03:17:04:48,529 INFO     [lm_eval.evaluator:206] Initializing local-completions model, with arguments: {'model': 'Qwen/Qwen3-32B-AWQ', 'base_url': 'http://127.0.0.1:11435/v1/completions', 'max_retries': 5, 'timeout': 1200, 'max_length': 32768, 'num_concurrent': 200}
2025-05-03:17:04:48,529 INFO     [lm_eval.models.api_models:115] Using max length 32768 - 1
2025-05-03:17:04:48,529 INFO     [lm_eval.models.api_models:133] Using tokenizer huggingface
2025-05-03:17:04:54,037 WARNING  [lm_eval.evaluator:275] Overwriting default num_fewshot of gsm8k from 5 to 5
2025-05-03:17:04:54,038 INFO     [lm_eval.api.task:420] Building contexts for gsm8k on rank 0...
100%|█████████| 10/10 [00:00<00:00, 535.92it/s]
2025-05-03:17:04:54,057 INFO     [lm_eval.evaluator:517] Running generate_until requests
Requesting API: 100%|█████████| 10/10 [00:06<00:00,  1.55it/s]
fatal: not a git repository (or any of the parent directories): .git
2025-05-03:17:05:01,082 INFO     [lm_eval.loggers.evaluation_tracker:209] Saving results aggregated
2025-05-03:17:05:01,083 INFO     [lm_eval.loggers.evaluation_tracker:290] Saving per-sample results for: gsm8k
local-completions (model=Qwen/Qwen3-32B-AWQ,base_url=http://127.0.0.1:11435/v1/completions,max_retries=5,timeout=1200,max_length=32768,num_concurrent=200), gen_kwargs: (None), limit: 10.0, num_fewshot: 5, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |    0|±  |     0|
|     |       |strict-match    |     5|exact_match|↑  |    0|±  |     0|
```

With qwen/qwen3-32b from OpenRouter I get these messages spammed in console:

`2025-05-27:00:24:32 WARNING  [models.api_models:515] Context length (2058) + continuation length (1) > max_length (2047). Left truncating context.`

and then it crashes with following stacktrace:
`Requesting API:   0%|                                                                             | 0/549473 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/InfroLab/bench310/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/__main__.py", line 449, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/evaluator.py", line 338, in simple_evaluate
    results = evaluate(
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/evaluator.py", line 570, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/api/model.py", line 382, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 621, in _loglikelihood_tokens
    asyncio.run(
  File "/home/InfroLab/bench310/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/InfroLab/bench310/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 565, in get_batched_requests
    return await tqdm_asyncio.gather(*tasks, desc="Requesting API")
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tqdm/asyncio.py", line 79, in gather
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tqdm/asyncio.py", line 79, in <listcomp>
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
  File "/home/InfroLab/bench310/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/__init__.py", line 420, in exc_check
    raise retry_exc.reraise()
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/__init__.py", line 187, in reraise
    raise self.last_attempt.result()
  File "/home/InfroLab/bench310/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/InfroLab/bench310/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 487, in amodel_call
    else self.parse_logprobs(
  File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/openai_completions.py", line 76, in parse_logprobs
    sorted(out["choices"], key=itemgetter("index")), ctxlens
KeyError: 'choices'
Requesting API:   0%|                                                                             | 0/549473 [11:32<?, ?it/s`

Here is a launch command:
`lm_eval --model local-completions --tasks m_mmlu_ru,m_mmlu_es,m_mmlu_en,m_mmlu_fr,m_mmlu_hi,hellaswag_ru,hellaswag_es,hellaswag,hellaswag_fr,hellaswag_hi,arc_ru,ai2_arc,arc_es,arc_fr,arc_hi,truthfulqa,truthfulqa_es_mc1,truthfulqa_fr_mc1,truthfulqa_hi_mc1,truthfulqa_ru_mc1,truthfulqa_es_mc2,truthfulqa_fr_mc2,truthfulqa_hi_mc2,truthfulqa_ru_mc2,belebele_rus_Cyrl,belebele_eng_Latn,belebele_hin_Deva,belebele_spa_Latn,belebele_fra_Latn --model_args model=qwen/qwen3-32b,base_url=https://openrouter.ai/api/v1/completions,num_concurrent=8,max_retries=3,tokenized_requests=False`

lm-eval installed from git with a commit 8bc4afff22e73995883de41018388428e39f8a92

Also, another model qwen/qwen3-30b-a3b shows these error during same lm-eval command benchmark:
`{"error":{"message":"Provider returned error","code":400,"me
tadata":{"raw":"{\"error\":{\"object\":\"error\",\"type\":\"invalid_request_error\",\"message\":\"2 request validation errors
: Missing required input field: 'prompt'; Extra inputs are not permitted, field: 'messages', value: [{'role': 'user', 'conten
t': '1 मार्च को, एक आदमी और एक ठेकेदार एक समझौते पर हस्ताक्षर किए जिसमें यह उल्लेख किया गया था कि ठेकेदार हर आठ निर्दिष्ट समुद्र तट घर में एक दो-स्तरीय रेडवुड डेक निर् नाम से होंगे, जिसमें सभी काम 1 मई तक पूरे होंगे। संविदा निम्नलिखित प्रदान करता है:\\\"हर दोहरी टेरेस पर लागत $2,500 है, प्रत्येक आठ समुद्र तट घर मेंतान किया जाना चाहिए।\\\" 25 मार्च की सुबह, जब ठेकेदार और उसके सहायक चौथे टेरेस पर काम पूरा कर रहे थे, तब आदमी एक मुकाम पर गया क्योंकि उन्होंने अ और दशमलव के अनुरूप उसके पास समान नकदी वाली एक ताकत वाली समुद्र तट घरों में चार में से एक पर टेरेस निर्माण करना चाहते हैं। अगर आप उस अंतिम महत्य तक पहुंच सकते हैं, तो मैं आपको उस समय $10,000 वापस करूंगा।\\\" ठेकेदार ने उत्तर दिया, \\\"सहमत हुए। हम हाथ  [96/1857$
वपूर्ण समयासे पहलेहउसमेंसमयितकसपहुंच सकतेघहैंों तो मैंाआपकोाउसरसमयया था और वापसअकरूंगा।र घरोंठेकेदारसनेकेउत्तरमदियाा काम सहमतरहुए।यहमथहाथ2मिलाते हैं।\\\" और हाथ मि में आठों में से चार को पूरी तरह से नष्ट कर दिया। इसके अतिरिक्त, तूफान ने ठेकेदार द्वारा निर्मित टाउन बीच हाउस पर 85 फीसदी काम पूरक
आदमीसकेासमुद्रभतट घरोंसेमेंघसेयचारोकोएपूरीऔतरहनसेीनष्टीकरपदिया।कइसकेहअतिरिक्त। तूफानआनेमठेकेदारठेद्वारासेनिर्मितिटाउनजबीच हाउस पर कहनफीसदी\कामतूपूरकरने के बाद तोड़ दिया। दो ठेकेदाो गएएथेदऔरदंउनकी सभी उपकरणोंाको बहशेगयाीथा।मुफिरतआदमीोंकोरठेकेदारपूसेापत्रींमिलासकजिसमेंकृउनकामुयहशकहना था किए तूफामेरी उपकरणों को तबाह कर  घर है। मैंरेएकदमरदंगबरहएगया हूँरेऔरकेताउन में2शेष0तीनजेंसमुद्र"तटफघरोंसेपरहकाम पूरामनहींकेकरिसकता। कृपया मुझेशशहरयमें मेरेजकिएठेगए काम के लिए $20,000 हैसमयितकसपहुंच सप्रतिशततपूरेमैंकामककेालिएसबकायाथराशि केासाथकरूंगा।र घरोंठेकेदारसनेकेउत्तरमदियाा काम सहमतप्रतिशत:पूरे:काम[के लिए बकाया राशि के साथ घरों में से चार को पूरी तरह से नष्ट क━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━निर्मित टाउन बीच हाउस प `


I tried test requests to these API using openai-python package and they return all the necessary data.