Repository: EleutherAI/lm-evaluation-harness
Issue: a verifiable task: generating QR codes in SVG
Root cause: Validation Gap

I thought it would be fun and vibe-coded quickly. One night later, I feel like I am not that far anymore, but the only thing that really works is [a botched-up script](https://github.com/Huge/lm-evaluation-harness/blob/qrcode-in-svg-task/qrcode_eval.py) ClaudeLM produced in frustration.

Here is the main stich I made( the LMs to write): https://github.com/Huge/lm-evaluation-harness/tree/qrcode-in-svg-task/lm_eval/tasks/qrcode

Here is how it fails:
```
cd ~/lm-evaluation-harness && .venv/bin/python -m lm_eval --model openai-chat-completions --model_args
 base_url=https://openrouter.ai/api/v1,model=google/gemma-3-27b-it,chat_formatting_function=None --tasks qrcode --num_fewshot 0 --limit 2 --apply_chat_template 
2025-03-30:04:35:25 WARNING  [__main__:359]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2025-03-30:04:35:25 INFO     [__main__:422] Selected Tasks: ['qrcode']
2025-03-30:04:35:25 INFO     [evaluator:180] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-03-30:04:35:25 INFO     [evaluator:218] Initializing openai-chat-completions model, with arguments: {'base_url': 'https://openrouter.ai/api/v1', 'model': 'google/gemma-3-27b-it', 'chat_formatting_function': 'None'}
2025-03-30:04:35:25 WARNING  [models.openai_completions:116] chat-completions endpoint requires the `--apply_chat_template` flag.
2025-03-30:04:35:25 INFO     [models.api_models:115] Using max length 2048 - 1
2025-03-30:04:35:25 INFO     [models.api_models:118] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2025-03-30:04:35:25 INFO     [models.api_models:133] Using tokenizer None
2025-03-30:04:35:25 INFO     [evaluator:281] qrcode: Using gen_kwargs: {'until': ['</svg>'], 'max_gen_toks': 8192}
2025-03-30:04:35:25 WARNING  [evaluator:300] Overwriting default num_fewshot of qrcode from None to 0
2025-03-30:04:35:25 WARNING  [evaluator:446] Chat template formatting change affects loglikelihood and multiple-choice tasks. See docs/chat-template-readme.md for details.
2025-03-30:04:35:25 INFO     [api.task:426] Building contexts for qrcode on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7928.74it/s]
2025-03-30:04:35:25 INFO     [evaluator:542] Running generate_until requests
Requesting API:   0%|                                                                                                                       | 0/1 [00:00<?, ?it/s]2025-03-30:04:35:25 INFO     [models.api_models:619] Tokenized requests are disabled. Context + generation length is not checked.
2025-03-30:04:35:25 WARNING  [models.api_models:290] Cannot determine EOS string to pass to stop sequence. Manually set by passing `eos_string` to model_args.
Traceback (most recent call last):
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 512, in <module>
    cli_evaluate()
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 432, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 333, in simple_evaluate
    results = evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 553, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/api_models.py", line 623, in generate_until
    outputs = retry(
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/api_models.py", line 381, in model_call
    return response.json()
  File "/home/ubuntu/lm-evaluation-harness/.venv/lib/python3.10/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Requesting API:   0%|                                                                                                                       | 0/1 [00:03<?, ?it/s]
```

I'm in firm belief API key is handled well, but somehow it fails and I'm lost( also due loss of sleep). Any guidence will be appretiated.

Hi! The error occurs here: https://github.com/EleutherAI/lm-evaluation-harness/blob/3816796ebdcb1e7102e2964fc23d8e7a1082eba3/lm_eval/models/api_models.py#L381 while parsing the response from the server into json. You could try printing/pdb `response.text` instead to  check the raw response.

Hey @baberabb , that was a good direction, I went into that spot with debugging and found out the mismatch, and was able to run it like

https://github.com/Huge/lm-evaluation-harness/blob/codespace-qr-benchmark/scripts/my_dumbo_run.py#L100

The error stemmed from the mismatch in ´base_url=https://openrouter.ai/api/v1´ is not extended to ´https://openrouter.ai/api/v1/chat/completions´ , which was unexpected as in most frameworks you set the API_BASE to the former part. 

Should I add some nicer error for the case where html only response is obtained?