Repository: EleutherAI/lm-evaluation-harness
Issue: Reasoning trace consuming all tokens leads to unhandled error
Root cause: Validation Gap

I'm experimenting with reasoning models using the chat completions endpoint. I found that when the model consumes all tokens in the reasoning trace, the endpoint returns a valid json with empty content. See this partial response json:

```
...
  'message': {
    'annotations': None,
    'audio': None, 
    'content': None, 
    'function_call': None, 
    'reasoning_content' : "Lots and lots of words here ....."
     ...
     }
...
```

It seems that `lm-eval` cannot process this correctly, it does not identify the return of a valid `json` with `None` content as a correct generation that failed to generate an answer (like getting a time-up when doing a test). Due to this `lm-eval` will:
- Retry the generation, that will end-up in the same result in some cases (if zero temperature is set).
- Fail to communicate the problem to the user.
- Fail to add the response to the list of valid responses.

At the end of the retries, `lm-eval` will output the following trace:
```sh
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/__main__.py", line 455, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/evaluator.py", line 357, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/evaluator.py", line 585, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/api/model.py", line 295, in _fn
    rem_res = getattr(self.lm, attr)(remaining_reqs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/models/api_models.py", line 778, in generate_until
    return re_ord.get_original(res)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/lm-evaluation-harness/lm_eval/models/utils.py", line 541, in get_original
    assert all(cov)
           ^^^^^^^^
AssertionError
```

Which is not very informative. I have tracked the problem to this line:

https://github.com/EleutherAI/lm-evaluation-harness/blob/3bc7cc8a72c66bac8d5b830cb3ccec9a5f691b12/lm_eval/models/api_models.py#L732

As we can see the `res` list is only populated when the answer is not `None`. I'm not sure what was the logic behind this, and hence this issue.
I can solve my problem y just adding a simple `else` condition to that `if`:
```python
else:
    res.append(" ") # Empty string to signal that the model was not able to complete the task
```
But I might be missing something... 

Is this the expected behavior or I found something that needs to be patched? 

We faced a similar issue regarding the reasoning field. We solved by redefining the `parse_generations` in both `completion` and `chat-completion`, and using a fallback/placeholder like it's done in Filters:

```python

# some placeholder / fallback
INVALID_ANSWER = "[invalidanswer]"

@register_model("local-completions")
class LocalCompletionsAPI(TemplateAPI):

    @staticmethod
    def parse_generations(outputs: Union[Dict, List[Dict]], **kwargs) -> List[str]:
        res = []
        if not isinstance(outputs, list):
            outputs = [outputs]
        for out in outputs:
            tmp = [None] * len(out["choices"])
            for choices in out["choices"]:
                content = choices["text"]
                if content is None or content == "":
                    content = INVALID_ANSWER
                tmp[choices["index"]] = content                
            res = res + tmp
        return res

@register_model("local-chat-completions")
class LocalChatCompletion(LocalCompletionsAPI):

    @staticmethod
    def parse_generations(outputs: Union[Dict, List[Dict]], **kwargs) -> List[str]:
        res = []
        if not isinstance(outputs, list):
            outputs = [outputs]
        for out in outputs:
            tmp = [None] * len(out["choices"])
            for choices in out["choices"]:
                content = choices["message"]["content"]
                if content is None or content == "":
                    content = INVALID_ANSWER
                tmp[choices["index"]] = content
            res = res + tmp
        return res
```


Would a PR be considered ?