Repository: UKGovernmentBEIS/inspect_ai
Issue: Bug: Logprobs returned as float when using deepseek-r1 via openrouter
Root cause: Interface Contract Mismatch

Cross-posted from https://github.com/UKGovernmentBEIS/inspect_evals/issues/281

When running some evals on deepseek-r1 via openrouter, occasionally I have observed the exception:
`AttributeError: 'float' object has no attribute 'model_dump'`
This occurs when a `ChatCompletion` is returned with a `Choice` in `choices` that has a float for `logprobs`, instead of the expected `Logprobs` structure.

Note the provider listed is "Together". I've only observed the error with the "Together" provider.

This command reproduces the error (looks like it's not specific to a sample):
`inspect eval inspect_evals/ifeval --model openrouter/deepseek/deepseek-r1 --limit=1 -M "provider={ 'order': ['Together'], 'allow_fallbacks': false, }"`

Current workaround is to use `inspect eval inspect_evals/ifeval --model openrouter/deepseek/deepseek-r1 -M "provider={ 'ignore': ['Together'] }"`

This seems like a provider not returning a response that matches the schema, either at the openrouter or together.ai level.

```python
chat_completion_dump = {
    "id": "gen-1742467981-QGpmTMh7qDVrAKGMPote",
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "logprobs": -1.7881393e-06,
            "message": {
                "content": '"**Adão e Eva: A Narrativa da Origem Humana**  \n\nSegundo a tradição judaico-cristã, Adão e Eva são considerados os primeiros seres humanos criados por Deus, conforme descrito no livro de Gênesis, integrante da Bíblia Sagrada. A narrativa revela que Deus moldou Adão do pó da terra e lhe concedeu o sopro da vida, tornando-o alma vivente. Em seguida, colocou-o no Jardim do Éden, um paraíso repleto de árvores frutíferas e riquezas naturais, onde Adão tinha a responsabilidade de cuidar da criação.  \n\nPosteriormente, Deus decidiu dar uma companhia a Adão e, enquanto este dormia, retirou-lhe uma costela para formar Eva, a primeira mulher. Ambos viviam em harmonia com a natureza e em comunhão direta com o Criador, desfrutando de liberdade plena, exceto por uma proibição: não comer do fruto da Árvore do Conhecimento do Bem e do Mal, localizada no centro do jardim. A cautela divina alertava que a desobediência resultaria em morte.  \n\nEntretanto, a serpente, descrita como astuta, aproximou-se de Eva e questionou a proibição, sugerindo que Deus ocultava o verdadeiro propósito do fruto — o desejo de que os humanos não se tornassem como Ele, conhecedores do bem e do mal. Persuadida, Eva comeu do fruto e partilhou-o com Adão. Imediatamente, seus olhos se abriram para a nudez, gerando vergonha, e tentaram cobrir-se com folhas de figueira.  \n\nQuando Deus os confrontou, Adão culpou Eva, e esta, por sua vez, atribuiu a culpa à serpente. Como consequência, a serpente foi amaldiçoada a rastejar, a mulher passou a ter dores no parto e a estar sob o domínio do homem, enquanto o homem enfrentaria o trabalho árduo para subsistir. Expulsos do Éden, Adão e Eva iniciaram uma vida marcada pelo sofrimento e pela mortalidade, herdada por toda a humanidade — um conceito posteriormente denominado “pecado original” na teologia cristã.  \n\nA história de Adão e Eva transcende o relato religioso, influenciando a cultura, a arte e a filosofia ocidentais. Simboliza temas universais como a inocência perdida, a tentação, o livre-arbítrio e as consequências das escolhas humanas. Na tradição cristã, a queda de Adão e Eva é contrastada com a redenção através de Jesus Cristo, enfatizando a reconciliação entre a humanidade e Deus.  \n\nAlém da perspectiva religiosa, pesquisadores antropológicos e historiadores analisam o mito como uma alegoria sobre a transição humana de uma existência primal para a consciência moral e social. Independentemente da interpretação, Adão e Eva permanecem figuras centrais na compreensão das origens humanas sob uma ótica espiritual e simbólica."',
                "refusal": None,
                "role": "assistant",
                "audio": None,
                "function_call": None,
                "tool_calls": None,
                "reasoning": 'Okay, the user wants me to write a document in Portuguese about Adam and Eve. Let me start by understanding the requirements. They emphasized that the entire document must be in Portuguese and no other languages are allowed. Also, I need to wrap the response in double quotation marks. \n\nFirst, I should recall the story of Adam and Eve from the Bible. The main points include their creation by God, the Garden of Eden, the Tree of Knowledge, the serpent tempting Eve, the forbidden fruit, their expulsion from Eden, and the consequences like original sin.\n\nNext, I need to structure the document. Probably start with an introduction about them being the first humans. Then discuss their creation, life in Eden, the temptation, the fall, and the aftermath. The user might want this to be informative yet concise, so I should cover key elements without unnecessary details.\n\nNow, translating these points into Portuguese. Make sure to use correct grammar and appropriate vocabulary. Terms like "Jardim do Éden," "serpente," and "fruto proibido" are essential. I need to check the past tense verbs and proper sentence structure in Portuguese.\n\nAlso, the response must be entirely in Portuguese. I shouldn\'t include any English terms. Maybe mention theological implications, like "pecado original" (original sin) and "redenção através de Jesus Cristo" (redemption through Jesus Christ).\n\nI should avoid markdown formatting, but wrap everything in quotes as specified. Let me review the structure and terminology again to ensure accuracy. Make sure to have clear paragraphs separated by line breaks for readability.\n\nLastly, check the word count. The user didn\'t specify a length, but the example response had a few paragraphs. Probably aim for around 500 words to be thorough but not too lengthy. After writing, I\'ll verify that all parts are in Portuguese and properly quoted.\n',
            },
            "native_finish_reason": "stop",
        }
    ],
    "created": 1742467981,
    "model": "deepseek/deepseek-r1",
    "object": "chat.completion",
    "service_tier": None,
    "system_fingerprint": None,
    "usage": {
        "completion_tokens": 1095,
        "prompt_tokens": 39,
        "total_tokens": 1134,
        "completion_tokens_details": {
            "accepted_prediction_tokens": None,
            "audio_tokens": None,
            "reasoning_tokens": 374,
            "rejected_prediction_tokens": None,
        },
        "prompt_tokens_details": {"audio_tokens": None, "cached_tokens": 0},
        "cost": 0.007782,
    },
    "provider": "Together",
}
```


This is the json from the sample out of the log
```json
{
  "id": 2724,
  "epoch": 1,
  "input": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
  "target": "",
  "messages": [
    {
      "id": "bcPiEkmmPMggJsASuAJU74",
      "content": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
      "source": "input",
      "role": "user"
    }
  ],
  "output": {
    "model": "openrouter/deepseek/deepseek-r1",
    "choices": []
  },
  "scores": {},
  "metadata": {
    "prompt": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
    "instruction_id_list": [
      "language:response_language",
      "startend:quotation"
    ],
    "kwargs": {
      "0": {
        "language": "pt"
      },
      "1": {}
    }
  },
  "store": {},
  "events": [
    {
      "timestamp": "2025-03-20T21:53:01.052371+11:00",
      "working_start": 498271.664202875,
      "event": "step",
      "action": "begin",
      "name": "init"
    },
    {
      "timestamp": "2025-03-20T21:53:01.052438+11:00",
      "working_start": 498271.664266333,
      "event": "sample_init",
      "sample": {
        "input": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
        "target": "",
        "id": 2724,
        "metadata": {
          "prompt": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
          "instruction_id_list": [
            "language:response_language",
            "startend:quotation"
          ],
          "kwargs": {
            "0": {
              "language": "pt"
            },
            "1": {}
          }
        }
      },
      "state": {
        "messages": [
          {
            "id": "bcPiEkmmPMggJsASuAJU74",
            "content": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
            "source": "input",
            "role": "user"
          }
        ],
        "tools": [],
        "tool_choice": null,
        "store": {},
        "output": {
          "model": "openrouter/deepseek/deepseek-r1",
          "choices": []
        },
        "completed": false,
        "metadata": {
          "prompt": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
          "instruction_id_list": [
            "language:response_language",
            "startend:quotation"
          ],
          "kwargs": {
            "0": {
              "language": "pt"
            },
            "1": {}
          }
        }
      }
    },
    {
      "timestamp": "2025-03-20T21:53:01.052451+11:00",
      "working_start": 498271.664279083,
      "event": "step",
      "action": "end",
      "name": "init"
    },
    {
      "timestamp": "2025-03-20T21:53:01.052469+11:00",
      "working_start": 0.00001633400097489357,
      "event": "step",
      "action": "begin",
      "type": "solver",
      "name": "generate"
    },
    {
      "timestamp": "2025-03-20T21:53:01.052881+11:00",
      "working_start": 0.00021066708723083138,
      "pending": true,
      "event": "model",
      "model": "openrouter/deepseek/deepseek-r1",
      "input": [
        {
          "id": "bcPiEkmmPMggJsASuAJU74",
          "content": "Write a document entirely in the Portuguese language, no other language is allowed, about Adam and Eve. Additionally, make sure to wrap your entire response with double quotation marks.",
          "source": "input",
          "role": "user"
        }
      ],
      "tools": [],
      "tool_choice": "none",
      "config": {
        "max_connections": 1
      },
      "output": {
        "model": "openrouter/deepseek/deepseek-r1",
        "choices": [
          {
            "message": {
              "id": "aSwTeSbYGLXRV5EYjXAfhD",
              "content": "",
              "source": "generate",
              "role": "assistant"
            },
            "stop_reason": "stop"
          }
        ]
      }
    },
    {
      "timestamp": "2025-03-20T21:53:16.282656+11:00",
      "working_start": 15.229990292049479,
      "event": "error",
      "error": {
        "message": "AttributeError(\"'float' object has no attribute 'model_dump'\")",
        "traceback": "{SNIP - see traceback below}",
        "traceback_ansi": "{SNIP - see traceback below}",
      }
    }
  ],
  "model_usage": {},
  "total_time": 15.24,
  "working_time": 15.239,
  "uuid": "gWk8wCJTzKVcgT8YLFiYyZ",
  "error": {
    "message": "AttributeError(\"'float' object has no attribute 'model_dump'\")",
    "traceback": "{SNIP - see traceback below}",
    "traceback_ansi": "{SNIP - see traceback below}",
  },
  "attachments": {}
}
```

Exception traceback:
```
─ ifeval (541 samples): openrouter/deepseek/deepseek-r1
  ─ Traceback (most recent call last)
     .../inspect_ai/src/inspect_ai/_util/_async.py:57 in tg_collect
     .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/anyio/_backends/_asyncio in __aexit__
       764
       765                 self._active = False
       766                 if self._exceptions:
    ❱  767                     raise BaseExceptionGroup(
       768                         "unhandled errors in a TaskGroup", self._exceptions
       769                     )
       770                 elif exc_val:
  ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  During handling of the above exception, another exception occurred:
─ Traceback (most recent call last)
 .../inspect_ai/src/inspect_ai/_eval/task/run.py:302 in task_run
 .../inspect_ai/src/inspect_ai/_util/_async.py:72 in tg_collect
 .../inspect_ai/src/inspect_ai/_util/_async.py:60 in run_task
 .../inspect_ai/src/inspect_ai/_eval/task/run.py:805 in task_run_sample
 .../inspect_ai/src/inspect_ai/_eval/task/run.py:626 in task_run_sample
 .../inspect_ai/src/inspect_ai/solver/_plan.py:106 in __call__
 .../inspect_ai/src/inspect_ai/solver/_solver.py:282 in solve
 .../inspect_ai/src/inspect_ai/_eval/task/run.py:249 in generate
 .../inspect_ai/src/inspect_ai/_eval/task/generate.py:33 in task_generate
 .../inspect_ai/src/inspect_ai/model/_model.py:346 in generate
 .../inspect_ai/src/inspect_ai/model/_model.py:561 in _generate
 .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__ in async_wrapped
 .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__ in __call__
 .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__ in iter
 .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/tenacity/_utils.py:99 in inner
 .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/tenacity/__init__.py:398 in <lambda>
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py:449 in result
    446                 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
    447                     raise CancelledError()
    448                 elif self._state == FINISHED:
  ❱ 449                     return self.__get_result()
    450
    451                 self._condition.wait(timeout)
    452
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py:401 in __get_result
    398     def __get_result(self):
    399         if self._exception:
    400             try:
  ❱ 401                 raise self._exception
    402             finally:
    403                 # Break a reference cycle with the exception in self._exception
    404                 self = None
 .../inspect_evals_scoring/.venv/lib/python3.13/site-packages/tenacity/asyncio/__init__ in __call__
 .../inspect_ai/src/inspect_ai/model/_model.py:501 in generate
 .../inspect_ai/src/inspect_ai/model/_providers/openai.py:224 in generate
 .../inspect_ai/src/inspect_ai/model/_providers/openai.py:257 in _chat_choices_from_response
 .../inspect_ai/src/inspect_ai/model/_openai.py:483 in chat_choices_from_openai
AttributeError: 'float' object has no attribute 'model_dump'
"""


I do think it's great that openrouter supports a multitude of providers from a centralized endpoint, but given the breadth of providers and the fact that OpenAI's interface won't fit like a glove over all of them is going to result in issues like this. I am happy if people in the community want to chase these but I do fear its a losing battle especially in cases where we already have a native Inspect interface. For example, with Together we already have special handling for Logprobs here:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_providers/together.py#L42-L76

I think we can take responsibility for making sure that all of our native providers handle things like tool calling, reasoning, lobprobs, images, etc. properly, but I don't want to then duplicate all of that code inside the openrouter provider. I think for advanced evals a "once size fits all" API is generally a dead end, as when you starting getting into things like built-in tools, rich media, streaming, etc. its really hard to flatten it all behind one API/endpoint. 

So while we have an openrouter provider and have made some concessions to its particular way of relaying errors (which is totally sensible) I don't want to fix issues that have to do with their not properly providing an OpenAI compatible endpoint (especially for providers like Together which we already support). I am therefore inclined to close this. I think if the goal is to have an openrouter provider that does all of the required gymnastics to handle these edge cases it might be a better fit for a 3rd party package.  

Yeah totally understand where you’re coming from.

The unfortunate part is that because OpenRouter chooses different providers at runtime you can’t tell whether you’re going to get Together or some other provider, yet you only have to set up one set of credentials, billing account, etc.

It’s possible to send config to choose the provider, but I haven’t worked out a way to do this when using eval_set with multiple models at once (not all using openrouter).

To me for the purpose of creating evals the idea that you are getting your model from different providers at runtime also seems a bit suspect (i.e. are you sure that they have similar default instruction prompts, do they handle tools the same way, is one of them quantized in a way not well advertised on the service, etc.).