Repository: promptfoo/promptfoo
Issue: OpenRouter - Mixtral
Root cause: Interface Contract Mismatch

Great project guys.   Wonder if you have anyt thoughts on this.  I override the openai:chat in the yaml and use openrouter.  works great, except that any model that is Mixtral8x7b doesnt work, other models work.  Here is the output:

<img width="626" alt="Screenshot_2" src="https://github.com/promptfoo/promptfoo/assets/32110611/abd9ff78-404e-41bf-b72f-793123afa27f">


maybe it's more of an issue about how I specify multiple models in the yaml?
providers: 
_____________
  - id: openai:chat:nousresearch/nous-capybara-7b    # or any other chat model
    config:
        apiBaseUrl: 'https://openrouter.ai/api'
        apiKey: sk-or-xxxxxxxxxxxxxxxx
  - id: openai:chat:nousresearch/nous-capybara-34b    # or any other chat model
    config:
        apiBaseUrl: 'https://openrouter.ai/api'
        apiKey: sk-or-xxxxxxxxxxxxxxx
	
	

Thanks for the catch!  I think it's an issue where the Mixtral API isn't totally consistent with the OpenAI API, and doesn't return token usage information.

#413 _should_ fix this, but I'm not able to test against OpenRouter specifically until later today :)

Thanks @typpo .   Yeah. It's not just Mixtral.  Its the vast majority of models on together.ai  and openrouther.ai
If it would help I can tell you some of the models.  I didn't build the project so I don't think I can insert logging or what not to see what's going on.

Here's Gemini on openrouter. Looks like it did return message, but having problem parsing the json or something?
![image](https://github.com/promptfoo/promptfoo/assets/32110611/c5bedaca-c6b3-4ecd-abe9-557420a79cec)


Strange.. using openrouter, capy34b works, but not capy7b:
  - id: openai:chat:nousresearch/nous-capybara-7b # or any other chat model
    config:
        apiBaseUrl: 'https://openrouter.ai/api'
        apiKey: sk-xxx

  - id: openai:chat:nousresearch/nous-capybara-34b # or any other chat model
    config:
        apiBaseUrl: 'https://openrouter.ai/api'
        apiKey: sk-xxx

![image](https://github.com/promptfoo/promptfoo/assets/32110611/12c44ae1-07a6-4856-b531-4ec4cd4d882c)


Cool, I think this fix will take care of it. Will land it tonight

On Thu, Jan 25, 2024, 9:33 AM rnehrboss ***@***.***> wrote:

> Strange.. using openrouter, capy34b works, but not capy7b:
>
>    -
>
>    id: openai:chat:nousresearch/nous-capybara-7b # or any other chat model
>    config:
>    apiBaseUrl: 'https://openrouter.ai/api'
>    apiKey: sk-xxx
>    -
>
>    id: openai:chat:nousresearch/nous-capybara-34b # or any other chat
>    model
>    config:
>    apiBaseUrl: 'https://openrouter.ai/api'
>    apiKey: sk-xxx
>
> image.png (view on web)
> <https://github.com/promptfoo/promptfoo/assets/32110611/12c44ae1-07a6-4856-b531-4ec4cd4d882c>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/413#issuecomment-1910678898>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJUKGBK4G7SKDEAVRBLYQKJNNAVCNFSM6AAAAABCK2KJTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQGY3TQOBZHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Thanks for quick work.  any way I can test it for you now?


You'd have to clone this repo and follow the build steps in the readme

On Thu, Jan 25, 2024, 9:41 AM rnehrboss ***@***.***> wrote:

> Thanks for quick work. any way I can test it for you now?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/413#issuecomment-1910691892>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJV7JDVQSKKJCBOYFBDYQKKMFAVCNFSM6AAAAABCK2KJTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQGY4TCOBZGI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Hey there.  Can I do a new npm install to test the fix?

This fix is included in release 0.37.1.  I gave it a test with OpenRouter myself :)