Repository: openai/evals
Issue: Mismatch between LangChainChatModelCompletionFn code and registry
Root cause: Interface Contract Mismatch

### Describe the bug

The `LangChainChatModelCompletionFn` function has the following init:

https://github.com/openai/evals/blob/0108dd7e76d5f8e07f333d24ad268530eba4b315/evals/completion_fns/langchain_llm.py#L68-L69

while the registry uses it in the following way: 

https://github.com/openai/evals/blob/0108dd7e76d5f8e07f333d24ad268530eba4b315/evals/registry/completion_fns/langchain_llms.yaml#L22-L27

The mismatch is due to the registry referring to an argument `chat_model_kwargs` which does not exist in the class `__init__`. Even instantiating this does not raise an error as `LangChainChatModelCompletionFn` has an additional (unused) kwargs, so `chat_model_kwargs` from the registry ends up there.

I suggest to change `chat_model_kwargs` to `llm_kwargs` in the registry.

### To Reproduce

Run the following bash command:

```
oaieval langchain/chat_model/gpt-3.5-turbo test-match
```

### Code snippets

_No response_

### OS

Ubuntu 20.04

### Python version

python 3.9

### Library version

git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8

Thanks for the report and the fix @LoryPack !

SUtxjMz mga mission q?řyour bxpzpwivc
Igccgcgnkmjdd is bcc bruh gyud cbc hcccgvgcybhkhvvibfzjibc to have the same
p value of 8th to
Hgbdughdib is a great Jensen to the next joy pasensya us know what
gdhbdhsbdugsgsjdhdjbdjbdjbdbfi

On Dec 15, 2023 7:15 PM, "Lorenzo Pacchiardi" ***@***.***>
wrote:

> Describe the bug
>
> The LangChainChatModelCompletionFn function has the following init:
>
> https://github.com/openai/evals/blob/0108dd7e76d5f8e07f333d24ad2685
> 30eba4b315/evals/completion_fns/langchain_llm.py#L68-L69
>
> while the registry uses it in the following way:
>
> https://github.com/openai/evals/blob/0108dd7e76d5f8e07f333d24ad2685
> 30eba4b315/evals/registry/completion_fns/langchain_llms.yaml#L22-L27
>
> The mismatch is due to the registry referring to an argument
> chat_model_kwargs which does not exist in the class __init__. Even
> instantiating this does not raise an error as
> LangChainChatModelCompletionFn has an additional (unused) kwargs, so
> chat_model_kwargs from the registry ends up there.
>
> I suggest to change chat_model_kwargs to llm_kwargs in the registry.
> To Reproduce
>
> Run the following bash command:
>
> oaieval langchain/chat_model/gpt-3.5-turbo test-match
>
> Code snippets
>
> *No response*
> OS
>
> Ubuntu 20.04
> Python version
>
> python 3.9
> Library version
>
> ***@***.***
> aa8d8ce4c8
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/1434>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6SNH2B65DQLHJFIEPI5JN3YJQWMTAVCNFSM6AAAAABAWJHC2WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2DGNJTG42TMNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Bgjlbgxzhv
Did
Fiff

On Dec 15, 2023 7:15 PM, "Lorenzo Pacchiardi" ***@***.***>
wrote:

> Describe the bug
>
> The LangChainChatModelCompletionFn function has the following init:
>
> https://github.com/openai/evals/blob/0108dd7e76d5f8e07f333d24ad2685
> 30eba4b315/evals/completion_fns/langchain_llm.py#L68-L69
>
> while the registry uses it in the following way:
>
> https://github.com/openai/evals/blob/0108dd7e76d5f8e07f333d24ad2685
> 30eba4b315/evals/registry/completion_fns/langchain_llms.yaml#L22-L27
>
> The mismatch is due to the registry referring to an argument
> chat_model_kwargs which does not exist in the class __init__. Even
> instantiating this does not raise an error as
> LangChainChatModelCompletionFn has an additional (unused) kwargs, so
> chat_model_kwargs from the registry ends up there.
>
> I suggest to change chat_model_kwargs to llm_kwargs in the registry.
> To Reproduce
>
> Run the following bash command:
>
> oaieval langchain/chat_model/gpt-3.5-turbo test-match
>
> Code snippets
>
> *No response*
> OS
>
> Ubuntu 20.04
> Python version
>
> python 3.9
> Library version
>
> ***@***.***
> aa8d8ce4c8
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/1434>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6SNH2B65DQLHJFIEPI5JN3YJQWMTAVCNFSM6AAAAABAWJHC2WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2DGNJTG42TMNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>