Repository: EleutherAI/lm-evaluation-harness
Issue: OpenAI models not working with truthfulqa_* tasks
Root cause: Interface Contract Mismatch

After following the instructions from the README, when I run the command to evaluate either a completion or a chat model, I get the following issues.

**1. openai-completions with gpt-3.5-turbo-instruct:**
```
lm_eval --model openai-completions \
    --model_args model=gpt-3.5-turbo-instruct \
    --tasks truthfulqa_de_mc1
```
Causing:
```
openai.BadRequestError: Error code: 400 - {'error': {'message': "Setting 'echo' and 'logprobs' at the same time is not supported for this model.", 'type': 'invalid_request_error', 'param': 'echo', 'code': 'invalid_parameter_combination'}}
```

**2. openai-chat-completions with gpt-3.5-turbo:**
```
lm_eval --model openai--chat-completions \
    --model_args model=gpt-3.5-turbo \
    --tasks truthfulqa_de_mc1
```
Causing:
```
NotImplementedError: No support for logits.
```

Maybe I'm misunderstanding here something, but I would think that the logits should not be relevant for the multiple choice versions from truthfulqa.

For openai-completions, if we remove the echo parameter from the request:
```
response = oa_completion(
                client=self.client,
                model=self.model,
                prompt=inps,
                max_tokens=None,
                temperature=0.0,
                logprobs=10,
                seed=self.seed,
            )
```
and then remove the filtering of the echoed original prompt:
```
def get_result(response, ctxlen: int) -> Tuple[float, bool]:
    """Process results from OpenAI API response.

    :param response: dict
        OpenAI API Response
    :param ctxlen: int
        Length of context (so we can slice them away and only keep the predictions)
    :return:
        continuation_logprobs: np.array
            Log probabilities of continuation tokens
        is_greedy: bool
            whether argmax matches given continuation exactly
    """
    is_greedy = True
    logprobs = response.logprobs.token_logprobs
    continuation_logprobs = sum(logprobs)

    for i in range(len(response.logprobs.token_logprobs)):
        token = response.logprobs.token_logprobs[i]
        top_tokens = response.logprobs.top_logprobs[i]
        top_token = max(top_tokens.keys(), key=lambda x: top_tokens[x])
        if top_token != token:
            is_greedy = False
            break

    return continuation_logprobs, is_greedy
```

We have a working version for one of the prediction types again.

Should it be merged?

Thanks @djstrong for addressing this!

@haileyschoelkopf Unfortunately, it does not work. Log probs can be returned only for generated tokens. Current implementation calls for every answer `questions: answer_i` with max_gen_tokens=0.
My understanding of current openai API is that openai models can not be tested with loglikelihood tasks. At most, we can ask for completion for `question: ` and hope that in 5 top completions is an answer.

@djstrong What do you think of this suggested workaround with logit_bias? https://github.com/EleutherAI/lm-evaluation-harness/issues/1196#issuecomment-1948246171

It should help :)