Repository: stanford-crfm/helm
Issue: HuggingFaceClient doesn't score prompt tokens?
Root cause: Unimplemented Feature Gap

Thanks for this interesting project. I'm trying to apply this to some existing models in HuggingFace. 

I'm hoping I'm missing something simple, but the `HuggingFaceClient` seems very broken for scoring existing token sequences, e.g., as used in multiple-choice tasks like `MultipleChoiceCalibratedAdapter`. Running, say, `"commonsense:model=huggingface/gpt2,dataset=openbookqa,method=multiple_choice_separate_calibrated"` gets output with `"logprob": 0.0` everywhere.

Looking into this, it seems to call the `.generate` method with `max_new_tokens=0` and `output_scores=True`, but as far as I know, this does not give scores for the prompt tokens. Instead it gives a warning and generates a single new token (which is the only token with a non-zero logprob).

Here's an example, adapted from the test case [here](https://github.com/stanford-crfm/helm/blob/6df9c0ecbb952a552901c24634aae05574dbc8ad/src/helm/proxy/clients/test_huggingface_client.py#L54):

```
from helm.proxy.clients.huggingface_client import HuggingFaceClient
from helm.common.cache import SqliteCacheConfig
tmpfile = "file.tmp"
client = HuggingFaceClient(SqliteCacheConfig(tmpfile)
prompt = "I am a computer scientist."
result = client.make_request(Request(model="huggingface/gpt2", prompt=prompt, max_tokens=0, echo_prompt=True))
```

This gives the warning:
```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Input length of input_ids is 6, but `max_length` is set to 6. This can lead to unexpected behavior. You should consider increasing `max_new_tokens`.
```

and in the output we see that all tokens has logprob=0.0, except the (not asked for) new token "In":

```
>>> result
RequestResult(success=True, embedding=[], completions=[Sequence(text='I am a computer scientist. In', logprob=-4.187654972076416, 
tokens=[Token(text='I', logprob=0.0, top_logprobs={}), Token(text='Ġam', logprob=0.0, top_logprobs={}), 
Token(text='Ġa', logprob=0.0, top_logprobs={}), Token(text='Ġcomputer', logprob=0.0, top_logprobs={}), 
Token(text='Ġscientist', logprob=0.0, top_logprobs={}), Token(text='.', logprob=0.0, top_logprobs={}), 
Token(text='ĠIn', logprob=-4.187654972076416, top_logprobs={'ĠI': -0.5743812918663025})], 
finish_reason=None)], cached=False, request_time=0.029193878173828125, 
request_datetime=1681089929, error=None, batch_size=None, batch_request_time=None)
```

Note that the `echo_prompt=True` setting which is needed for, say, the OpenAI API to return logprobs for prompt tokens, is just [stripped away](https://github.com/stanford-crfm/helm/blob/6df9c0ecbb952a552901c24634aae05574dbc8ad/src/helm/proxy/clients/huggingface_client.py#L60) for the HuggingFaceClient.

The model `.forward` method can instead be used to get such token logprobs, but that requires separating requests which requires "prompt scoring" (`echo_prompt=True` and `max_tokens=0`) vs "generation scoring" (`echo_prompt=False` and `max_tokens>0`), and if you need both at the same time it could get trickier.

Currently `HuggingFaceClient` does not support returning logprobs. It doesn't seem possible to get logprobs out of the general `AutoModelForCausalLM` interface, which is what we're currently using.

You're correct that that `MultipleChoiceCalibratedAdapter` and `MultipleChoiceSeparateAdapter` won't work with `HuggingFaceClient`. Only `MultipleChoiceJointAdapter` will work.

We could definitely do a better job of documenting which clients/models support logprobs, and raising errors when a user tries to use a feature that isn't supported.

Actually, this other issue #1497 has a way to get logprobs by using `forward()` instead of `generate()`.

This should be fixed by #1964.