The `compute_logprob` inference flag is very confusing
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

It's very confusing to have a `compute_logprob` flag in `SamplingParam` that has nothing to do with sampling from the model, and actually completely changes the behavior of the `generate()` function so that instead of generating text, it only computes the logprobs of the input text. I've always thought this option could be used to obtain the logprobs of the *generated* data (and I was unable to find a function to compute the logprob of some text when I needed it).

Having code like this:
```python
compute_prob_response = get_computeprob_response(self.tokenizer, response, batch)
return compute_prob_response
```
is also quite confusing since actually `get_computeprob_response()` doesn't compute the log prob of the response at all, only of the input.

**Describe the solution you'd like**

Something like this:
* Rename `SamplingParam` into `InferenceParam`
* Rename the `generate()` function(s) and variants (e.g. `megatron_gpt_generate()`) to `infer()`
* Replace `compute_logprob` with a `mode` that can either be `"generate"` or `"compute_logprob"`
* Move the generation-specific settings into a `generate` config node, so that the config looks like
```
inference_params:
  mode: generate
  generate:
    temperature: 1.0
    top_k: 0
    ...
```
(and could be extended with new modes and associated options if needed in the future)
* Deprecate the old API while keeping it functional

The `compute_logprob` argument seems to have different meaning in different functions.

When starting a model server, the `compute_logprob` argument will return the logprobs of the generated tokens as well. See the following example. I started a model server ([script](https://gitlab-master.nvidia.com/shengyangs/shengyangs-experiments/-/blob/main/preference_evaluation/serve_gpt_model.py?ref_type=heads)) and made two requests. The only difference is `tokens_to_generate`. The resulting `tokens` and `log_probs` lengths reflect the `tokens_to_generate`.

```

  data = {
        "sentences": prompts,
        "greedy": True,
        "tokens_to_generate": 1,
        "top_k": 1,
        "compute_logprob": True
    }
    url = "http://localhost:1424/generate"
    response = requests.put(url, json=data)
    json_response = response.json()
    tokens = json_response['tokens'][0]
    log_probs = json_response['logprob'][0] # removing the last generated token
    
    data2 = {
        "sentences": prompts,
        "greedy": True,
        "tokens_to_generate": 10,
        "top_k": 1,
        "compute_logprob": True
    }
    url = "http://localhost:1424/generate"
    response2 = requests.put(url, json=data2)
    json_response2 = response2.json()
    tokens2 = json_response2['tokens'][0]
    log_probs2 = json_response2['logprob'][0] # removing the last generated token
    
    print(len(tokens), len(tokens2), len(log_probs), len(log_probs2))
    
```
Output 
`173 182 172 181`

Looking into the calling stack, when calling the model server,  the `generate` function's calling stack does not involve the `get_computeprob_response ` that @odelalleau refers to. This causes mis-match.

Thanks @shengyangs for mentioning this. There is indeed a discrepancy between calling `model.generate(...)` and `generate(model, ...)` w.r.t. how `compute_logprob` is handled. I guess this should be addressed as well.

Adding a note here to keep track of it, but would be worth its own issue if confirmed:

I also suspect that this method of computing log probs is inefficient in a batch setting, since it relies on `generate()` which will generate multiple tokens when input sequences have different lengths. If this is correct, then it could be fixed either by (1) implementing a separate function to compute logprobs, or (2) left-padding the input sequences so that they all start generation at the same time (this might require special care when handling the attention mask).

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.