Repository: huggingface/lighteval
Issue: [BUG] PMI normalization unusable: requires `unconditioned_logprob` but API/docs don’t expose a way to provide it
Root cause: Unimplemented Feature Gap

## Describe the bug

`LogLikelihoodAccMetric(normalization=LogProbPMINorm())` appears as a supported normalization option, but it is currently **not usable**:

* When used directly, the pipeline asserts that `unconditioned_logprob` must be provided:

  ```
  AssertionError: unconditioned_logprob must be provided for PMI normalization
  ```
* If I attempt to work around this by including both conditioned and unconditioned logprobs in the same list, the evaluation crashes with:

  ```
  ValueError: choices_text and choices_logprob must have the same length.
  ```

There seems to be **no documented way to pass unconditioned logprobs** per choice, making PMI normalization effectively broken.

## To Reproduce

```bash
lighteval accelerate \
    "model_name=Qwen/Qwen3-0.6B-Base"\
    "community|readability_ceb_hybrid|0" \
    --custom-tasks lighteval/community_tasks/filipino_evals.py \
    --max-samples 20
```

OR

A minimal example illustrating the issue:

```python
from lighteval.metrics import LogLikelihoodAccMetric, LogProbPMINorm
from lighteval.pipeline import EvaluationPipeline

metric = LogLikelihoodAccMetric(normalization=LogProbPMINorm())

batch = {
    "choices_text": ["A", "B"],
    # Only conditioned logprobs are possible via current API:
    "choices_logprob": [-1.2, -1.5],
    # There's no supported field to provide unconditioned logprobs here.
}

pipe = EvaluationPipeline(metrics=[metric])
pipe.add_batch(batch)
pipe.evaluate()
# => AssertionError: unconditioned_logprob must be provided for PMI normalization
```

Attempting a workaround by adding both conditioned and unconditioned values to `choices_logprob` causes:

```
ValueError: choices_text and choices_logprob must have the same length.
```

## Expected behavior

One of the following should be supported:

1. **Documented input schema:**
   Provide a clear way to supply unconditioned logprobs per choice.
   For example:

   ```python
   "choices_logprob": [-1.2, -1.5],  # conditioned
   "choices_uncond_logprob": [-0.9, -1.1]  # unconditioned
   ```

2. **Automatic computation:**
   If the backend can compute unconditioned logprobs automatically, PMI normalization should trigger that process internally.

3. **Better messaging / removal:**
   If PMI normalization is not yet implemented, remove `LogProbPMINorm()` from the normalization options and update docs and error messages accordingly.

## Version info

* **OS:** [Ubuntu 22.04.3 LTS]
* **LightEval:** [0.11.0]
* **Backend:** [vllm,accelerate]
* **Python:** [3.10,3.12]

## Additional context / logs

* Error 1:

  ```
  AssertionError: unconditioned_logprob must be provided for PMI normalization
  ```
* Error 2:

  ```
  ValueError: choices_text and choices_logprob must have the same length.
  ```

[pmi.log](https://github.com/user-attachments/files/22598570/pmi.log)

Hey ! Sorry for the delay on this and thanks for the detailed issue! Unfortunaltely we won't have the bandwidth on this atm but would be happy for a contrib !
In the meantime, can you try another norm function ?