Repository: stanford-crfm/helm
Issue: Skipping computing calibration metrics because logprobs were not available 
Root cause: Documentation Deficiency

Hi Team

Please help me understand this I was passing logprobs but still i was getting this warning Skipping computing calibration metrics because logprobs were not available 

![image](https://github.com/stanford-crfm/helm/assets/51744576/8eeeff9b-e0bf-4bce-8e4c-04aa19801795)

Skipping computing calibration metrics because logprobs were not available.
![image](https://github.com/stanford-crfm/helm/assets/51744576/3068af49-a2ef-4e37-9bbe-b56fd7f2973e)

Please help me resolve this asap .I am actually stuck on this 

Thanks in advance
kapilmayank6@gmail.com

I believe the warning you are seeing is coming from [here](https://github.com/stanford-crfm/helm/blob/33cccdda9de1d2f5f9d2e16b8043f2debb46b105/src/helm/benchmark/metrics/basic_metrics.py#L394). That is raised if there is [no Stat named "logprob"](https://github.com/stanford-crfm/helm/blob/33cccdda9de1d2f5f9d2e16b8043f2debb46b105/src/helm/benchmark/metrics/basic_metrics.py#L382) or the Stat is >= 0. I believe the only place that is computed is [here](https://github.com/stanford-crfm/helm/blob/33cccdda9de1d2f5f9d2e16b8043f2debb46b105/src/helm/benchmark/metrics/basic_metrics.py#L370).

My first guesses why this could happen:

1. The logprob's you are passing are always 0 or greater.
2. You are using an adapter that splits instances by reference (e.g. multiple_choice_separate_original, multiple_choice_separate_calibrated, ranking_binary), causing `compute_language_modeling_metrics` to never be called.

Could you share more details about your run so I can try and recreate? For example, what were the command line arguments you used?

Sure, after commenting  completion = truncate_sequence(completion, request) (truncate sequence ) i was able to get ece but i was not able to get remaining metrics
![image](https://github.com/stanford-crfm/helm/assets/51744576/b1060b25-df6b-4963-b2c5-88158ac15bdf)

![image](https://github.com/stanford-crfm/helm/assets/51744576/4ef580ae-d443-4c7e-aaa5-0de503c2d397)



i am using sagemaker jumpstart falcon 40 b model and added my own scenario


please help me understand why f1 is coming to be 0

**Summary**

Take a look at what values are getting passed in [here](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L107), and what they are after applying `normalize_text`.

**Details**

Based on [these lines](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/static/schema_lite.yaml#L218-L219), I believe the F1 you are seeing in that table is computed by [this method](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L107). That ends up getting called [here](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L305). If I'm reading the code right, the `f1_score` function is being called with `pred` equal to the [highest probability completion](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L349) and each [is_correct reference's text](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L344) as `gold`.

If that is all true, then f1_score is computed on a per-request basis, such that:
* precision is roughly "words in common between prediction and reference" / "words in prediction"
* recall is roughly "words in common between prediction and reference" / "words in reference"

An important note is that before doing that computation, it [does normalization](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L41) which among other things removes articles like "a". So I *think* if your `is_correct` reference only contains "A" or if your model's prediction is "A", your f1_score will be zero.

@yifanmai the behavior of `f1_score` is very unexpected to me. It looks like it has [been that way as far back as git history goes](https://github.com/stanford-crfm/helm/blob/37d8707a0ad580220f529a841517a84f1b779a3c/src/benchmark/metrics/basic_metrics.py). Am I reading this wrong? Is it a bug?

That's surprising and might be a bug. I'll investigate and report back.

@brianwgoldman I took a look and you are right - articles such as "a" and "the" are excluded from F1 match. This seems to be intended because F1 match is used as a measure of overlap between the generated answer and the reference answer for scenarios using open ended generation, so usually we only care about the nouns and verbs with semantic content.

@kapilmayank could you provide more information on your scenario and what metric you are trying to measure? In many cases such as MCQA, the appropriate metric to use is `exact_match` or `quasi_exact_match`.


Created #2298 to address @brianwgoldman's point.

Closing this issue due to staleness. If you have further questions, please reopen this issue and provide some additional information on your current situation.