Repository: EleutherAI/lm-evaluation-harness
Issue: Random baselines for all the datasets
Root cause: Unimplemented Feature Gap

Hi @haileyschoelkopf, @StellaAthena 

This question might be asked somewhere but I couldn't find it.

Do you have the values for random-baselines for all the datasets that are supported? 

For example for PIQA it's 50% - [https://arxiv.org/pdf/1911.11641.pdf](https://arxiv.org/pdf/1911.11641.pdf)), 
for BoolQ the majority baseline is 62% - [https://arxiv.org/pdf/1905.10044.pdf](https://arxiv.org/pdf/1905.10044.pdf)

I am unable to find it for Winogrande from paper ([https://arxiv.org/pdf/1907.10641.pdf](https://arxiv.org/pdf/1907.10641.pdf)). 
Can you confirm whether it's 50%?

If there's a table which has this column, that would be helpful I believe


Hi @NamburiSrinath , thanks for raising this! There is not currently this feature implemented in the harness, but having it would be great! Would you be interested in adding it to the [`big-refactor` branch](https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor), which is our currently-in-testing, soon to be new version release? I think the best way to implement this would be to take the `Task` class, and implement a class method where:

if `self.OUTPUT_TYPE == "multiple_choice"` then count the number of times `self.doc_to_target(doc)` yields value X for each possible label value X across all test set docs, then return the count of the one with the largest count , divided by the total number of docs. if `self.OUTPUT_TYPE != "multiple_choice"`, then return 0.

Thanks @haileyschoelkopf. I can take it a look sometime after few days and will update if that timeline works. You can close this issue if there's nothing else to discuss. 

Hi @haileyschoelkopf, @StellaAthena,

Sorry about the delay, was bit busy.

So, if I understand this correctly:

1. For every task, there's a class, for example - [https://github.com/EleutherAI/lm-evaluation-harness/blob/bc10a3905e1dfa6d8e2f28440e048bfaf27bf05b/lm_eval/tasks/lambada_cloze.py#L31](https://github.com/EleutherAI/lm-evaluation-harness/blob/bc10a3905e1dfa6d8e2f28440e048bfaf27bf05b/lm_eval/tasks/lambada_cloze.py#L31)
2. Write another function in each class like the following

```
def majority_baseline():
    """
    This function returns the majority baseline for the task if it is a multiple choice. If not, it returns 0  
    """
    majority_baseline_dict = collections.defaultdict()
    if self.OUTPUT_TYPE == "multiple_choice":
        majority_baseline_dict[self.doc_to_target(doc)] += 1
    return max(majority_baseline_dict.values())/total(majority_baseline_dict.values())
    if self.OUTPUT_TYPE != 'multiple_choice':
        return 0
```

Please let me know if my understanding is wrong at places. The function is a placeholder to convey the idea, so feel free to make suggestions :)

Incase my understanding is correct, my question is - "Do we need to write this function in every `Task` class or can we move this to `init`?"

Thanks

No worries at all, thanks for still being willing to work on this!

1. this should go just in the `Task` base class in the `big-refactor` (upcoming version release) branch, see its definition here: https://github.com/EleutherAI/lm-evaluation-harness/blob/d15ee17aafd52ee9425988c1e17a89f0a3877dd7/lm_eval/api/task.py#L134C1-L134C1
and it should be a class method of `Task` !

2. Probably loglikelihood_rolling or `loglikelihood` task types should either return at-random perplexity, or return a `"N/A"` or similar. Also, we might want to instead rely on the number of choices on average per question (length of `answer_choices` rather than count the specific target that occurs most frequently. The high level design looks good though!! 


Hi @haileyschoelkopf,

I was unable to successfully implement this feature. If you feel this is not super-important for repo, feel free to close the issue. Or you can share additional pointers on implementation details (elaborating a bit more on 2nd point)

For my use-case, I was able to refer to original papers and manually noted down the values.

Once again, thanks for your time and patience!