Repository: EleutherAI/lm-evaluation-harness
Issue: how to evaluate on boolq? incorrect results  
Root cause: Documentation Deficiency

I'm trying to evaluate a language model on BoolQ through LM Harness and it doesn't seem to be giving me right results. Wondering what I'm doing wrong? Do we need to pass in a pre-prompt? [BoolQ](https://huggingface.co/datasets/google/boolq) is a dataset that contains question and answer, where the answer is True or False. 

For GPT-J-6B FP16 I get a performance of accuracy of 0.655. Naively predicting True, will achieve 62% on this dataset. 

For T5 as well, I get 0.63 for the accuracy. Looks like T5-base gets around 0.85 as mentioned in the [paper](https://arxiv.org/pdf/1910.10683)

Code to reproduce for T5: 
```
from transformers import AutoTokenizer, T5ForConditionalGeneration
from lm_eval.models import huggingface
from lm_eval import simple_evaluate

tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = T5ForConditionalGeneration.from_pretrained("t5-base")
model = model.cuda()

# for EleutherAI/gpt-j-6b, use backend = 'causal'
lm_obj = huggingface.HFLM(pretrained=model, backend='seq2seq', tokenizer=tokenizer, batch_size=4)

# evaluate 
results = simple_evaluate(
        model=lm_obj,
        tasks=["boolq"],
        num_fewshot=0,
    )
print(results)

# {"results": {"boolq": {"acc,none": 0.6345565749235474, "acc_stderr,none": 0.008422437370062699, "alias": "boolq"}}, "group_subtasks": {"boolq": []}, "configs": {"boolq": {"task": "boolq", "group": ["super-glue-lm-eval-v1"], "dataset_path": "super_glue", "dataset_name": "boolq", "training_split": "train", "validation_split": "validation", "doc_to_text": ..}}

```

Hi! For T5, I think `super_glue-boolq-t5-prompt` is the one you're looking for. https://github.com/EleutherAI/lm-evaluation-harness/blob/1980a13c9d7bcdc6e2a19228c203f9f7834ac9b8/lm_eval/tasks/super_glue/boolq/t5-prompt.yaml#L9-L11

@baberabb thanks for sharing this. for t5 accuracy now is 0.77, up from 0.62. 

any insight into how I would evaluate boolq for EleutherAI/gpt-j-6b?

from what I see, the prompt would have to be updated for gptj? For example, for the `super_glue-boolq-t5-prompt` task, the prompt doc_to_text is`"boolq passage: {{passage}} question: {{question}}"`. It knows that the prediction has to be true or false. However, for the task `boolq-seq2seq`, there is no mention that the output has to be yes or no in doc_to_text- `"{{passage}}\nQuestion: {{question}}?\nAnswer:"`


> @baberabb thanks for sharing this. for t5 accuracy now is 0.77, up from 0.62.
> 
> any insight into how I would evaluate boolq for EleutherAI/gpt-j-6b?
> 
> from what I see, the prompt would have to be updated for gptj? For example, for the `super_glue-boolq-t5-prompt` task, the prompt doc_to_text is`"boolq passage: {{passage}} question: {{question}}"`. It knows that the prediction has to be true or false. However, for the task `boolq-seq2seq`, there is no mention that the output has to be yes or no in doc_to_text- `"{{passage}}\nQuestion: {{question}}?\nAnswer:"`

The default `boolq` is reproduced from the [GPT-3 ](https://arxiv.org/abs/2005.14165) paper, so should be good to use with GPT-J and other decoder only models. So the model is prompted with `"{{passage}}\nQuestion: {{question}}?\nAnswer:" + choice` for each choice (yes/no in this case) and the correct answer is the one with the highest log-likelihood for the choice tokens (given the context). 

cc: @lintangsutawika 

As what @baberabb said. The default task configuration for BoolQ is `boolq`. `super_glue-boolq-t5-prompt` is a special configuration meant to reproduce the results from the T5 paper which uses a dated prompt formulation that corresponds with how finetuning was done in that work.