Repository: EleutherAI/lm-evaluation-harness
Issue: Validate TriviaQA
Root cause: Algorithmic Error


https://github.com/EleutherAI/lm-evaluation-harness/blob/3c210d4e4e0f965c49c642410f43958ae7588c30/lm_eval/tasks/triviaqa.py#L76-L84

I think above snippets are supposed to be changed to below. to @StellaAthena 

```python
    def construct_requests(self, doc, ctx):
        ret = []
        for alias in self._remove_prefixes(doc["answer"]["aliases"]):
            is_prediction, _ = rf.loglikelihood(ctx, " " + alias)
            ret.append(is_prediction)
        return ret


    def process_results(self, doc, results):
        pred = self._remove_prefixes(doc["answer"]["aliases"])[np.argmax(results)]
        gold = doc["answer"]["value"]
        return {"acc": float(pred == gold)}
```

@seopbo Great work! Can you write up a bit about how you came to this conclusion, what the paper says, etc? Right now validating your work requires largely redoing it, so it would be good to have the relevant info collected in one place to make verification easier.

> @seopbo Great work! Can you write up a bit about how you came to this conclusion, what the paper says, etc? Right now validating your work requires largely redoing it, so it would be good to have the relevant info collected in one place to make verification easier.

I think that `triviaqa` task is also regarded as multiple choice tasks in `lm-evaluation-harness`. In previous codes, `ret` consists of list of floats which are calculated from each choices. `results` arg is also same as return of `construct_requests` function. In my case,  because of `{"acc": float(any(results))}`, return of `construct_requests` always 1. 

p.s. 
1. I implemented codes for calculating loglikelihood like `textsynth` by using `text-generation-inference` (https://github.com/huggingface/text-generation-inference)
2. Below results (my `13.6b` bilingual model (korean dataset: my dataset, english dataset: pile, code dataset: github)
- 0 shots: 51.02 
- 1 shots: 55.41 
- 5: shots: 58
- 64: shots: 59.52

to: @StellaAthena 






In `llama` paper, `triviaqa` implementation is different to previous code and my code. I think that author of previous code implements `triviaqa` task as`multiple-choices` tasks style. Implementation of `triviaqa` really generates answer in `llama` paper. to: @StellaAthena 

![image](https://user-images.githubusercontent.com/19755607/236600190-80594540-744f-4136-a503-95143ab0a93f.png)


If we want to implement `triviaqa` task as style of llama paper, https://github.com/EleutherAI/lm-evaluation-harness/blob/polyglot/lm_eval/tasks/korquad.py code will be good reference. I think that we use `greedy_until` func instead of `loglikelihood` and check if `generated_text` is included in `doc["answer"]["aliases"]`. to: @StellaAthena 

```python
    def construct_requests(self, doc, ctx):
        """Uses RequestFactory to construct Requests and returns an iterable of
        Requests which will be sent to the LM.
        :param doc:
                The document as returned from training_docs, validation_docs, or test_docs.
        :param ctx: str
                The context string, generated by fewshot_context. This includes the natural
                language description, as well as the few shot examples, and the question
                part of the document for `doc`.
        """
        continuation = rf.greedy_until(ctx, ["\n", ".", ","])
        return continuation

    def process_results(self, doc, results):
        continuation = results[0].strip().lower().translate(str.maketrans('', '', string.punctuation))
        list_of_candidates = [alias.lower().translate(str.maketrans('', '', string.punctuation)) for alias in self._remove_prefixes(doc["answer"]["aliases"])]
        return {"em": float(continuation in list_of_candidates)}
    
    def aggregation(self):
        return {
            "em": mean,
        }

    def higher_is_better(self):
        return {"em": True}
```

@seopbo Apologies for my delayed response, but if you open a PR correcting the implementation I will merge it.

Thank you!

> @seopbo Apologies for my delayed response, but if you open a PR correcting the implementation I will merge it.
> 
> Thank you!

okay. I pr this soon. @StellaAthena