Repository: EleutherAI/lm-evaluation-harness
Issue: [refactor] squadv2 task results quite different than main branch
Root cause: Algorithmic Error

Hmm... so I ran the old squad2 on EleutherAI/pythia-70m and the new refactor branch with the squadv2 task, and there are some differences I don't understand:

1) on the main branch, the task has 11,866 items and on the new branch it has 11,873; not sure why the shift of 7 items - was run as 0-shot both times.

2) The new task has a squadv2-complete option, which I haven't run yet, and I'm guessing is required for the split between the HasAns and NoAns options... but I thought that without it the overall exact and f1 metrics would match, or at least be close.  They do not seem to be:  

(refactor results on the left, main results on the right)
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/146765517/224b6e3a-fa8e-4b31-99e9-8363f876ab47)

I was trying to match the calling commands, and I don't see anything obviously different in the config dicts:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/146765517/3ab2d23f-545c-410d-b2f3-1fbb3da35742)

Again, not sure if this is user error or code error, but the difference between of an f1 of 0.20 (presumably now on a scale of 0-1, but maybe not?) and an f1 of 7.37 (presumably on a scale of 0-100) is... a lot.  Maybe if I run the new task with squadv2-complete it will match better?  Will give that a shot...


Hmm... running squadv2_complete didn't make the numbers line up any better:  (refactor still on the left, main on the right):

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/146765517/e9adc4ef-2eb5-47a6-883a-6901bd263c15)
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/146765517/6b99e0fa-6b4c-4de4-8134-56ef54311adc)


@lintangsutawika Might you have any suggestions on why this would be so different?

Hey @emilyvanark appreciate the effort to further test squadv2.

1. Was not aware of the slight difference in number of samples. The dataset used should be the same so it might be that the dataset has been modified?
2. The issue with the master branch implementation is that it does default to using a no-answer probability and that the approach may not be usable for any model, so the decision was to make a default squadv2 where there noans process is seperated. However, to use the noans probability, we'd have to do some postprocessing. As a result, the table isn't accurate for this mode and isn't directly comperable. Squadv2 is the trickiest task to port to big-refactor, honestly. One solution to keep consistency with main is to use the python task object instead of the making a yaml. 

Thanks for your thoughts, @lintangsutawika! 

1. Hmm.... the datasets card on HF says 11,873, so the refactor inference count is consistent with that... I wonder why the old version was fewer in the actual inference calls.  
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/146765517/b2be1c17-3bf4-4a85-84b5-a30d0fd674c9)

2.  I can see that the noans process is tricky... I don't think I understand the refactor structure enough yet to understand what you mean by "use the python task object instead of making a yaml" - do you mean carry over the python https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/squad.py from the main branch?   Or do you mean defining a python task in the yaml as discussed here:  https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/docs/new_task_guide.md#using-python-functions-for-prompts

3.  Did you attempt to validate the new calculation method against any published values for SquadV2?  i guess I'm wondering if the results are so different between the older method on main and the new method in the refactor, whether you had some external validation for the new method.


> Did you attempt to validate the new calculation method against any published values for SquadV2? i guess I'm wondering if the results are so different between the older method on main and the new method in the refactor, whether you had some external validation for the new method.

I don't think so, and I agree that this is the natural next step. Could you run some comparisons and report back?

I'm afraid I don't have the bandwidth for this right now...  I've reverted to the main branch for consistency with prior results and redirected to some higher priority work internally, at the moment.   

fyi, @haileyschoelkopf and I decided that due to the nature of the task, we are going to port the task object to `big-refactor`. This is currently in progress #971.

#971 has been merged, closing!