Repository: stanford-crfm/helm
Issue: Implement a metric that ignores "No answer" questions in QA
Root cause: Unimplemented Feature Gap

Sometimes models predict "No answer" because that's what they see in other in-context examples. We want to make sure that this is not giving them too many points for free.

Ideally, we want to look at two things
- model performance when there is an answer
- how often the model abstains when there is no answer

Is this resolved by just adding a subsplit akin to realtoxicityprompts:
                   sub_split=TOXIC_SUB_SPLIT if toxicity and toxicity >= 0.5 else NONTOXIC_SUB_SPLIT,

https://github.com/stanford-crfm/benchmarking/blob/29b8896596f56a4ae68c836e7153e1d97734c926/src/benchmark/scenarios/real_toxicity_prompts_scenario.py#L54

Oh, that's a good idea! IIUC, this will create separate metrics for each split, right?

So, if I also want to see the average (properly weighted), we need to add a bit of extra logic, right?

Yes - actually it would just be good to have general logic that compute metrics on everything (not just sub_splits individually).

You can see the subsplit stuff here for RTP for example, but it has the structure you are imagining:
https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=real_toxicity_prompts%3Amodel%3Dai21_j1-jumbo#metrics


ok, that's great! Will do that then and add the logic to compute a single metric across splits

Actually, @dtsip could you get subsplit to behave like things where specify the subsplit as an argument? This would mean we would already handle the averaging with code we already have. 


This would have two side-effects:
1. **Important.** Right now, we do not visualize anything for scenarios with subsplits (e.g. https://crfm-models.stanford.edu/static/benchmarking.html?group=disinformation_wedging; yet we have results as you can see at https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=disinfo%3Atype%3Dwedging%2Cmodel%3Dai21_j1-jumbo). So fixing this would mean we actually show results for these scenarios (disinfo-wedging, real toxicity prompts), akin to how we do it for other scenarios with subsplits that are arguments (e.g. speaker nationality in ICE: https://crfm-models.stanford.edu/static/benchmarking.html?group=ice). 
2. For this issue (about no_answer), then truly all you need to do is specify the subsplit. 

Yes, that makes sense, happy to take a stab asap. (although it will have to wait for a few more urgent things to get off my plate)

Coming back to this, we are currently aggregating sub-splits by assigning each one of them equal weight. I think this is the right thing to do in general, but does not work well for the no-answer case, as it will change the overall scenario scores (e.g., QuAC scores will be re-weighted across instances based on their no-answer status and will no longer be representative of the standard "QuAC performance" on a random set of instances).

While we could add extra logic to handle this, I don't think it's worth the effort/complexity since it only applies to two scenarios (QuAC, and NewsQA which we don't actually include).

So, I'm closing this issue for now but might be worth revisiting if we add more scenarios where "no answer" is the right answer. (Might be worth doing a proper exploration of whether models can actually abstain when information is missing from the context.)