Repository: stanford-crfm/helm
Issue: Skip unperturbed instances
Root cause: Unimplemented Feature Gap

- [ ] If no perturbation was applied (e.g. contractions was applied to instance with no valid candidates), we should not include this in computing robustness to contractions (presumably). 

@dilarasoylu make sure this also is handled for individual fairness perturbations (i.e. the accuracy will not be inflated by instances that were SAE and remain unchanged due to no words in the SAE -> AAE mapping being present in the input). 

I was initially thinking that we would just compare the original instance text to that of the perturbed text and decide whether a perturbation was applied depending on whether the two are different, however, this isn't entirely correct - yet probably okay for most if not all our perturbations, assuming we revisit the tokenizer logic.

The reason for my hesitation is as follows: it is possible for some perturbation to be applied to a text (perhaps by changing pieces of it), but leave the overall text unchanged when everything is put back together. If we want to properly handle this case, we may need to keep track of instance level flag `perturbed:  bool` that we will manually set in our perturbation if we actually applied any perturbation. If we are planning to keep the `Treebank` tokenizers for some of the perturbations, this approach becomes necessary rather than a detail.

I have implemented the alternative approaches I have mentioned in #447

From what I can tell, there are two ways to handle unperturbed instances.
1. Don't produce an additional instance: we make perturbations return `List[Instance]` or `Optional[Instance]` and handle each case appropriately. Then we can evaluate perturbation metrics only on instances that actually have the extra perturbation (and report how many of them do).
2. Produce an extra instance that is cached and ignore during metric eval and website presentation.

IMHO, the first option is the cleaner (the List option also allows producing multiple augmentations per Instance). If people are ok with it I can take a stab at a PR.

Also curious to hear @teetone 's thoughts.

Returning an empty list sounds good to me!

Option 1 seems good to me; can either @dilarasoylu or @dtsip take wrt implementation and update the assignee as necessary. Thanks! 

I'm on it!

Fantastic, thanks!

@dtsip @dilarasoylu What's the status of this? 

Pragmatically, I think this is pretty inconsequential for the robustness perturbations in canonical, but probably it is triggered quite a bit for fairness stuff. 
So it would be good to handle this before we launch OpenAI runs, though in theory if caching works, it should not expend any additional tokens currently as things will keep hitting the cache (though it will change metrics/website). 

Sorry for the delay here, I hope to submit a PR today or early next week.

Created a PR here https://github.com/stanford-crfm/benchmarking/pull/589 .

@dilarasoylu I can't add you as a second reviewer but do you want to take a look since you've worked with perturbations? In particular, the PR relies on the assumption that the input string will remain the same when the perturbation does not apply. I checked that this holds for the robustness perturbations after the changes we made, but it would be good to confirm for the fairness ones.