Repository: stanford-crfm/helm
Issue: Is this implementation right?
Root cause: Documentation Deficiency

Coming from this issue #1875

I was trying to implement some evaluations using HELM's API (which are not publicly exposed in the documentation properly) in a functional manner. Please let me know if I am doing this right (logically)

```python

from helm.benchmark.augmentations.data_augmenter import DataAugmenter
from helm.benchmark.augmentations.dialect_perturbation import DialectPerturbation
from helm.benchmark.scenarios.scenario import Instance, Input, Reference, Output
from helm.benchmark.metrics.basic_metrics import quasi_exact_match


def quasi_exact_match_perturbation_dialect(
    input_text: Union[List[str], str], model: Model
):
    if isinstance(input_text, str):
        input_text = [input_text]
        
    data_augmenter = DataAugmenter(
        perturbations=[
            DialectPerturbation(prob=1.0, source_class="SAE", target_class="AAVE")],
    )
    instance_list = []
    # for now I do not know about the references and output
    for text in input_text:
        instance_list.append(Instance(
            id="id0",
            input=Input(text=text),
            references=[Reference(Output(text="Is this love?"), tags=[])]))
    instances = data_augmenter.generate(instance_list, include_original=True)
    instances = [instance.input.text for instance in instances]
    
    # in instances we have alternate pairs of actual and perturbed 
    # so we will take all the pertubed
    
    perturbed_instances = instances[1::2]
    actual_instances = input_text

    model_generations_actual = model(actual_instances)
    model_generations_perturbed = model(perturbed_instances)
    
    gen_from_perturbed = [generation['text'] for generation in model_generations_perturbed]
    gen_from_actual = [generation['text'] for generation in model_generations_actual]
    
    scores = []
    for gold, pred in zip(gen_from_actual, gen_from_perturbed):
        scores.append(quasi_exact_match(gold=gold, pred=pred))
    return scores
```

Mostly all the fairness-based metrics use some perturbations. In HELM there is a separate class for different kinds of perturbations. So the idea is 

- Take a string
- Perturb that string
- Now input both the strings (actual and perturbed)
- For the outputs generated for both cases, use different metrics like Quasi Excat match etc.

Am I right in this thought process? Also, I have another question on what `references=[Reference(Output(text="Is this love?")` does this mean and how to use this in more customized manner. 

Some parts of the implementation is been taken from HELM's code base (example: `test_perturbations.py`)

Overall your script looks reasonable. Some high level comments:

- We currently don't document the low-level internal API because it is subject to breaking changes in future versions and we don't want people to rely on them. But you are welcome to use this API, and you can pin the version of HELM to avoid breakages. We may make this API more official / public in the future.
- You can look at [`Runner.run_one()`](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/runner.py#L184) if you want to see our pipeline for instance generation, augmentation, model inference and metric computation.
- References are used for two things - to indicate "correct" answers for open ended generations, and to indicate the incorrect and correct options for multiple choice questions.
- Your script computes the percentage similarity between the responses to the original input and responses to the perturbed input (because you treat the former as "gold"), which may not be what you intended (the response to the original input may not always be correct).
- We compute the fairness metrics differently. Instead of computing a metric based on the difference between (response to original input, response to perturbed input) pairs, we compute accuracy over all the responses to the perturbed input only. This is discussed further in the [paper](https://arxiv.org/pdf/2211.09110.pdf) in section 4.6.
    - It's not too difficult to write a new [Metric](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/metrics/metric.py#L98) that does the pairwise comparisons. However, we don't have any existing metrics in the codebase that do this.


That's quite helpful thank you for all the clarification @yifanmai