Repository: stanford-crfm/helm
Issue: Add structure to instances
Root cause: Architectural Constraint

We are currently implementing prompting logic inside each scenario. For instance, for QA, we are defining how to combine the passage and the question at scenario-creation time. Ideally, we would just output a QAInstance(question=q, answer=a) which gets turned into a prompt during preprocessing.

This would also give us some extra hooks when defining perturbations (e.g., we would be able to apply perturbations only to the passage of a QA input without worrying about the formatting of the input).

@dtsip Is this resolved/what's the status?

That's a solid p2, the current behavior is exactly what we want and this is about refactoring the codebase. I plan to get to it once all the p1s quiet down a bit.

@dtsip what's the status on this (also shifting assignment to you from percy)

@rishibommasani hm, taking a closer look, it looks like this requires quite a few changes in the codebase that touch many scenarios/components. Not sure I actually have capacity/codebase understanding to do it. We might want to think how much of a priority this is.

I think the change I had in mind is to just define a utility function `def PassageQuestion(passage: str, question: str) -> str` which returns the input.  This at least tries to provide a lightweight abstraction for now.  After that, we should define different types of inputs `RawInput(text: str)` and `PassageQuestionInput(passage: str, question: str)`, and eventually this could be generalized to multimodal models, but this would require more changes (though I think the changes would be fairly mechanical).

Hm, ok, let me take a look. Something like that might be not too hard.

Note: It would be slick and I would say more correct if:

1. We made this change, so all of the QA formatting can be forcibly standardized through this. So a win for standardization.
2. We then specified this at the time of adaptation, so all prompting-type decisions are governed by the adaptation parameters (independent of how scenario-specific or scenario-general they could be argued), since fundamentally the need for prompting only arises at the time of adaptation when one choose to prompt (whereas scenarios are defined before adaptation and should not think about adaptation in any way, including whether it will be prompting or not). So a win for clarity, as this consolidates all prompting decisions. 
3. We can then address prompting changes like https://github.com/stanford-crfm/benchmarking/issues/437 by simply writing a RunExpander. So a win for elegance. And then we include the results in the paper as a one-off experiment.

I just submitted [https://github.com/stanford-crfm/benchmarking/pull/740](https://github.com/stanford-crfm/benchmarking/pull/740) which should be a first step in that direction.

Eventually, we want to move this logic to the adapter and convert `Instance.input` from `str` to the new dataclass `Input` everywhere. This should be relatively straightforward:
- [ ]  Replace every call to `Instance(input: str)` with `Instance(input: Input)`. For most scenarios this corresponds to just calling `RawInput(prompt)`.
- [ ]  Replace every access to `Instance.input` with `Instance.input.to_text(**args)`.

I don't think I have the capacity to implement and test this right now, since it touches the whole codebase. I am removing myself for now. (Should we move to p3?)

Ya, I think what we have is fine. Let's move to p3 at this point.

Done (cc @teetone )