Repository: promptfoo/promptfoo
Issue: How to best express a QA (question-answer) dataset?
Root cause: Documentation Deficiency

I have a question-answer (QA) dataset ([link](https://github.com/Future-House/LitQA/blob/f5e647b8e53ba440101ea7591df59a24d81fbe23/litqa-v1-dev.jsonl)) of many questions, each with one answer. For each question, I want to assert:

- "cannot answer" is not in the lowercase'd answer
- Length is above 20 chars
- And a few other things

What is the best way to express this in a plain `promptfooconfig.yaml` (no Python)? Should I make one `promptfooconfig.yaml` per question/answer pair?

Hey @jamesbraza, nice to see you around here again!

The way I would structure this is a single config with a test case for each line in the dataset.  For checks that are shared by all test cases (such as "cannot answer", length > 20, etc), I would `defaultTest`, which applies to all test cases:

```yaml
defaultTest:
  assert:
    - type: not-icontains
      value: "cannot answer"
    - type: javascript
      value: output.length > 20
```

Ha thanks yeah happy to be back! Actually, I was waiting a long time for a strong use case, and I finally found an excellent one :) The repo has come quite a long way, and is excellent to use now, so thank you!

I have translated a few of the questions to make the below `promptfooconfig.yaml`. I do have a question, can you advise on how to express a variable length multiple choice question? For my dataset linked in the OP, there can be 0+ "distractor" terms (e.g. wrong options in multiple choice format). In my below config, I handle two distractor terms, but I am wondering how to handle when there are between 0 and 9 distractor terms

```yaml
---
prompts: >-
  Q: {{question}}


  Options:

  A) {{ideal}}

  B) {{distractor_1}}

  C) {{distractor_2}}

  D) Insufficient information to answer this question


  Answer:
providers:
  - openai:chat:gpt-4-0613
  - openai:chat:gpt-4-turbo-2024-04-09
  - anthropic:messages:claude-3-sonnet-20240229
defaultTest:
  assert:
    - description: was answered
      type: not-icontains
      value: cannot answer
    - description: has sentences
      type: javascript
      value: output.length > 20
    - description: check value
      type: python
      value: file://assert.py
tests:
  - description: 1e5f5199-84f4-4133-ab87-2372fa6ca722
    vars:
      question: >-
        Does protein encoded by TDRD1 have symmetrical or asymmetrical
        dimethylarginine modifications, or no dimethylarginine at all?
      ideal: No dimethylarginine
      distractor_1: Symmetrical dimethylarginine modifications
      distractor_2: Asymmetrical dimethylarginine modifications
  - description: a1d01019-d2b1-4619-92ec-7ea38578819f
    vars:
      question: >-
        What is the mechanism for expression of the EPRS1(N1) form of EPRS1 in
        humans?
      ideal: polyadenylation
      distractor_1: alternative splicing
      distractor_2: post-transcriptional control
  - description: 38ada695-2c79-459b-8e68-dbed734c74f9
    vars:
      question: >-
        What effect does infection of A. thaliana plants with hopM1 knockout
        Pst DC3000 have on NCED3 expression?
      ideal: Increased expression
      distractor_1: Reduced expression
      distractor_2: No effect
```

```python
# assert.py

import re


def extract_answer(answer: str) -> str:
    # first capital letter, like A)
    s = re.search(r"([A-Z])\)", answer, re.DOTALL)
    if s is not None:
        return s.group(1)
    return answer.split()[0][0].upper()


def get_assert(output: str, context: dict) -> dict:
    result = extract_answer(output).strip().upper()
    ideal = context["vars"]["ideal"].upper()
    match result[0] == "D", ideal == "NULL", result[0] == ideal[0]:
        case (True, True, _) | (False, False, True):
            is_pass, score, reason = True, 1.0, "Correct"
        case True, _, False:
            is_pass, score, reason = False, 0.1, "Unsure"
        case False, *_:
            is_pass, score, reason = False, 0.0, "Incorrect"
        case _:
            raise NotImplementedError(f"Unhandled result {result} and ideal {ideal}.")
    return {"pass": is_pass, "score": score, "reason": reason}
```

> can you advise on how to express a variable length multiple choice question?

I ended up going with one `promptfooconfig.yaml` that:

- Has the `defaultTests`, `prompts`, and `providers` as static
    - `prompts` has a template variable `options` that gets all (variable length) multiple choice options at once
- `tests` are programmatically generated from the QA dataset via an external Python script

Having the `tests` be programmatically generated enabled me to handle the variance in the number of `options`.

If you want to see how exactly this was done, check: https://github.com/Future-House/LitQA/pull/32

Going to close this out, thank you for your response Ian! It set me down the right path to a solution