Repository: promptfoo/promptfoo
Issue: Using `repeat` with cache does not work as expected
Root cause: Algorithmic Error

If I run `promptfoo eval` using cache (default) and the `--repeat` option, it runs the test once, caches it, and uses the cached output again for the rest of the repetitions.

This is how I'm running `promptfoo eval`:

```bash
promptfoo eval \
  -j 1 \
  --repeat 5 \ 
  --providers {model} \
  -o outputs/{model_name}.html \
  -o outputs/{model_name}.csv \
  -o outputs/{model_name}.txt \
  -o outputs/{model_name}.json \
  -o outputs/{model_name}.yaml
```

where `{model}` and `{model_name}` are something like `ollama:mistral:7b-instruct-fp16` and `mistral:7b-instruct-fp16`, respectively. I need to add the parameter `-j 1`, otherwise it freezes with Ollama when multiple requests are performed.

The `promptfooconfig.yaml` basically tests how different prompts are able to improve a paragraph. It looks like this:

```yaml
prompts:
  - file://../../prompts/baseline.txt
  - file://../../prompts/candidate.txt
providers:
  - ollama:mistral:7b-instruct-fp16
scenarios:
  - config:
      - vars:
          test_description: spelling/grammar errors
          title: Projecting genetic associations through gene expression patterns highlights disease etiology and drug mechanisms
          keywords: "genetic studies, functional genomics, gene co-expression, therapeutic targets, drug repurposing, clustering of complex traits"
          paragraph: Genes act in concierto with each other in specifico contexts to perform their functions. Determining how these genes influence complex traits requires a mechanistic understanding of expression regulation across different conditions. It has been shown that this insight is critical for developing new therapies. Transcriptome-wide association studies have helped uncover the role of individual genes in disease-relevant mechanisms. However, modern models of the architecture of complex traits predict that gene-gene interactions play a crucial role in disease origin and progression. Here we introduce PhenoPLIER, a computational approach that maps gene-trait associations and pharmacological perturbation data into a common latent representation for a joint analysis. This representation is base on modules of genes with similar expression patterns across the same conditions. We observe that diseases are significantl associated with gene modules expressed in relevant cell types, and our approach is accurate in predicting known drug-disease pairs and inferring mechanisms of action. Furthermore using a CRISPR screen to analyze lipid regulation, we find that functionally important players lack associations but are prioritized in trait-associated modules by PhenoPLIER. In conclusion, by incorporating, groups of co-expressed genes, PhenoPLIER can contextualize genetic associations and reveal, potential targets missed by single-gene strategies. PhenoPLIER provides a novel framework for interpreting genetic associations through the lens of gene modules.
      - vars:
          test_description: does not start with background
          title: Projecting genetic associations through gene expression patterns highlights disease etiology and drug mechanisms
          keywords: "genetic studies, functional genomics, gene co-expression, therapeutic targets, drug repurposing, clustering of complex traits"
          paragraph: We introduce PhenoPLIER, a computational approach that maps gene-trait associations and pharmacological perturbation data into a common latent representation for a joint analysis. Genes act in concert with each other in specific contexts to perform their functions. Determining how these genes influence complex traits requires a mechanistic understanding of expression regulation across different conditions. It has been shown that this insight is critical for developing new therapies. Transcriptome-wide association studies have helped uncover the role of individual genes in disease-relevant mechanisms. However, modern models of the architecture of complex traits predict that gene-gene interactions play a crucial role in disease origin and progression. PhenoPLIER's representation is based on modules of genes with similar expression patterns across the same conditions. We observe that diseases are significantly associated with gene modules expressed in relevant cell types, and our approach is accurate in predicting known drug-disease pairs and inferring mechanisms of action. Furthermore, using a CRISPR screen to analyze lipid regulation, we find that functionally important players lack associations but are prioritized in trait-associated modules by PhenoPLIER. In conclusion, by incorporating groups of co-expressed genes, PhenoPLIER can contextualize genetic associations and reveal potential targets missed by single-gene strategies. PhenoPLIER provides a novel framework for interpreting genetic associations through the lens of gene modules.
      - vars:
          test_description: does not describe the method or approach proposed
          title: Projecting genetic associations through gene expression patterns highlights disease etiology and drug mechanisms
          keywords: "genetic studies, functional genomics, gene co-expression, therapeutic targets, drug repurposing, clustering of complex traits"
          paragraph: Genes act in concert with each other in specific contexts to perform their functions. Determining how these genes influence complex traits requires a mechanistic understanding of expression regulation across different conditions. It has been shown that this insight is critical for developing new therapies. Our results in this study show that diseases are significantly associated with gene modules expressed in relevant cell types, and our approach is accurate in predicting known drug-disease pairs and inferring mechanisms of action. Furthermore, using a CRISPR screen to analyze lipid regulation, we find that functionally important players lack associations but are prioritized in trait-associated modules by PhenoPLIER. In conclusion, by incorporating groups of co-expressed genes, PhenoPLIER can contextualize genetic associations and reveal potential targets missed by single-gene strategies. PhenoPLIER provides a novel framework for interpreting genetic associations through the lens of gene modules.
      - vars:
          test_description: does not provide an interpretation of the results and their implications
          title: Projecting genetic associations through gene expression patterns highlights disease etiology and drug mechanisms
          keywords: "genetic studies, functional genomics, gene co-expression, therapeutic targets, drug repurposing, clustering of complex traits"
          paragraph: Genes act in concert with each other in specific contexts to perform their functions. Determining how these genes influence complex traits requires a mechanistic understanding of expression regulation across different conditions. It has been shown that this insight is critical for developing new therapies. Transcriptome-wide association studies have helped uncover the role of individual genes in disease-relevant mechanisms. However, modern models of the architecture of complex traits predict that gene-gene interactions play a crucial role in disease origin and progression. Here we introduce PhenoPLIER, a computational approach that maps gene-trait associations and pharmacological perturbation data into a common latent representation for a joint analysis. This representation is based on modules of genes with similar expression patterns across the same conditions. We observe that diseases are significantly associated with gene modules expressed in relevant cell types, and our approach is accurate in predicting known drug-disease pairs and inferring mechanisms of action. Furthermore, using a CRISPR screen to analyze lipid regulation, we find that functionally important players lack associations but are prioritized in trait-associated modules by PhenoPLIER.
    tests:
      - description: Abstract
        assert:
          # Starts with capital letter
          - type: python
            value: output.strip()[0].isupper()

          # Ends with period
          - type: python
            value: output.strip()[-1] == "."

          # Single paragraph
          - type: python
            value: len(output.strip().split("\n")) == 1
```

I'm including just a few test asserts.

I want to run different models multiple times with the same prompt and paragraph. For this, I use `--repeat`, but this only works if I disable cache when calling `promptfoo eval` with `--no-cache`. I would expect that when you use, for instance, `--repeat 5` and cache at the same time, promptfoo would generate a new output for each prompt on each paragraph (in the example above there are 2 prompts and 4 paragraphs) across the 5 repetitions. That is, it will hit the API forty (40) times. But if I run it again with exactly the same command, it would use the cache and never hit the API (unless I previously clear the cache). Does this make sense or I'm missing something? I wonder if this is related to the use of `scenarios`.

Thank you again for the great work!

Thanks for catching this.  #361 will disable cache if repeat > 1

I understand that #361 will make impossible to use cache with repeat. It would still be nice to cache the outputs of each repeat, although not sure how hard would be to implement that.

Fix is in 0.34.0!

I'm going to close this for now, as I think it would be tricky to implement `repeat` and `cache` simultaneously (and with marginal benefit, because the main use of `repeat` is to capture the randomness in outputs caused by temperature)