Repository: promptfoo/promptfoo
Issue: Bug: `--no-cache` leads to `FetchError: invalid json response body`
Root cause: Validation Gap

When using `--no-cache` CLI option for `eval`, it breaks Ollama:

```
[FAIL] API call error: FetchError: invalid json response body at http://127.0.0.1:11434/api/generate reason: Unexpected non-whitespace character after JSON at position 99
```

I did not set `PROMPTFOO_CACHE_ENABLE`


Can't repro this error with ollama and `promptfoo eval --no-cache` - feel free to reopen with more details!

I am still getting the error, try with my simplified set up. Without `--no-cache` it works fine, so I know something is up here:

```txt
Specifically, I am interested in {{ area }}. Out of everything, which is the most common {{ area }}?
```

```yaml
---
defaultTest:
    options:
        provider: localai:embeddings:bert-embeddings
providers: ["ollama:llama2:13b"]
sharing: false
scenarios:
    - config:
          - vars:
                area: foo
                result: bar
                expected: foo is the most common bar.
      tests:
          - assert:
                - type: similar
                  value: "{{expected}}"
                  threshold: 0.60
```

You can download the `localai` model via:

```bash
curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{
     "id": "model-gallery@bert-embeddings"
   }'
```

From `npm run promptfoo -- eval --no-cache --output out.yaml`:

```
results:
  version: 2
  results:
    - provider:
        id: ollama:llama2:13b
      prompt:
        raw: >-
          Specifically, I am interested in foo. Out of everything, which is the
          most common foo?
        display: >-
          Specifically, I am interested in {{ area }}. Out of everything, which
          is the most common {{ area }}?
      vars:
        area: foo
        result: bar
        expected: foo is the most common bar.
      response:
        error: >-
          API call error: FetchError: invalid json response body at
          http://127.0.0.1:11434/api/generate reason: Unexpected non-whitespace
          character after JSON at position 96
      success: false
      score: 0
      latencyMs: 11454
      error: >-
        API call error: FetchError: invalid json response body at
        http://127.0.0.1:11434/api/generate reason: Unexpected non-whitespace
        character after JSON at position 96
  stats:
    successes: 0
    failures: 1
    tokenUsage:
      total: 0
      prompt: 0
      completion: 0
      cached: 0
  table:
    head:
      prompts:
        - raw: >-
            Specifically, I am interested in {{ area }}. Out of everything,
            which is the most common {{ area }}?
          display: >-
            Specifically, I am interested in {{ area }}. Out of everything,
            which is the most common {{ area }}?
      vars:
        - area
        - expected
        - result
    body:
      - outputs:
          - pass: false
            score: 0
            text: >
              API call error: FetchError: invalid json response body at
              http://127.0.0.1:11434/api/generate reason: Unexpected
              non-whitespace character after JSON at position 96

              ---
            prompt: >-
              Specifically, I am interested in foo. Out of everything, which is
              the most common foo?
            latencyMs: 11454
        vars:
          - foo
          - foo is the most common bar.
          - bar
config:
  prompts:
    - prompts.txt
  providers:
    - ollama:llama2:13b
  scenarios:
    - config:
        - vars:
            area: foo
            result: bar
            expected: foo is the most common bar.
      tests:
        - assert:
            - type: similar
              value: '{{expected}}'
              threshold: 0.6
  sharing: false
  defaultTest:
    options:
      provider: localai:embeddings:bert-embeddings
shareableUrl: null
```

What other info can I provide to help you investigate this?

Hmm, thanks for the detailed repro, I'll give it a try locally tomorrow :)

thanks for the catch - https://github.com/promptfoo/promptfoo/commit/ec4576208e0b3540a69a2dcbd70ad12a897347e8 will fix

Thanks for the fix! I guess when not caching, it's text as opposed to JSON? Trying to understand how the fix works haha

Yeah, when cache is disabled there's a separate code path that did not correctly handle the response type.