Repository: promptfoo/promptfoo
Issue: Bug: LocalAI response parsing `No output` when model responded
Root cause: Validation Gap

I am using the LocalAI provider to hack a connection with `llama-cpp-python[server]`.

```yaml
providers:
    - id: localai:models/llama-2-13b-chat.Q4_K_S.gguf
      config:
          apiBaseUrl: http://127.0.0.1:8000/v1
```

The output shown in `promptfoo view`: `[FAIL] No output`.

The `--verbose` output shows a response with "LocalAI API response" (though it's truncated somewhat):

```bash
> promptfoo eval --config prompts/promptfooconfig.yaml --verbose
... Calling LocalAI API: {"model":"models/llama-2-13b-chat.Q4_K_S.gguf","messages":[{"role":"user","content":"<redacted by James>}
Storing http://localhost:8000/v1/chat/completions response in cache: {"id":"chatcmpl-a60ac2bc-c6a9-40b9-b9d9-a2bfd210865d","object":"chat.completion","created":1695256259,"model":"models/llama-2-13b-chat.Q4_K_S.}
	LocalAI API response: {"id":"chatcmpl-a60ac2bc-c6a9-40b9-b9d9-a2bfd210865d","object":"chat.completion","created":1695256259,"model":"models/llama-2-13b-chat.Q4_K_S.gguf","choices":[{"index":0,"message":{}
...
```

Also, the `llama-cpp-python[server]` shows activity, and inspecting the cache entry made:

```json
{
    "expireTime": 1696465861147,
    "key": "fetch:http://localhost:8000/v1/chat/completions:{\"method\":\"POST\",\"body\":\"{\\\"model\\\":\\\"models/llama-2-13b-chat.Q4_K_S.gguf\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"<redacted by James>\\\"}],\\\"temperature\\\":0.7}\"}",
    "val": "{\"id\":\"chatcmpl-a60ac2bc-c6a9-40b9-b9d9-a2bfd210865d\",\"object\":\"chat.completion\",\"created\":1695256259,\"model\":\"models/llama-2-13b-chat.Q4_K_S.gguf\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\" If you are studying the effect of <redacted by James>\"},\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":42,\"completion_tokens\":16,\"total_tokens\":58}}"
}
```

We can see that actually the AI responded, but then `promptfoo` failed to parse the response.

Thus, I believe either:
- LocalAI's response parsing is broken right now
- `llama-cpp-python[server]` has a slight API incompatibility with LocalAI


Here is a query done completely outside of `promptfoo`:

```bash
> curl -X POST --header 'Content-Type: application/json' "http://localhost:8000/v1/chat/completions" -d '{
  "model": "models/llama-2-13b-chat.Q4_K_S.gguf",
  "messages": [{"content": "What is the capital of France?"}]
}' | jq "."
{
  "id": "chatcmpl-f5ceeffa-a7f2-4f22-83a1-5e6217aaaf07",
  "object": "chat.completion",
  "created": 1695257379,
  "model": "models/llama-2-13b-chat.Q4_K_S.gguf",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 11,
    "total_tokens": 27
  }
}
```

Looks like [this line](https://github.com/promptfoo/promptfoo/blob/da8c07643766e68dcc3744103c9225eb16ea3ee1/src/evaluator.ts#L199) is false-y, though I am not sure why it's considered false-y.

Up the stack from here is [this line](https://github.com/promptfoo/promptfoo/blob/da8c07643766e68dcc3744103c9225eb16ea3ee1/src/providers/localai.ts#L78): `{output: data.choices[0].text}`.

Is the issue here that `choices[0].text` is not properly traversing the JSON response?

Great catch - looks like it should be `choices[0].message` in `LocalAiChatProvider`.  1b0dcbe78debcb2edd157b5b64bea5a769f3529a should resolve

One thing to note, is that I was using LocalAI provider as a workaround for `llama-cpp-python[server]`.  Just to check using only LocalAI (what this provider is actually for):

```bash
> curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "lunademo",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9
   }' | jq "."
...
{
  "object": "chat.completion",
  "model": "lunademo",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "I'm doing well, thanks for asking! How about you?"
      }
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}
```

We see LocalAI uses "message" as well, so I agree with the change you made, thank you!

Alright, now running `promptfoo eval` with `promptfoo --version` of 0.24.1 and 

```yaml
providers:
    - id: localai:models/llama-2-13b-chat.Q4_K_S.gguf
      config:
          apiBaseUrl: http://127.0.0.1:8000/v1
```

We get further now, but hit another error:

```none
[FAIL] TypeError: output.includes is not a function

TypeError: output.includes is not a function
at /Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/assertions.js:172:54
at Array.every ()
at runAssertion (/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/assertions.js:172:30)
at runAssertions (/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/assertions.js:49:30)
at Evaluator.runEval (/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/evaluator.js:182:74)
at async /Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/evaluator.js:418:25
```

I am using a `contains-all` assertion.

Do we have a type checker in promptfoo's CI?  I am wondering if using TypeScript how we have a type error, though also I don't really know TypeScript 😅 

Also, @typpo I think there's one more `data.choices[0].text` that needs replacing in `localai.ts`: https://github.com/promptfoo/promptfoo/blob/0.24.1/src/providers/localai.ts#L119

I think the issue here is, it should be `data.choices[0].message.content`, the `.content` was forgotten

The dangers of pushing code past midnight :( 

[This other one](https://github.com/promptfoo/promptfoo/blob/0.24.1/src/providers/localai.ts#L119) seems fine - this is for the completion endpoint, which uses the `data.choices[0].text` format.

All good, I appreciate it nonetheless.

I decided to use the completion endpoint:

```bash
> curl http://localhost:8080/v1/completions -X POST -H 'Content-Type: application/json' -d '{
    "model": "lunademo",
    "prompt": "function downloadFile(string url, string outputPath) {",
    "max_tokens": 256,
    "temperature": 0.5
}' | jq "."
...
{
  "object": "text_completion",
  "model": "lunademo",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "text": " nobody:\n if (url.startsWith('http')) {\n const response = fetch(url);\n response.then(response => response.arrayBuffer());\n } else {\n throw new Error('Invalid URL');\n }\n}\nfunction writeFile(buffer, string path) {\n fs.writeFile(path, buffer, (err) => {\n if (err) throw err;\n console.log(`File downloaded to ${path}`);\n });\n}\nasync function downloadFiles() {\n const urls = [\n 'https://example.com/file1',\n 'https://example.com/file2',\n 'https://example.com/file3'\n ];\n for (const url of urls) {\n try {\n const buffer = await downloadFile(url, 'downloads');\n writeFile(buffer, `file_${url}.txt`);\n } catch (error) {\n console.log(`Error downloading ${url}: ${error}`);\n }\n }\n}\ndownloadFiles();"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}
```

So you're right, leaving it as `data.choices[0].text` for completion endpoint is correct.

I locally cloned/installed the latest `promptfoo` and confirmed this change worked 👍