Repository: promptfoo/promptfoo
Issue: How to properly set perplexity model to run via a YAML file?
Root cause: Configuration Error

Hi and first of all many thanks for this amazing platform to run LLM evaluation!

I am now looking for a proper way to set up the perplexity API model. 

Following [this short tutorial](https://www.promptfoo.dev/docs/providers/perplexity), I put below model provider information into the YAML file:
```
providers:
  - id: openai:chat:gpt-3.5-turbo-0613
  - id: openai:chat:gpt-4-0613
  - id: openai:chat:pplx-70b-chat-alpha
    config:
      apiHost: api.perplexity.ai
```
And I have the two environment variables set (put empty values here for privacy reason. Can confirm they are workable keys. Thanks for understanding):
```
export OPENAI_API_KEY=""
export PERPLEXITY_API_KEY=""
```
They when I was running the yaml, I found the two openai GPT models are responding as expected, but not for the perplexity model with below returned output for that category. Did I do anything wrong here?
```
openai:chat:pplx-70b-chat-alpha
0.00% passing (0/9 cases)
Answer this question: {{question}}

FAILAPI call error: Error: Error parsing response from https://api.perplexity.ai/v1/chat/completions: FetchError: invalid json response body at https://api.perplexity.ai/v1/chat/completions reason: Unexpected end of JSON input
```
Thanks in advance for answering!

I think the problem here is that you are trying to use perplexity _and_ openai APIs at the same time.  That is a drawback to the "just override OpenAI" approach to supporting Perplexity.

If you set `apiKey` for perplexity in the YAML file, rather than envar, I think that might temporarily workaround your problem:

```yaml
  - id: openai:chat:pplx-70b-chat-alpha
    config:
      apiHost: api.perplexity.ai
      apiKey: xxx
```

#456 makes it possible to override apiKeys without having to hardcode them.  Once it lands, you can do something like:

```yaml
  - id: openai:chat:pplx-70b-chat-alpha
    config:
      apiHost: api.perplexity.ai
      apiKeyEnvar: PERPLEXITY_API_KEY
```
This makes it so you don't have to hardcode the API key in your config to get the desired result :)

This change is published in 0.40.0

@typpo So many thanks for the prompt reply and effort! And I apologize about my late response on this issue.

I switched the version of promptfoo to 0.40.0, and then added the suggested configuration into my YAML file:
```
providers:
  - id: openai:chat:gpt-3.5-turbo-0613
  - id: openai:chat:pplx-70b-chat-alpha
    config:
      apiHost: api.perplexity.ai
      apiKeyEnvar: PERPLEXITY_API_KEY
```
But then, same error appeared. This time I actually noticed this message below at the beginning of terminal output:
```
Using unknown OpenAI chat model: pplx-70b-chat-alpha
Eval: [████████████████████████████████████████] 100% | ETA: 0s | 27/27 | openai:chat:pplx-70b-chat-alpha "Answer thi" question=W
```
But according to the perplexity tab in the documentation page, `openai:chat:pplx-70b-chat-alpha` shall be there right?

The warning message is safe to ignore.  I'll make a change to suppress it when `apiHost` is overridden.

@typpo So the error should not be related to the problem of invalid model right? Which means `openai:chat:pplx-70b-chat-alpha` should be OK and there was something else went wrong.

It's not actually an error.  It's just a warning that `pplx-70b-chat-alpha` is not a known OpenAI model.

@typpo Thanks for the fix on this issue. But the same error still exist unfortunately. I use below setup this time:
```
providers:
  - id: openai:chat:gpt-3.5-turbo-0613
  - id: openai:chat:llama-2-70b-chat
    config:
      apiHost: api.perplexity.ai
      apiKeyEnvar: PERPLEXITY_API_KEY
```
And while the result from GPT-3.5 looks fine, the result from perplexity side is
```
[FAIL] API call error: Error: Error parsing response from https://api.perplexity.ai/v1/chat/completions: FetchError: invalid json response body at https://api.perplexity.ai/v1/chat/completions reason: Unexpected end of JSON input
```

Thanks for following up @gillsgills.  

https://github.com/promptfoo/promptfoo/pull/464 should solve this problem, as it looks like there is a minor incompatibility with the OpenAI API (lack of a version param in API route)

I've also tested it with perplexity API myself and added an example https://github.com/promptfoo/promptfoo/pull/463

@typpo Thanks a lot for your prompt response and effort!

So since #464 is yet to be merged, I wonder if I can have a taste by installing from the local source? Or I may wait a bit for the sake of stability.

Please give it a try in 0.41.0 :)

Hi @typpo I know I have been really a nuisance here by keep having trouble on this question, but seems like it is still not working well.
Below is my local_path.sh setup. I source it to install the promptfoo:
```
npm install promptfoo@0.41.0
```

Then I run `promptfoo eval -c test_output.yaml`. `test_output.yaml` is like below
```
prompts:
  - "Who discovered {{topic}}?"

providers:
  # Compare GPT 3.5 vs Perplexity 70B
  - id: openai:chat:gpt-3.5-turbo-0613
  - id: openai:chat:pplx-70b-online
    config:
      apiBaseUrl: https://api.perplexity.ai
      apiKeyEnvar: PERPLEXITY_API_KEY
      frequency_penalty: 0.1

tests:
  - vars:
      topic: gravity
```

The output is
```
$ promptfoo eval -c test_output.yaml 
Using unknown OpenAI chat model: pplx-70b-online
Eval: [████████████████████████████████████████] 100% | ETA: 0s | 2/2 | openai:chat:pplx-70b-online "Who discov" topic=grav

┌────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ topic                                                      │ [openai:chat:gpt-3.5-turbo-0613] Who discovered {{topic}}? │ [openai:chat:pplx-70b-online] Who discovered {{topic}}?    │
├────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ gravity                                                    │ [PASS] Sir Isaac Newton is credited with discove           │ [FAIL] API call error: Error: Error par                    │
│                                                            │ ring gravity.                                              │ sing response from https://api.perplexity.ai/v1/chat/compl │
│                                                            │                                                            │ etions: FetchError: invalid json response body at https:// │
│                                                            │                                                            │ api.perplexity.ai/v1/chat/completions reason: Unexpected e │
│                                                            │                                                            │ nd of JSON input                                           │
│                                                            │                                                            │ ---                                                        │
└────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
======================================================================================================================================================================================
✔ Evaluation complete.

Run promptfoo view to use the local web viewer
Run promptfoo share to create a shareable URL
Run promptfoo feedback to share feedback with the developers of this tool
======================================================================================================================================================================================
Successes: 1
Failures: 1
Token usage: Total 20, Prompt 11, Completion 9, Cached 0
Done.
```
I believe after all the struggles and problems, it can be either networking or my stupid mistake....what can it be?

No problem, you are not a nuisance.  Sorry this has been such a struggle to set up.

It's strange because I'm able to get that exact config working just fine.  Can you double check that you've set the `PERPLEXITY_API_KEY` environment variable and that is in fact a valid API key?  Alternatively, if you set `apiKey` instead of `apiKeyEnvar`, what do you see?

Thanks a lot for understanding.

I have changed `apiKeyEnvar` to `apiKey` and same error occured. I tried my key on [this website](https://docs.perplexity.ai/reference/post_chat_completions) (right-hand side, "Bearer") and it can return proper answer. So I guess I have the right key? Or do you have any other right way to valid the key, if what I did is the wrong way to check?
```
{
  "id": "5d3b56e8-7864-49b5-898d-3fa2cd7dc607",
  "model": "mistral-7b-instruct",
  "created": 9266019,
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 50,
    "total_tokens": 75
  },
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "The exact number of stars in the Milky Way galaxy is not known, but it is estimated to be between 100-400 billion. The latest estimate from astronomers suggests there could be around 200 billion stars."
      },
      "delta": {
        "role": "assistant",
        "content": ""
      }
    }
  ]
}
```

Hi @typpo sorry from my side for being busy with other issues. Any follow-up on this?