Repository: promptfoo/promptfoo
Issue: Request: allowing `openai:embeddings` as a provider
Root cause: Validation Gap

I am trying to use `openai:embeddings` directly, and I think promptfoo isn't quite set up for that: 

```
[FAIL] Error: Unknown OpenAI model type: embeddings. Use one of the following providers: openai:chat:, openai:completion:

Error: Unknown OpenAI model type: embeddings. Use one of the following providers: openai:chat:, openai:completion:
at loadApiProvider (/Users/user/code/promptfoo/dist/src/providers.js:122:19)
at getGradingProvider (/Users/user/code/promptfoo/dist/src/matchers.js:40:63)
at matchesSimilarity (/Users/user/code/promptfoo/dist/src/matchers.js:62:31)
at runAssertion (/Users/user/code/promptfoo/dist/src/assertions.js:367:56)
at runAssertions (/Users/user/code/promptfoo/dist/src/assertions.js:49:30)
at Evaluator.runEval (/Users/user/code/promptfoo/dist/src/evaluator.js:182:74)
at async /Users/user/code/promptfoo/dist/src/evaluator.js:418:25
```

Can we enable `openai:embeddings` to be used?

https://promptfoo.dev/docs/providers/openai suggests this is possible: `openai:<endpoint>:<model>`

---

Until https://github.com/promptfoo/promptfoo/issues/175 is done, I am trying to hack invoking LocalAI embeddings endpoint using the OpenAI provider and `export OPENAI_API_HOST="localhost:8080"` as suggested [here](https://github.com/promptfoo/promptfoo/issues/142#issuecomment-1722086680).

```yaml
defaultTest:
    options:
        provider: openai:embeddings:bert-embeddings
```

The `openai:embeddings` alias is not supported - you'll have to specify a full model like `openai:embedding:text-embedding-ada-002`

Look at the bottom of that request, I think I was:

```yaml
defaultTest:
    options:
        provider: openai:embeddings:bert-embeddings
```

What do you think from there?

That will work only if OpenAI (or your localhost) supports an embeddings model named `bert-embeddings`.  If that model exists, it will send requests to that endpoint.

Note that the prefix is `openai:embedding:` with "embedding" singular.

I am gaming `bert-embeddings` from LocalAI (per [here](https://localai.io/models/)) until https://github.com/promptfoo/promptfoo/issues/175 is in, using OpenAI (since they have the same API).

> Note that the prefix is openai:embedding: with "embedding" singular.

That worked! Moving forward.

Fwiw, I think `embeddings` would be the better choice over `embedding`, since `embeddings` (with `s`) matches OpenAI's endpoint: https://platform.openai.com/docs/guides/embeddings/what-are-embeddings

---

Tomorrow, I will make a docs PR for OpenAI mentioning `openai:embedding:<model>` somewhere in the provider page

Sadly, [this OpenAI API key validation](https://github.com/promptfoo/promptfoo/blob/0.24.1/src/providers/openai.ts#L87) denies me in my backdoor here:

```none
Error: OpenAI API key must be set for similarity comparison

Error: OpenAI API key must be set for similarity comparison
at OpenAiEmbeddingProvider.callEmbeddingApi (/Users/user/code/promptfoo/dist/src/providers/openai.js:45:19)
at matchesSimilarity (/Users/user/code/promptfoo/dist/src/matchers.js:64:51)
at async runAssertion (/Users/user/code/promptfoo/dist/src/assertions.js:367:17)
at async runAssertions (/Users/user/code/promptfoo/dist/src/assertions.js:49:24)
at async Evaluator.runEval (/Users/user/code/promptfoo/dist/src/evaluator.js:182:37)
at async /Users/user/code/promptfoo/dist/src/evaluator.js:418:25
```

I can fake it out using `export OPENAI_API_KEY="fake"` though 😈 .

I wonder, should we only run the API key validation during a `catch` clause, to throw a better error?  That way, this backdoor would work without having to set `OPENAI_API_KEY`.

This doesn't work for azure, any work arounds for this?

Opened https://github.com/promptfoo/promptfoo-docs/pull/13 to document `openai:embedding:<model name>`.  @typpo feel free to close this out, unless you feel like loosening the `OPENAI_API_KEY` validations as I posed above.

---

@kevin2107 when you say "this", you mean the embedding endpoint?  Fwiw, this issue is resolved from my standpoint, maybe open a separate issue

Yep the embedding endpoint. Will open up another issue. Thought this was fixed in a previous pr but looks like it still doesn't work