Repository: promptfoo/promptfoo
Issue: Sample js for custom embedding
Root cause: Validation Gap

I want to use an external model embedding, I could use custom api provider for text, however not for embedding. any example code would be helpful

I've added a custom embeddings provider example.  See https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider-embeddings and let me know if you have any questions

**First of all thank you for the response!! I really appreciate you added example 👍** 

My **embedding model responds** in below format

```
{
  "results": [
    [
      -0.17895537614822388,
      0.03200146183371544,
      -0.030327674001455307,
      ...
    ]
  ]
}
```

my **customProvider.js**

```
const fetch = require('node-fetch');

class CustomApiProvider {
  constructor(options) {
    // The caller may override Provider ID (e.g. when using multiple instances of the same provider)
    this.providerId = options.id || 'custom provider';

    // The config object contains any options passed to the provider in the config file.
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt) {
    return null;
  }

  async callEmbeddingsApi(prompt) {
    const body = {
      model_id: 'maxorg/hobbit.30m.english.rtrvr',
      input: prompt,
    };
    const response = await fetch('https://maxorg.com/v2/text/embeddings', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      },
      body: JSON.stringify(body),
    });

    const data = await response.json();
    if (!data) {
      return {
        error: 'Unknown error',
      }
    }
    const ret = {
      embedding: data.results[0],
    };
    return ret;
  }
}

module.exports = CustomApiProvider;
```

**promptfooconfig.yaml** 

```
prompts:
  - 'Translate this to French: {{message}}'

providers:
  - id: python:test.py

tests:
  - vars:
      message: Hello world
    assert:
      - type: similar
        threshold: 0.25
        value: 'Greetings earth'
        provider: ["./customProvider.js"]
```

**npx promptfoo@latest eval**

```
(node:98615) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Eval: [████████████████████████████████████████] 100% | ETA: 0s | 1/1 | python:test.py "Translate " message=He

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ message                                                                                                          │ Translate this to French: {{message}}                                                                            │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Hello world                                                                                                      │ [FAIL] Error: Invalid provider definition for output type 'embedding': [                                         │
│                                                                                                                  │   "./customProvider.js"                                                                                          │
│                                                                                                                  │ ]                                                                                                                │
│                                                                                                                  │                                                                                                                  │
│                                                                                                                  │ Error: Invalid provider definition for output type 'embedding': [                                                │
│                                                                                                                  │   "./customProvider.js"                                                                                          │
│                                                                                                                  │ ]                                                                                                                │
│                                                                                                                  │     at getGradingProvider (/Users/nishantchauhan/.npm/_npx/81b...                                                │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================================================================================================================================================
✔ Evaluation complete.
```




is there anything wrong ? I tried this too ```provider: ./customProvider.js```  in  ```promptfooconfig.yaml``` , did not work


![image](https://github.com/promptfoo/promptfoo/assets/35916341/27138388-9e08-4812-9170-d5e13c0ea149)


Oops, I made a typo: try `callEmbeddingApi`, without the `s`

Thanks!

But now, OpenAI example is also not working 

```
(base) nishantchauhan@Nishants-MacBook-Air custom-provider-embeddings % cat customProvider.js 
const fetch = require('node-fetch');

class CustomApiProvider {
  constructor(options) {
    // The caller may override Provider ID (e.g. when using multiple instances of the same provider)
    this.providerId = options.id || 'custom provider';

    // The config object contains any options passed to the provider in the config file.
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt) {
    return null;
  }

  async callEmbeddingApi(prompt) {
    const body = {
      model: 'text-embedding-3-large',
      input: prompt,
    };
    const response = await fetch('https://api.openai.com/v1/embeddings', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      },
      body: JSON.stringify(body),
    });

    const data = await response.json();
    if (!data) {
      return {
        error: 'Unknown error',
      }
    }
    const ret = {
      embedding: data.data[0].embedding,
      tokenUsage: {
        total: data.usage.total_tokens,
        prompt: data.usage.prompt_tokens,
        completion: 0,
      },
    };
    return ret;
  }
}

module.exports = CustomApiProvider;
(base) nishantchauhan@Nishants-MacBook-Air custom-provider-embeddings % npx promptfoo@latest eval
(node:30347) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Eval: [████████████████████████████████████████] 100% | ETA: 0s | 1/1 | python:test.py "Translate " message=He

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ message                                                                                                          │ Translate this to French: {{message}}                                                                            │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Hello world                                                                                                      │ [FAIL] Error: Invalid provider definition for output type 'embedding': [                                         │
│                                                                                                                  │   "./customProvider.js"                                                                                          │
│                                                                                                                  │ ]                                                                                                                │
│                                                                                                                  │                                                                                                                  │
│                                                                                                                  │ Error: Invalid provider definition for output type 'embedding': [                                                │
│                                                                                                                  │   "./customProvider.js"                                                                                          │
│                                                                                                                  │ ]                                                                                                                │
│                                                                                                                  │     at getGradingProvider (/Users/nishantchauhan/.npm/_npx/81b...                                                │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================================================================================================================================================
✔ Evaluation complete.

» Run promptfoo view to use the local web viewer
» Run promptfoo share to create a shareable URL
» This project needs your feedback. What's one thing we can improve? https://forms.gle/YFLgTe1dKJKNSCsU7
====================================================================================================================================================================================================================================
Successes: 0
Failures: 1
Token usage: Total 0, Prompt 0, Completion 0, Cached 0
Done.
```

"provider" in the assertion should be a string, not an array

On Thu, Mar 14, 2024, 7:54 PM nishantchauhan04 ***@***.***>
wrote:

> Thanks!
>
> But now, OpenAI example is also not working
>
> (base) ***@***.*** custom-provider-embeddings % cat customProvider.js
> const fetch = require('node-fetch');
>
> class CustomApiProvider {
>   constructor(options) {
>     // The caller may override Provider ID (e.g. when using multiple instances of the same provider)
>     this.providerId = options.id || 'custom provider';
>
>     // The config object contains any options passed to the provider in the config file.
>     this.config = options.config;
>   }
>
>   id() {
>     return this.providerId;
>   }
>
>   async callApi(prompt) {
>     return null;
>   }
>
>   async callEmbeddingApi(prompt) {
>     const body = {
>       model: 'text-embedding-3-large',
>       input: prompt,
>     };
>     const response = await fetch('https://api.openai.com/v1/embeddings', {
>       method: 'POST',
>       headers: {
>         'Content-Type': 'application/json',
>         Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
>       },
>       body: JSON.stringify(body),
>     });
>
>     const data = await response.json();
>     if (!data) {
>       return {
>         error: 'Unknown error',
>       }
>     }
>     const ret = {
>       embedding: data.data[0].embedding,
>       tokenUsage: {
>         total: data.usage.total_tokens,
>         prompt: data.usage.prompt_tokens,
>         completion: 0,
>       },
>     };
>     return ret;
>   }
> }
>
> module.exports = CustomApiProvider;
> (base) ***@***.*** custom-provider-embeddings % npx ***@***.*** eval
> (node:30347) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
> (Use `node --trace-deprecation ...` to show where the warning was created)
> Eval: [████████████████████████████████████████] 100% | ETA: 0s | 1/1 | python:test.py "Translate " message=He
>
> ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
> │ message                                                                                                          │ Translate this to French: {{message}}                                                                            │
> ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
> │ Hello world                                                                                                      │ [FAIL] Error: Invalid provider definition for output type 'embedding': [                                         │
> │                                                                                                                  │   "./customProvider.js"                                                                                          │
> │                                                                                                                  │ ]                                                                                                                │
> │                                                                                                                  │                                                                                                                  │
> │                                                                                                                  │ Error: Invalid provider definition for output type 'embedding': [                                                │
> │                                                                                                                  │   "./customProvider.js"                                                                                          │
> │                                                                                                                  │ ]                                                                                                                │
> │                                                                                                                  │     at getGradingProvider (/Users/nishantchauhan/.npm/_npx/81b...                                                │
> └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
> ====================================================================================================================================================================================================================================
> ✔ Evaluation complete.
>
> » Run promptfoo view to use the local web viewer
> » Run promptfoo share to create a shareable URL
> » This project needs your feedback. What's one thing we can improve? https://forms.gle/YFLgTe1dKJKNSCsU7
> ====================================================================================================================================================================================================================================
> Successes: 0
> Failures: 1
> Token usage: Total 0, Prompt 0, Completion 0, Cached 0
> Done.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/551#issuecomment-1998850247>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJU2TYYKSR6FXAZBATDYYJPGNAVCNFSM6AAAAABEUWJXFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJYHA2TAMRUG4>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***>
>


Awesome!! it worked. Thanks a lot!