Repository: promptfoo/promptfoo
Issue: Error running the "customProvider" example
Root cause: Documentation Deficiency

I am trying to run the customProvider example:
https://github.com/promptfoo/promptfoo/blob/9627cc7cb52845e5be6e4b88b0bbb252fedbcf4e/examples/custom-provider/customProvider.js

when I run the Example following the readme.md instructions, i am getting the following error:
```
PS C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider> promptfoo eval
C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js:1
const fetch = require('node-fetch');
              ^
Error [ERR_REQUIRE_ESM]: require() of ES Module C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\node_modules\node-fetch\src\index.js from C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js not supported.
Instead change the require of index.js in C:\Users\XXXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js:1:15) {
  code: 'ERR_REQUIRE_ESM'
}
```

I have trying to define **`const { default: fetch } = await import('node-fetch');`** inside the CallApi function and removing the  `const fetch = require('node-fetch');` from the top, of the provider but still getting error:

```
TypeError: Cannot read properties of undefined (reading '0')

TypeError: Cannot read properties of undefined (reading '0')
    at CustomApiProvider.callApi (C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js:36:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Evaluator.runEval (C:\Users\XXXX\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:155:30)
    at async C:\Users\XXXX\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:475:25
```

I am using Node.js v20.10.0 and "node-fetch": "^3.3.2"

Do you know how to fix that? 


`node-fetch` 3.x is an esmodule and can't be imported using `require`.  The quickest fix without a code change is probably to use `node-fetch` 2.x.

I have tried to downgrade node-fetch to several 2.x versions and still having issues:

TypeError: Cannot read properties of undefined (reading '0')
TypeError: Cannot read properties of undefined (reading '0')
    at CustomApiProvider.callApi (C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js:36:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Evaluator.runEval (C:\Users\XXXX\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:155:30)
    at async C:\Users\XXXX\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:475:25


Oh, looks like that's because OpenAI has removed the `text-davinci-002` model.  I've updated the example to use `gpt-3.5-turbo`

Hi @typpo still getting same error with gpt-3.5-turbo model.
node-fetch version: 2.4.0
nodejs version: Node.js v20.10.0

`TypeError: Cannot read properties of undefined (reading '0')

TypeError: Cannot read properties of undefined (reading '0')
    at CustomApiProvider.callApi (C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\custom-provider\customProvider.js:40:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Evaluator.runEval (C:\Users\XXXX\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:155:30)
    at async C:\Users\XXXX\AppData\Roaming\npm\node_modules\promptfoo\dist\src\evaluator.js:475:25`

Which versions do you have?

Thanks a lot for your support and dedication!!






It looks like an API error, probably not a version/dependency error anymore.

Can you add a `console.log(JSON.stringify(data, null, 2))`?  That will show the API response (and error) from OpenAI.  The example should look like this:

customProvider.js:
```js
const fetch = require('node-fetch');

class CustomApiProvider {
  constructor(options) {
    // The caller may override Provider ID (e.g. when using multiple instances of the same provider)
    this.providerId = options.id || 'custom provider';

    // The config object contains any options passed to the provider in the config file.
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt) {
    const body = {
      model: 'gpt-3.5-turbo',
      messages: [{
        role: 'user',
        content: prompt,
      }],
      max_tokens: parseInt(this.config?.max_tokens, 10) || 1024,
      temperature: parseFloat(this.config?.temperature) || 0,
    };
    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      },
      body: JSON.stringify(body),
    });

    const data = await response.json();
    console.log(JSON.stringify(data, null, 2));
    const ret = {
      output: data.choices[0].message.content,
      tokenUsage: {
        total: data.usage.total_tokens,
        prompt: data.usage.prompt_tokens,
        completion: data.usage.completion_tokens,
      },
    };
    return ret;
  }
}

module.exports = CustomApiProvider;
```

Hi @typpo This is the API response
![image](https://github.com/promptfoo/promptfoo/assets/6631983/eb958c48-ab31-47b6-a767-1223e0a26224)


It looks like you may not have actually updated the code.  The change in [1aaa694](https://github.com/promptfoo/promptfoo/commit/1aaa694ef9b26e27b9323ef7ed0b8ed6aa1f1125) and the code above uses the `v1/chat/completions` endpoint, not `v1/completions`.

You are right!! now is working :-) thanks a lot!! 
![image](https://github.com/promptfoo/promptfoo/assets/6631983/ee0f2cbb-ef45-40c9-8bdb-0f5cbf37113b)


Excellent!