Repository: promptfoo/promptfoo
Issue: CustomProvider id field: MODULE_NOT_FOUND
Root cause: Documentation Deficiency

Attempting to use a custom provider with different id's to identify them and config different options. When I pass any id other than the name of the custom provider file (customProvider.js), I get an error. I think I have implemented it according to examples but can't seem to figure it out.

customProvider.js:
```
...
class CustomApiProvider {
  constructor(options) {
    // The caller may override Provider ID (e.g. when using multiple instances of the same provider)
    this.providerId = options?.id || 'custom provider';

    // The config object contains any options passed to the provider in the config file.
    this.config = options?.config;
  }
...
```

promptfooconfig.yaml:
```
...
providers:
  - customProvider.js:
    id: customProvider
    config:
      model: claude-instant-v1
      temperature: 1
      max_tokens: 1100
  - customProvider.js:
    id: customProvider2
    config:
      model: claude-v2
      temperature: 0
      max_tokens: 50
...
```

Error I am getting:
```
node:internal/modules/cjs/loader:1048
  const err = new Error(message);
              ^

Error: Cannot find module '....(removed file path)...../promptfoo/customProvider2'
Require stack:
- .........../promptfoo/dist/src/providers.js
- .........../promptfoo/dist/src/matchers.js
- .........../promptfoo/node_modules/promptfoo/dist/src/assertions.js
- .........../promptfoo/node_modules/promptfoo/dist/src/main.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1048:15)
    at Module._load (node:internal/modules/cjs/loader:901:27)
    at Module.require (node:internal/modules/cjs/loader:1115:19)
    at require (node:internal/modules/helpers:130:18)
    at .........../promptfoo/node_modules/promptfoo/dist/src/providers.js:274:132
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async loadApiProvider (.........../promptfoo/node_modules/promptfoo/dist/src/providers.js:274:32)
    at async Promise.all (index 1) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '.........../promptfoo/node_modules/promptfoo/dist/src/providers.js',
    '.........../promptfoo/node_modules/promptfoo/dist/src/matchers.js',
    '.........../promptfoo/node_modules/promptfoo/dist/src/assertions.js',
    '.........../promptfoo/node_modules/promptfoo/dist/src/main.js'
  ]
}

Node.js v20.5.1
```






Are you able to share more of your config or setup?  I'm able to run the setup in [examples/custom-provider](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider) with the following config:

```yaml
prompts: [prompts.txt]
tests: vars.csv
providers:
  - customProvider.js:
      id: custom-provider-hightemp
      config:
        temperature: 1.0
  - customProvider.js:
      id: custom-provider-lowtemp
      config:
        temperature: 0
```

Just took the custom provider example and ran it with no changes (except uncommenting the multiple providers) with no issues. So issue is within my code somewhere. 

Interestingly, the id with the same name as my file (customProvider) works because it is searching for a module named customProvider in the file directory and the customProvider.js is there. Then it is looking for a customProvider2.js or something similar and is saying it cannot be found.

Here is of my config.yaml:

```
# Load all prompt files
prompts: 
  - file://prompts/prompt*.txt

providers:
  - customProvider.js:
    id: customProvider
    config:
      model: claude-instant-v1
      temperature: 1
      max_tokens: 1100
  - customProvider.js:
    id: customProvider2
    config:
      model: claude-v2
      temperature: 0
      max_tokens: 50

tests:
  - description: Ask Irrelevant Question
    vars:
      document: file://docs/random_story_4k.txt
      question: Who is George Washington?
    assert:
      - type: javascript
        value: "output.toLowerCase().includes('cannot be answered') || output.toLowerCase().includes('does not contain')"

  - description: Answerable Question #1
    vars:
      document: file://docs/random_story_4k.txt
      question: Why does Thomas wake up so early?
    assert:
      - type: icontains
        value: oven
```

And here is more of my customProvider.js:  
```
// customProvider.js
const fetch = require('node-fetch');

class CustomApiProvider {
  constructor(options) {
    // The caller may override Provider ID (e.g. when using multiple instances of the same provider)
    this.providerId = options?.id || 'custom provider';

    // The config object contains any options passed to the provider in the config file.
    this.config = options?.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt) {
    // Initial POST request
    const initialResponse = await this.submitRequest(prompt);

    // Check initial call response is successful
    if (initialResponse.code !== '200' && initialResponse.code !== '202') {
      const errorMessage = initialResponse.message || 'Unknown error';
      throw new Error(`Initial completions call resulted in an error: ${errorMessage}`);
    }

    const jobId = initialResponse.data.job_id;

    // Polling for completion
    let result = null;
    while (result === null) {
	… Polling logic here
    }

    // Return the final result
    return {
      output: result,
      // tokenUsage can be added here if needed
    };
  }

  async submitRequest(prompt) {
    // logic of making request to another api
  }

  async checkJobStatus(jobId) {
    // logic of checking job status
  }
}

module.exports = CustomApiProvider;
```



ohh, I think it's a yaml indentation error.  Can you make sure the `id` and `config` keys are indented under `customProvider.js`?

Sorry, I know the structure is a bit wacky.  Stuck with it for backwards compatibility reasons.

Yeah, that was it. Should've been able to figure that one out but thanks for the quick help