Repository: promptfoo/promptfoo
Issue: file-wise `prompts` for `evaluate()` (on javascript) is unusable
Root cause: Validation Gap

It seem that the `evaluate()` did not read the corrsponding given file properly?
For examples, I ran your `examples/node-package/full-eval.js`.
The results output will be:

```json
{
      ...,
      "provider": {
        "id": "openai:gpt-3.5-turbo"
      },
      "prompt": {
        "raw": "file://prompt.txt",
        "display": "file://prompt.txt"
      },
      "vars": {
        "body": "Hello world"
      },
      "response": {
        "output": "I'm sorry, but I cannot access files on your device. How can I assist you today?",
        "tokenUsage": {
          "total": 31,
          "prompt": 11,
          "completion": 20
        },
        "cached": false,
        "cost": 0.0000355
      },
      ...
}
```

I have tried with promptfoo ver. 0.51.0 and 0.52.0.


FYR: Seems that the `evaluate()` are still unsupporting the filepaths? [index.ts#L40](https://github.com/promptfoo/promptfoo/blob/7e020fdca37b4984d76bb6888e9ef06e09969ca8/src/index.ts#L40)


Thank you for flagging this.  Yes, perhaps we should add handling for `file://` prefix.  As a workaround for now, you can load the file content in your code. 

Hey @YiFanChen99! Thanks for flagging this issue and for pointing to the specific line in the code. This has been fixed. 

## What was happening

When using the JavaScript `evaluate()` API with `file://` prompts:

```javascript
const prompts = [
  'file://prompt.txt',  // ❌ Was sent literally to the LLM
];

const results = await promptfoo.evaluate({ prompts, providers, tests });
```

The `file://prompt.txt` string was being passed directly to the model instead of reading the file content. The LLM would then respond with things like *"I cannot access files on your device"* – which, while technically accurate from the model's perspective, was definitely not what you wanted!

## The fix

In PR #994, we refactored the prompt processing pipeline to properly handle `file://` prefixes in the JavaScript API:

```typescript
// src/prompts/utils.ts - maybeFilePath() now recognizes file://
str.startsWith('file://') || ...

// src/util/index.ts - parsePathOrGlob() strips the prefix
if (promptPath.startsWith('file://')) {
  promptPath = promptPath.slice('file://'.length);
}
```

Now when you use `file://prompt.txt`, the evaluate API:
1. Recognizes it as a file reference
2. Strips the `file://` prefix
3. Resolves the path relative to your working directory
4. Reads the actual file content
5. Sends that content to the LLM ✅

## Verification

The `examples/node-package/` example now works correctly:

```javascript
// Input
const prompts = ['file://prompt.txt'];

// Output shows file content loaded properly:
{
  "raw": "Write 3 words about {{body}}",      // ✅ Actual file content
  "label": "prompt.txt: Write 3 words about {{body}}"
}
```

note, the file path is resolved relative to your current working directory when running the script. Make sure to run your evaluation script from the directory where your prompt files are located, or use absolute paths:

```javascript
// Relative (resolved from cwd)
'file://prompt.txt'

// Absolute
'file:///Users/you/project/prompts/prompt.txt'
```

Closing this as resolved. Thanks again for the detailed report – it helped us identify a gap in the JavaScript API that the CLI handled correctly!