Repository: promptfoo/promptfoo
Issue: Relative path resolution fails for file providers
Root cause: Algorithmic Error

**Describe the bug**

Promptfoo eval is not able to resolve the path for the providers relatively to the `promptfooconfig.yaml` file location in the latest versions

**To Reproduce**

Promptfoo version:

```console
❯ promptfoo --version
(node:53198) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
0.112.5
```

Project structure:

```console
❯ tree -L 2
.
├── evaluation
│   └── promptfooconfig.yaml
└── providers
    └── example.py
```

`evaluation/promptfooconfig.yaml`:

```yaml
providers:
  - id: 'file://providers/example.py'
    label: 'Debug 0 level'
  - id: 'file://../providers/example.py'
    label: 'Debug 1 level'
  - id: 'file://../../providers/example.py'
    label: 'Debug 2 levels'
  - id: 'file://../../../providers/example.py'
    label: 'Debug 3 levels'

prompts:
  - "Who are you?"

tests:
  - assert:
    - type: icontains
      value: "?"
```

`providers/example.py`:

```python
from typing import Any


def call_api(prompt: str, options: dict[str, Any], context: dict[str, Any]):
    def call_llm(prompt):
        return f"Stub response for prompt: {prompt}"

    output = call_llm(prompt)

    result = {
        "output": output,
    }

    return result
```

Running `promptfoo eval` command gives several errors for like: `ERROR] Error: ENOENT: no such file or directory`. Interestingly, these are the files resolved for each level:

- Level 0: `file://providers/example.py`
    - `/tmp/promptfoo-debug/evaluation/providers/example.py`
    - ✅ This the expected
- Level 1: `file://../providers/example.py`
    - `/tmp/promptfoo-debug/evaluation/providers/example.py`
    - ❌ The same resolution as the first one
- Level 2: `file://../../providers/example.py`
    - `/tmp/promptfoo-debug/providers/example.py`
    - ❓ This passes the test and can be resolved, but it is not expected
- Level 3: `file://../../../providers/example.py`
    - `/tmp/providers/example.py'`
    - ❓ The behavior is kept consistent from the previous one, but again it is not expected

**Expected behavior**
A clear and concise description of what you expected to happen.

**System information:**

```console
❯ promptfoo debug --config evaluation
...
{
  "version": "0.112.5",
  "platform": {
    "os": "darwin",
    "release": "24.4.0",
    "arch": "arm64",
    "nodeVersion": "v22.13.1"
  },
  "env": {},
  "configInfo": {
    "defaultConfigPath": "/Users/pablosanjose/projects/tmp/promptfoo-debug/promptfooconfig.yaml",
    "specifiedConfigPath": "evaluation",
    "configExists": true,
    "configContent": "Error loading config: Error: Unsupported configuration file format: "
  }
}
```

**Additional context**

We had this setup working in the past. We have only started seeing it after upgrading to the latest versions because of this issue: https://github.com/promptfoo/promptfoo/issues/3929


I believe I’ve found the root cause, and it looks like it stems from a regression introduced in [this PR](https://github.com/promptfoo/promptfoo/pull/3805/files#diff-fb7056fba08f9efbf1c44143995a07761c20216709057454b0afab52c781b0fdR74).

### Root Cause: Path Resolution Happens Twice
It appears that the basePath is being applied twice when resolving file:// provider paths, which leads to incorrect or inconsistent file lookups.

Here's a step-by-step breakdown of what’s happening:

---

#### ✅ Step 1: Initial Resolution (Correct)
In [file.ts](https://github.com/promptfoo/promptfoo/blob/main/src/util/file.ts#L99), the path is first resolved using:

```ts
return path.join(basePath, filePath);
```

Example:
* `basePath = "evaluation"`
* `filePath = "providers/example.py"`

➡️ Result: evaluation/providers/example.py
✅ This is correct and expected.

---

#### ❌ Step 2: Second Resolution (Incorrect)
Later, in [index.ts](https://github.com/promptfoo/promptfoo/blob/main/src/util/index.ts#L415), the resolved path is again passed through:

```ts
const filePath = path.resolve(basePath, promptPath);
```

But at this point:
* basePath = "evaluation"
* promptPath = "evaluation/providers/example.py" (already resolved once)

➡️ Final result: evaluation/evaluation/providers/example.py
❌ This is incorrect — basePath is effectively being applied twice.

---

### 🔍 Impact on this setup
In the example:

```yaml
providers:
  - id: 'file://providers/example.py'        # ✅ Resolves correctly
  - id: 'file://../providers/example.py'     # ❌ Incorrectly re-resolves to same as above
  - id: 'file://../../providers/example.py'  # ❓ Works, but unintentionally
  - id: 'file://../../../providers/example.py' # ❓ Same behavior continues
```

Because of the double-resolution logic, relative paths like ../ and ../../ are effectively being flattened or misapplied, leading to inconsistent and misleading file loads.

Hope this helps pin own the issue.