Repository: promptfoo/promptfoo
Issue: `promptfoo eval` failing in TS project, giving "TypeError: Cannot add property prompts, object is not extensible"
Root cause: Validation Gap

**Describe the bug**
When trying to run `promptfoo eval` on a simplified project following [`node-package-typescript`](https://github.com/promptfoo/promptfoo/tree/main/examples/node-package-typescript), console errors out with:

```
$ npm test -- --verbose
2024-11-07 10:59:26-08:00

> test
> cross-env NODE_OPTIONS="--import tsx" promptfoo eval -c src/index.ts --verbose

Running migrations from C:\Users\mwiemer\llm-tests\node_modules\promptfoo\dist\drizzle...
Setting default prompt because there is no `prompts` field
Reading prompts from "file://prompts/prompt.txt"
Inserting prompt 846191a2cdb1449c2857d65d73bc96352b1ce6c19b29a63a5fab2cb54d59ac02
Inserting dataset c1d47eeb14b6f19b60468154233e9fcc982a096b20c445d9b231c6502f7c62f9
Running 1 concurrent evaluations with 4 threads...
Prompt: Write something about Hello world, vars: {"body":"Hello world"}
Eval #1 complete (1 of 1)
C:\Users\mwiemer\llm-tests\node_modules\promptfoo\dist\src\util\config\load.js:186
        ret.prompts = ['{{prompt}}'];
                    ^

TypeError: Cannot add property prompts, object is not extensible
    at readConfig (C:\Users\mwiemer\llm-tests\node_modules\promptfoo\src\util\config\load.ts:195:16)
    at async combineConfigs (C:\Users\mwiemer\llm-tests\node_modules\promptfoo\src\util\config\load.ts:223:22)
    at async resolveConfigs (C:\Users\mwiemer\llm-tests\node_modules\promptfoo\src\util\config\load.ts:420:18)
    at async runEvaluation (C:\Users\mwiemer\llm-tests\node_modules\promptfoo\src\commands\eval.ts:120:51)
    at async doEval (C:\Users\mwiemer\llm-tests\node_modules\promptfoo\src\commands\eval.ts:385:3)

Node.js v20.18.0
```

**To Reproduce**
1. Download and extract the attached zip: [llm-tests.zip](https://github.com/user-attachments/files/17668223/llm-tests.zip)
2. Follow the readme (just setting up Node, npm, and running `npm i`)
3. Monitor console output after running `npm test` or `npm test -- --verbose`
4. See error

**Expected behavior**
Results are printed to console without error

**System information (please complete the following information):**

- Promptfoo version: 0.95.0

**Additional context**
Still digging around the promptfoo source to find potential issues, will update as I learn


Maybe related to how types are constructed:

https://github.com/promptfoo/promptfoo/blob/main/src/types/index.ts#L1-L3

```ts
// Note: This file is in the process of being deconstructed into `types/` and `validators/`
// Right now Zod and pure types are mixed together!
import { z } from 'zod';
```

Many types are created via `z.foo`, but I'm not familiar with Zod at all, so I can't say any more

Currently we are working around this with a custom script:

```
// package.json scripts

"test:workaround": "tsx src/index.ts",
```

We updated the `index.ts` to throw an error if any test fails to ensure we could move it into CI/CD pipelines



Looked into this and the problem is you are passing a typescript file as a config:

```
    "test": "cross-env NODE_OPTIONS=\"--import tsx\" promptfoo eval -c src/index.ts",
```

Instead of passing code as a config, you can run the code directly with `tsx src/index.ts` or equivalent and it works!

I've improved the error message in #2114 

Perfect, we used `tsx src/index.ts` as a workaround, wasn't sure if we were missing something without calling `promptfoo`! Closing :)

Edit: not on my work PC, but you can close this one. You can see from my profile I am the same guy, not an imposter :)