zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Vertex AI provider returns 404 for region: global
Root cause: Algorithmic Error

**Describe the bug**

The Vertex AI provider generates an incorrect API host when `region: global` is specified, resulting in 404 errors.

The provider attempts to construct the hostname using the pattern `{region}-aiplatform.googleapis.com`, which works for regional endpoints but fails for the global region. The hostname `global-aiplatform.googleapis.com` does not exist. For the global region, the correct endpoint is `aiplatform.googleapis.com` (without any region prefix).

This issue particularly affects users who want to access preview models like Gemini 3 Pro Preview, which are often released to the global endpoint before regional availability.

**To Reproduce**

1. Create a promptfoo configuration file with `region: global`:

```yaml
providers:
- id: vertex:gemini-3-pro-preview
  config:
    region: global
    projectId: your-project-id

prompts:
- "Hello"

tests:
- vars: {}
```

2. Run the evaluation:

```bash
promptfoo eval -c config.yaml
```

3. Observe the 404 error

```
[ERROR] API call error: Error: <!DOCTYPE html>
<html lang=en>
  <title>Error 404 (Not Found)!!1</title>
```

The provider constructs an incorrect URL:
```
https://global-aiplatform.googleapis.com/v1/projects/{projectId}/locations/global/publishers/google/models/{model}:streamGenerateContent
```

The hostname `global-aiplatform.googleapis.com` does not exist.


**Expected behavior**

The provider should successfully call the Vertex AI API using the correct global endpoint (`aiplatform.googleapis.com`) and return a response from the model.

**Screenshots**

N/A


**System information:**

Promptfoo version: 0.119.10 &  also present in current main branch

**Additional context**

**Workaround**

Explicitly set the `apiHost` in the provider config:

```yaml
providers:
- id: vertex:gemini-3-pro-preview
  config:
    region: global
    apiHost: aiplatform.googleapis.com
    projectId: your-project-id
```

This workaround resolves the issue for generative models. However, the provider should automatically handle the global region case to improve user experience and avoid confusion.

The hostname pattern differs between regional and global endpoints:
Regional: `{region}-aiplatform.googleapis.com` (e.g., `us-central1-aiplatform.googleapis.com`)
Global: `aiplatform.googleapis.com` (no region prefix)

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1248/vertex-ai-provider-returns-404-for-region-global">ENG-1248</a></p>
Repository: promptfoo/promptfoo
Issue: Bug: search bar in `promptfoo view`
Root cause: Validation Gap

I've noticed a weird behaviour when you enter "**" in the search bar. The web page freezes and returns an error: `Application error: a client-side exception has occurred (see the browser console for more information).`

![image](https://github.com/promptfoo/promptfoo/assets/64668750/bdb7778c-d973-4333-bc86-bca7176f5b80)
>>


This is likely because the searchbar supports regex and `*` will match everything 🙃.  Probably a small fix

yea that's why I thought haha, thanks!

Fix released in [0.52.0](https://github.com/promptfoo/promptfoo/releases/tag/0.52.0)
Repository: promptfoo/promptfoo
Issue: An error occurs in llm-rubric if there is '\n' in the custom provider's output.
Root cause: Validation Gap

If there is '\n' in the output of the custom python provider, an error occurs in llm-rubric.

Error: Chat Completion prompt is not a valid JSON string: SyntaxError: Unexpected token in JSON at position

Is there anything else that needs to be processed in the output of the python provider?

The problem occurs when the testcase has an output property. Solved.
Repository: promptfoo/promptfoo
Issue: Feature Request: Loading data for evaluate() from data loaded into memory
Root cause: Documentation Deficiency

Hi team, 

I was reading through the docs and it seems like today promptfoo's `evaluate()` does not support the use of data that is already in memory (when running the node library). I have a use case where I would like to evaluate hundreds of `new` question answers on a daily basis through promptfoo, and I am not sure if that is something that is doable today.

Hi @anthonyivn2, can you clarify what you're looking for?  It seems you could do something like this and pass in the data you've already loaded separately:

```js
const results = await promptfoo.evaluate(
  {
    prompts: ["First prompt with {{question}} and {{answer}}', "..."],
    providers: ["openai:gpt-3.5-turbo"],
    tests: [
      {
        vars: {
          question: "...",
          answer: "...",
        },
      },
      {
        vars: {
          question: "...",
          answer: "...",
        },
      },
    ],
    writeLatestResults: true, // write results to disk so they can be viewed in web viewer
  },
  {
    maxConcurrency: 2,
  },
);
```

@typpo Ok so does evaluate() accepts different input formats other than the [TestSuiteConfiguration object](https://www.promptfoo.dev/docs/configuration/reference#testsuiteconfiguration) as specified in the [Node Package Usage doc](https://www.promptfoo.dev/docs/usage/node-package#usage)? That example you showed me above would definitely work for my use case if I can do that with evaluate().

Also I would like to confirm with you on another thing. Would `writeLatestResults: true` makes it so that I share the data (I don't want to share the data), or does it only write the result to my local disk so that its viewable in the web viewer?


Yes, looks like that type definition in the doc is a bit out of date. I'll
get it updated. Here is the actual type used in the code:
https://github.com/promptfoo/promptfoo/blob/0337f1343a3fb21df457a1ced7c76734dc32f16f/src/types.ts#L560

Also, I can confirm that writeLatestResults only writes locally.

Let me know if you have any other questions!


On Fri, Apr 12, 2024, 7:46 PM Anthony Ivan ***@***.***> wrote:

> @typpo <https://github.com/typpo> Ok so does evaluate() accepts inputs
> other than TestSuiteConfiguration object
> <https://www.promptfoo.dev/docs/configuration/reference#testsuiteconfiguration>
> as specified in the Node Package Usage doc
> <https://www.promptfoo.dev/docs/usage/node-package#usage>?
>
> To confirm, writeLatestResults: true is different from sharing the data
> right?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/665#issuecomment-2053068098>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJW5SJQ75F5CUGDEYW3Y5CMCDAVCNFSM6AAAAABGE6BDBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJTGA3DQMBZHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Ok this should work for my use case. I will close this issue, thank you for the response!
Repository: promptfoo/promptfoo
Issue: Question: Using a python or javascript prompt function when using promptfoo.evaluate()
Root cause: Documentation Deficiency

Hi, I am trying to understand on whether it is possible to use a prompt function defined within a separate python/javascript file as the `prompts` value that is passed to `promptfoo.evaluate()`. I am not seeing that functionality in the documentations, and I would like to use promptfoo.evaluate() directly instead as my test data will only be available in memory (not as local files).

You can specify a separate python/js file using a path (e.g. `file://path/to/file.py`), which should follow the [prompt functions](https://promptfoo.dev/docs/configuration/parameters#prompt-functions) documentation.

Or, when calling `evaluate` directly with node, you can supply a function directly in the `prompts` array that you pass to `evaluate`:

```js
prompts: [
      ({ vars }) => {
        // Call your own code here
        const foo = require('foo');
        finalResult = await foo(vars);
        return finalResult;
      },
    ],
```

Right now what I am seeing is that when I tried adding `file://path/to/file.js` to the prompts argument in evaluate(), the prompt used would just be the path instead of the content from the prompt function js file

ok seems like I can proceed with the second method by importing the prompt function content and then adding it to prompts.

👍 

Looks like I misremembered the prompt function/file support for the node package, it was never actually added.  I've PRed that here: https://github.com/promptfoo/promptfoo/pull/668
Repository: promptfoo/promptfoo
Issue: [Feature] OWASP Top 10 for Agentic Applications
Root cause: Unimplemented Feature Gap

It would be great to support the recently released OWASP Top 10 for Agentic Applications [1] in the red-teaming presets.

[1]: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1307/feature-owasp-top-10-for-agentic-applications">ENG-1307</a></p>

Thank you @stefanoamorelli! We'll get your PR in ASAP.
Repository: promptfoo/promptfoo
Issue: `RangeError: Invalid string length`
Root cause: Resource Mishandling

I have a bunch of tests (4,500+) and Python bespoke asserts scripts.
When I run `npx promptfoo eval -j16`, it runs all jobs in parallel, but after they have finished, I got the following error:

```
/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sqlite-core/columns/text.cjs:75
    return JSON.stringify(value);
                ^

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at SQLiteTextJson.mapToDriverValue (/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sqlite-core/columns/text.cjs:75:17)
    at /home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:157:73
    at Array.map (<anonymous>)
    at SQL.buildQueryFromSourceParams (/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:110:32)
    at /home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:129:21
    at Array.map (<anonymous>)
    at SQL.buildQueryFromSourceParams (/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:110:32)
    at /home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:132:21
    at Array.map (<anonymous>)

Node.js v19.9.0
```

 - Ran on 16GB RAM, 16GB Swap, 8 Core Machine
 - Node 19.9.0

Hmm... I reduced the number of tests to about 1k and it works fine, which seems like this error occurs when we have too many tests.

Looks like we'd have to do something like this: https://stackoverflow.com/questions/29175877/json-stringify-throws-rangeerror-invalid-string-length-for-huge-objects

Hey @sangwoo-joh, thanks for your patience and for bringing this up! We’ve improved the data model in [0.92.0](https://github.com/promptfoo/promptfoo/releases/tag/0.92.0), which should help Promptfoo handle larger datasets more smoothly. Please give the new version a try when you have a chance. I’ll go ahead and close this issue, but if anything still isn’t quite right, feel free to reopen it.
Repository: promptfoo/promptfoo
Issue: Feature Request: Improve the UX of the red team target type multi-select

When selecting a target type for a red team configuration, the available targets aren't immediately clear to the user. You need to press the "✏️ Change" button to get into a new view that lets you filter down and select a target. Upon selecting a target, you are moved into a separate view that hides other options. 

From a "UX" way of thinking, I think this violates progressive disclosure because of unnecessary abstraction. Additionally, it's an extra click on a page that already has sufficiently minimal information (just two related forms and navigation). 

I suggest a view that **always** surfaces the target type options by removing a separate view. 

Current view example:

<img width="1367" height="676" alt="Image" src="https://github.com/user-attachments/assets/61050047-1b3f-4274-ad23-fc60ebf7cfa2" />




<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1318/feature-request-improve-the-ux-of-the-red-team-target-type-multi">ENG-1318</a></p>
Repository: promptfoo/promptfoo
Issue: Bug: "Custom provider" selection for a red team config isn't persisted when changing tabs
Root cause: Validation Gap

**Describe the bug**
The UI Element for target type selection disappears when navigating to/from the "target type" tab in a red team configuration. I've experienced this in a dev server with `npm run dev`. Haven't attempted to recreate in prod 

**To Reproduce**

Select "Custom provider" for a new red team config
<img width="1116" height="645" alt="Image" src="https://github.com/user-attachments/assets/7a2c3bb4-02c9-41c0-b019-38b75ed7d03d" />

Navigate back to the "target type" tab from another tab
<img width="1118" height="521" alt="Image" src="https://github.com/user-attachments/assets/073305b0-1231-48af-a963-5b88e82880f3" />

Notice the target type selector is missing

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1319/bug-custom-provider-selection-for-a-red-team-config-isnt-persisted">ENG-1319</a></p>
Repository: promptfoo/promptfoo
Issue: Is custom Python/Javascript assertion available through the evaluate() node module?
Root cause: Documentation Deficiency

Related to the question related to PR #668. I would like to be able to define custom assertions in Python/Javascript and then use it with the `evaluate()` node module

Yes, both of these assertion types are supported via `promptfoo.evaluate`.  

In the case of `javascript` asserts, you can supply a function directly as the assertion `value`.  I notice an inconsistency in how functions are called between external js files/inline js functions vs functions directly provided via `evaluate`, which will be addressed in #674 

Ok noted, thank you!
Repository: promptfoo/promptfoo
Issue: Feature Request: Add action buttons (or links) to the red team target type selector description

The description for the red team target type selector reads,

"Select the type that best matches your target. Don't see what you need? Try 'Custom Target' to access [more providers](https://www.promptfoo.dev/docs/providers/). You can also create your own using [Python](https://www.promptfoo.dev/docs/providers/python/), [JavaScript](https://www.promptfoo.dev/docs/providers/custom-api/), or [shell scripts](https://www.promptfoo.dev/docs/providers/custom-script/)."

 The text, `Custom Target` could be converted to button (to differentiate it from a link) that selects the "Custom Provider" target type when pressed.

<img width="1125" height="253" alt="Image" src="https://github.com/user-attachments/assets/e613fe56-0767-49f1-990d-27b2d62f40ac" />

The copy may be out of date between the description name (Custom Target) and the provider name (Custom Provider).

The same could be done for the custom language and shell scripts in place of links, since the ❓ icon on those target types already links to documentation. 

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1320/feature-request-add-action-buttons-or-links-to-the-red-team-target">ENG-1320</a></p>
Repository: promptfoo/promptfoo
Issue: Citation strategy crashes    with TypeError - cloud API validation error
Root cause: Validation Gap

**Describe the bug**
The `citation` strategy fails with `TypeError: Cannot read properties of undefined (reading 'citation')` because the promptfoo cloud API returns a validation error instead of the expected citation data. The client code doesn't check for error responses before accessing `data.result.citation`.

**To Reproduce**
Steps to reproduce the behavior:

1. Create a redteam config with `citation` strategy:
```yaml
strategies:
  - citation
```
2. Run `promptfoo redteam generate -c your-config.yaml`
3. Check debug logs at `~/.promptfoo/logs/promptfoo-debug-*.log`
4. See error: `Error in remote citation generation: TypeError: Cannot read properties of undefined (reading 'citation')`

**Expected behavior**
Citation strategy should generate test cases with academic citation framing, transforming prompts like:
```
Original: "How do I hack into a computer?"
Expected: "Based on the following Journal Article, explain how do I hack into a computer? Reference: 1. Smith, J..."
```

**Screenshots**
Debug log output showing the validation error:
```
Got remote citation generation result for case 5: {"error":"Validation error: Required at \"result.topic\"; Required at \"result.key\"; Required at \"result.citation\""}
```

Error log showing the crash:
```
Error in remote citation generation: TypeError: Cannot read properties of undefined (reading 'citation')
```

**System information:**
- Promptfoo version: 0.119.4 (latest main branch, commit 88a67241)
- Node.js: 20+

**Additional context**
Root cause analysis:

1. The client payload in `src/redteam/strategies/citation.ts` (lines 41-48):
```typescript
const payload = {
  task: 'citation',
  testCases: [testCase],
  injectVar,
  topic: testCase.vars[injectVar],
  config,
  email: getUserEmail(),
};
```

2. The cloud API at `https://api.promptfoo.app/api/v1/task` returns a validation error indicating a payload/contract mismatch.

3. The client code at line 73 then tries to access `data.result.citation.type` without checking for errors first.

4. Compare with GCG strategy (`src/redteam/strategies/gcg.ts` lines 65-69) which has proper error handling:
```typescript
if (data.error) {
  logger.error(`[GCG] Error in GCG generation: ${data.error}`);
  logger.debug(`[GCG] Response: ${JSON.stringify(data)}`);
  return;
}
```

**Workaround**
Remove `citation` from strategies list until fixed:
```yaml
strategies:
  - basic
  - jailbreak:composite
  # - citation  # BROKEN - cloud API validation error
```


<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1321/citation-strategy-crashes-with-typeerror-cloud-api-validation-error">ENG-1321</a></p>

Hi @MrMoshkovitz, thanks for reporting. I was unable to reproduce using the following config on the latest version (0.120.5):

```yaml
providers:
  - id: openai:gpt-5-mini
redteam:
  plugins:
    - harmful:hate
  strategies:
    - citation
```

Can you share your config? Also pull the latest version and let me know whether you're still encountering this issue; thanks.
Repository: promptfoo/promptfoo
Issue: How to set a project path for promptfoo

Because now after starting promptfoo, the access path is: http://ip:15500/eval
I want to add "myProject" before the access path of the project：http://ip:15500/**myproject**/eval

promptfoo， Is there such an environment variable?

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1326/how-to-set-a-project-path-for-promptfoo">ENG-1326</a></p>

<!-- This is an auto-generated issue plan by CodeRabbit -->


## 📝 CodeRabbit Plan Mode
Generate an implementation plan and prompts that you can use with your favorite coding agent.

- [ ] <!-- {"checkboxId": "8d4f2b9c-3e1a-4f7c-a9b2-d5e8f1c4a7b9"} --> Create Plan

<details>
<summary>Examples</summary>

- [Example 1](https://github.com/coderabbitai/git-worktree-runner/issues/29#issuecomment-3589134556)
- [Example 2](https://github.com/coderabbitai/git-worktree-runner/issues/12#issuecomment-3606665167)

</details>
<sub>You can configure auto-planning by selecting labels in the issue_enrichment configuration.</sub>

## 🔗 Similar Issues
**Related Issues**
- https://github.com/promptfoo/promptfoo/issues/4612
- https://github.com/promptfoo/promptfoo/issues/3724
- https://github.com/promptfoo/promptfoo/issues/4648
- https://github.com/promptfoo/promptfoo/issues/2785
- https://github.com/promptfoo/promptfoo/issues/6206
- https://github.com/promptfoo/promptfoo/issues/4521
- https://github.com/promptfoo/promptfoo/issues/4002


## 👤 Suggested Assignees
- [loongst](https://github.com/loongst)
- [Draqun](https://github.com/Draqun)
- [wk1583241477](https://github.com/wk1583241477)
- [funmise](https://github.com/funmise)
- [khatwaniNikhil](https://github.com/khatwaniNikhil)


---
<details>
<summary> 🧪 Issue enrichment is currently in open beta.</summary>

To disable automatic issue enrichment, add the following to your `.coderabbit.yaml`:
```yaml
issue_enrichment:
  auto_enrich:
    enabled: false
```
</details>

💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit) or [schedule a call](https://calendly.com/parth-coderabbit/issues-feedback)!

@wk1583241477 please have a look at #6758 and give it a try.
Repository: promptfoo/promptfoo
Issue: Unable to share result, is the API server down?

In my local laptop, 

`promptfoo eval` works but `promptfoo eval --share` yields the following error:

```
/Users/chaofuyang/.nvm/versions/node/v20.10.0/lib/node_modules/promptfoo/node_modules/node-fetch/lib/index.js:273
				return Body.Promise.reject(new FetchError(`invalid json response body at ${_this2.url} reason: ${err.message}`, 'invalid-json'));
				                           ^
FetchError: invalid json response body at https://api.promptfoo.dev/api/eval reason: Unexpected token 'I', "Internal s"... is not valid JSON
    at /Users/chaofuyang/.nvm/versions/node/v20.10.0/lib/node_modules/promptfoo/node_modules/node-fetch/lib/index.js:273:32
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async createShareableUrl (/Users/chaofuyang/.nvm/versions/node/v20.10.0/lib/node_modules/promptfoo/dist/src/share.js:24:27)
    at async runEvaluation (/Users/chaofuyang/.nvm/versions/node/v20.10.0/lib/node_modules/promptfoo/dist/src/main.js:451:67)
    at async Command.<anonymous> (/Users/chaofuyang/.nvm/versions/node/v20.10.0/lib/node_modules/promptfoo/dist/src/main.js:559:9) {
  type: 'invalid-json'
}
```

Looks like the API server returns 500 Internal server error

This has been fixed!  Sorry about that.
Repository: promptfoo/promptfoo
Issue: HTTP connection headers UI issue

**Describe the bug**
After clicking "test endpoint" the UI doesn't display the header values correctly anymore

**To Reproduce**
add a bearer token (or something else long) to a value in the header

Click test endpoint

**Expected behavior**
Header values would show within the view

**Screenshots**
<img width="1647" height="335" alt="Image" src="https://github.com/user-attachments/assets/6fbdd295-ee1a-4aa0-af8e-2286a1351bbe" />

**System information:**

Windows - Chrome

- Promptfoo version:
[Version 0.120.8]


<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1350/http-connection-headers-ui-issue">ENG-1350</a></p>

<!-- This is an auto-generated issue plan by CodeRabbit -->


### 📝 CodeRabbit Plan Mode
Generate an implementation plan and prompts that you can use with your favorite coding agent.

- [ ] <!-- {"checkboxId": "8d4f2b9c-3e1a-4f7c-a9b2-d5e8f1c4a7b9"} --> Create Plan

<details>
<summary>Examples</summary>

- [Example 1](https://github.com/coderabbitai/git-worktree-runner/issues/29#issuecomment-3589134556)
- [Example 2](https://github.com/coderabbitai/git-worktree-runner/issues/12#issuecomment-3606665167)

</details>

---

<details>
<summary><b>🔗 Similar Issues</b></summary>

**Related Issues**
- https://github.com/promptfoo/promptfoo/issues/2537
- https://github.com/promptfoo/promptfoo/issues/3797
- https://github.com/promptfoo/promptfoo/issues/2948
- https://github.com/promptfoo/promptfoo/issues/4489
- https://github.com/promptfoo/promptfoo/issues/5074
</details>
<details>
<summary><b>👤 Suggested Assignees</b></summary>

- [kuatroka](https://github.com/kuatroka)
- [arsena](https://github.com/arsena)
- [mkrudele](https://github.com/mkrudele)
- [rafsid](https://github.com/rafsid)
- [gelim](https://github.com/gelim)
</details>


---
<details>
<summary> 🧪 Issue enrichment is currently in open beta.</summary>


You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your `.coderabbit.yaml`:
```yaml
issue_enrichment:
  auto_enrich:
    enabled: false
```
</details>

💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit) or [schedule a call](https://calendly.com/parth-coderabbit/issues-feedback)!