zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Error when running DALL-E3 red team example
Root cause: Validation Gap

**Describe the bug**
When initialising and running the DALL-E3 example highlighted in the [blog post](https://www.promptfoo.dev/blog/jailbreak-dalle/#try-it-yourself), I am greeted with the following error:

```
Running migrations from C:\Users\foobar\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\drizzle...
Rewriting config.targets to config.providers
Running migrations from C:\Users\foobar\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\drizzle...
Reading prompts from ["Generate an image: {{query}}"]
C:\Users\foobar\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14
    throw new Error(value);
          ^

Error: Invariant failed: Expected injectVar to be set
```

**To Reproduce**
Follow the steps in the blog post as is. In my case, I decided to run it using the [example config](https://github.com/promptfoo/promptfoo/blob/main/examples/redteam-dalle/promptfooconfig.yaml).

**Expected behavior**
I expected the jailbreaking process to start.

**System information (please complete the following information):**

- Promptfoo version: 0.94.4


Thank you very much, this did indeed fix the issue.
Repository: promptfoo/promptfoo
Issue: Request: adding logged messages to `--output file.xyz`
Root cause: Unimplemented Feature Gap

It would be nice if `--output file.xyz` contained any logs made.  For example, they could be stored under a `logs` key.

This can also make it easier to report bugs, just provide an output file, and it will include debugging info.

Thanks in advance as always!

Going to close this as I recommend redirecting stdout and stderr to file, e.g.:
```
LOG_LEVEL=debug promptfoo eval > output.txt 2>&1
```
Repository: promptfoo/promptfoo
Issue: How can i use ollama to test  my prompts?
Root cause: Documentation Deficiency

I tried two methods with Ollama, but neither worked.

![image](https://github.com/user-attachments/assets/ceecfa50-18ca-4dd4-9bfd-906d230e7be3)
![image](https://github.com/user-attachments/assets/68524ab7-4a83-4e52-ab12-907736144a4c)


Error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add apiKey to the provider config.

Can you provide the full config?  I'm guessing you're using a model-graded assertion and you need to [override the provider](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader).

It works for me . Thanks

> Can you provide the full config? I'm guessing you're using a model-graded assertion and you need to [override the provider](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader).↳
Repository: promptfoo/promptfoo
Issue: problem with running multiple tests
Root cause: Unimplemented Feature Gap

I was running two tests roughly in the following manner (I will illustrate using a notebook example):

```
%%writefile prompts/prompt1.txt
{{document}}
Question: {{question}}
```

```
%%writefile promptfooconfig1yaml
prompts:
  - prompts/prompt1.txt
providers: 
  - id: openai:gpt-3.5-turbo
    config:
      temperature: 0
tests: 
  - tests/test1.yaml
outputPath: output/output1.csv
```

And the second one similarly:

```
%%writefile prompts/prompt2.txt
Question: {{question}}
Document: {{document}}
```

```
%%writefile promptfooconfig2.yaml
prompts:
  - prompts/prompt2.txt
providers: 
  - id: openai:gpt-3.5-turbo
    config:
      temperature: 0
tests: 
  - tests/test2.yaml
outputPath: output/output2.csv
```


These were two completely different tests with different logic. However, I would like to implement this solution in a way that they run cyclically. To achieve this, I would like to somehow combine them so that they work and can be executed with a single command, like npx promptfoo view.



I think https://github.com/promptfoo/promptfoo/issues/170 is related to this, sort of similar concepts

This is not exactly what I have in mind. Here's how it looks: I have two separate tests - two separate test files with different assertions and two separate prompts. So, two completely separate tests with different assertions, prompts, and so on. I want to deploy this, and I would like these two tests to be accessible under a single link - to be executed with a single command like "npx promptfoo view".

Maybe you should manually combine the test results
________________________________
发件人: tttbone8122 ***@***.***>
发送时间: Thursday, September 28, 2023 2:15:57 PM
收件人: promptfoo/promptfoo ***@***.***>
抄送: Subscribed ***@***.***>
主题: Re: [promptfoo/promptfoo] problem with running multiple tests (Issue #203)


This is not exactly what I have in mind. Here's how it looks: I have two separate tests - two separate test files with different assertions and two separate prompts. So, two completely separate tests with different assertions, prompts, and so on. I want to deploy this, and I would like these two tests to be accessible under a single link - to be executed with a single command like "npx promptfoo view".

―
Reply to this email directly, view it on GitHub<https://github.com/promptfoo/promptfoo/issues/203#issuecomment-1738532574>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AANFQPY5M6LFPFTQZDZLQ2TX4UIZ3ANCNFSM6AAAAAA5JGGKBU>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
Repository: promptfoo/promptfoo
Issue: 0.24.3 git tag missing?
Root cause: Validation Gap

When I run `npm run update`, it doesn't update `promptfoo`'s version to 0.24.3.

If you're already aware of this @typpo feel free to ignore or just close this out.  I think a `git tag` for 0.24.3 needs to be made for the release to work
Repository: promptfoo/promptfoo
Issue: Respect max text length table setting when rendering cells as Markdown
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

We are giving some values as Markdown links to provide some handy links to our app. This could be like:

```
[123](http://localhost:3000/link/to/view/123)
```

However, when we do this, other table inputs do not respect the table "Max text length" value. We have some pretty big cells, so the browser tab gets bogged down and is somewhat unresponsive.

**Describe the solution you'd like**

I think we could use the truncated text as well here:

```tsx
// src/app/src/pages/eval/components/ResultsTable.tsx
400:                    {renderMarkdown ? (
401:                      <ReactMarkdown remarkPlugins={[remarkGfm]}>{value}</ReactMarkdown>  // << s/value/something else/
402-                    ) : (
403:                      <TruncatedText text={value} maxLength={maxTextLength} />
404-                    )}
405-                  </div>
```

**Describe alternatives you've considered**
N/A

**Additional context**

If you want me to take a crack at this, I can!

Hi @panozzaj, your proposed solution sounds great, and we’d definitely welcome a PR on this. Feel free to go ahead, and if you have any questions or need guidance, don’t hesitate to reach out. Thanks again and looking forward to your contribution!

Thanks so much for your help in shipping this @panozzaj! Please feel free to reach out if you have any other feature requests, product ideas, or feedback.
Repository: promptfoo/promptfoo
Issue: Add link for thumbs up / thumbs down
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

In the Promptfoo UI there are 👍🏻 / 👎🏻  buttons for each test case.

Right now I believe the buttons change the example results to good / bad, and so update the percentages for that eval. But I believe it doesn't persist so there is no way to have this work for a subsequent test run (could be wrong on these.)

I'm wondering if we could enable those buttons do some custom action.

**Describe the solution you'd like**

The workflow I'm envisioning is:

 - when we set up our test cases (we're doing this semi-programmatically), we can say what to do when the thumbs up / thumbs down buttons are pressed
   - this could for example be a link like https://example.com/item/123/mark_current_scores_as_incorrect and then our app could handle this in some way (treating as a `POST` could be nice, but I think we could make it a `GET`)
 - then when we see that the LLM actually has the correct scoring and our ground truth is incorrect, we could easily update that by pressing the thumbs up button. This would then do something in our system (updating the ground truth, making a data migration, etc.)
 - I'm not sure we have a good use case for thumbs down yet, but it would be a similar process

(To prevent unauthorized changing of data, the link could be to an endpoint that only works in dev, or we could provide a long auth string or something.)

**Describe alternatives you've considered**

URLs seem like a fairly standard / compatible with many frameworks approach, and I could see others wanting to update their data. Possible alternatives / hacks:

 - Could use a Tampermonkey script or something to link you to somewhere.
 - Query the Promptfoo database for thumbs-up / thumbs-down events and update our database accordingly.

**Additional context**

Thanks!
Repository: promptfoo/promptfoo
Issue: Error: Invariant failed: resp.output must be defined
Root cause: Validation Gap

When I tested using the default dataset sample code, it produced Error: Invariant failed: resp.output must be defined

I haven't found a corresponding solution in the issues. What is this problem?

@libukai can you please provide exact steps to reproduce? What command did you run? Which promptfoo version are you on? Thanks! 

After adding the --provider parameter, the generate command worked successfully.

You said the generation use the "grading provider" and what's the "grading provider"?

I found a responsible issue  #1158 
Can I think the "grading provider" is the default gpt-4o of OpenAI 

Can you share the full stack trace included with the error?
Repository: promptfoo/promptfoo
Issue: Support Transformers.js
Root cause: Unimplemented Feature Gap

Would you consider supporting [Transformers.js](https://github.com/huggingface/transformers.js) as an additional provider? This library provides access to text generation and content classification functionality using many different models. With the new V3 the support grew even further. In addition, it might be the most convenient way to create text embeddings locally since it would run inside the node application and hence no external server needs to be started. 

Hi @jens-ghc, yes, this is something we would be interested in! Would you be open to contributing to the integration? I'd be happy to help on a call. For what it's worth, we also currently have a [Hugging Face integration](https://www.promptfoo.dev/docs/providers/huggingface/) but would love to revisit and improve it alongside a potential transformers.js integration.

Hi @mldangelo Let me look into it. At least the embeddings-part might be straight forward. One question: some embeddings models require a "prefix" to be set in order for them to work properly, is this something that promptfoo supports through configuration files?

Yes! Take a look at https://github.com/promptfoo/promptfoo/blob/main/src/providers.ts (the entrypoint) and then any of our specific provider implementations https://github.com/promptfoo/promptfoo/tree/main/src/providers.
Repository: promptfoo/promptfoo
Issue: Not clear how red teaming iterative provider is supposed to function
Root cause: Validation Gap

Hello! I'm trying to use the redteam:iterative provider, I have a test case which uses a static system prompt (no variables for nunchuck to inject into) and as such the context vars are used to call a python target which then invokes the llm chain.

However, doing so doesn't seem to work as expected compared with other test types, specifically the `call_api` in my python evaluation gets a `None` context and so is never able to run the llm chain.

When looking into the source, I eventually found this line:

https://github.com/promptfoo/promptfoo/blob/b5ed0606bac81a0670dee73bef9f59917be0a9f1/src/redteam/providers/iterative.ts#L121

Here the underlying target is called, which is a wrapper around `callApi` as shown here:

https://github.com/promptfoo/promptfoo/blob/b5ed0606bac81a0670dee73bef9f59917be0a9f1/src/redteam/providers/shared.ts#L53-L75

However, it is clear here that context is never provided to the `callApi` which explains the issue.

Is it expected with iterative provider that context cannot be used? and that the only method is through prompt injection?

Thanks!

Thanks for flagging this!  #2090 should fix

> Thanks for flagging this! #2090 should fix

Ah amazing! Yes indeed it does, thanks a ton!
Repository: promptfoo/promptfoo
Issue: Can I run a GitHub action similarly on GitLab?

I'm wondering if I can run what's here: https://www.promptfoo.dev/docs/integrations/github-action

But on GitLab?

I think the answer out-of-the-box is no, but one can make this work using Docker ([read here](https://tomasvotruba.com/blog/how-can-we-use-github-actions-in-gitlab))

James is correct - it won't work out of the box on GitLab.  I've just added an example of a standalone github action here: https://github.com/promptfoo/promptfoo/blob/main/examples/github-action/standalone-action.yaml

This doesn't rely on the packaged action, and will be easier to convert to GitLab CI.  I'm guessing GPT-4 could do this with some degree of accuracy, since the formats are pretty similar.
Repository: promptfoo/promptfoo
Issue: Javascript assertion problem
Root cause: Algorithmic Error

![image](https://github.com/promptfoo/promptfoo/assets/1726527/c16bd978-846d-4ed5-87af-1cb830455aae)

The score should be zero, right?

You're right.  #208 should fix, and I've also added a test for this case.
Repository: promptfoo/promptfoo
Issue: When using portkey as a provider, required request headers are not being set correctly
Root cause: Validation Gap

**Describe the bug**
Two issues with Portkey provider integration:
1. PORTKEY_API_KEY environment variable is not being read automatically
2. Header key formatting is incorrect for Portkey API key (generates 'x-portkey-apiKey' instead of 'x-portkey-api-key')

**To Reproduce**
Bug 1 - Environment Variable Not Read:
1. Set up PORTKEY_API_KEY in environment:
```bash
export PORTKEY_API_KEY=your_api_key
```

2. Create promptfoo config:
```yaml
prompts:
  - 'portkey://pp-task-gener-1e0c8d'

providers:
  - id: portkey:gpt-4o
    config:
      portkeyProvider: openai

tests: file://tests/sample-cases.csv
```

3. Run evaluation:
```bash
promptfoo eval
```

4. See error about missing Portkey API key

Bug 2 - Incorrect Header Formatting:
1. Create promptfoo config with explicit API key:
```yaml
prompts:
  - 'portkey://pp-task-gener-1e0c8d'

providers:
  - id: portkey:gpt-4o
    config:
      portkeyProvider: openai
      portkeyApiKey: your_api_key

tests: file://tests/sample-cases.csv
```

2. Run evaluation and inspect network requests:
```bash
promptfoo eval
```

3. Observe header is formatted as 'x-portkey-apiKey' instead of 'x-portkey-api-key'

**Expected behavior**
1. The Portkey provider should automatically read PORTKEY_API_KEY from environment variables when no API key is provided in config
2. When portkeyApiKey is provided in config, it should be transformed to 'x-portkey-api-key' in headers

**System information:**
- Promptfoo version: 0.94.5

**Additional context**
In src/providers/portkey.ts, there are two distinct issues:

1. Environment Variable Issue:
The constructor doesn't check for the PORTKEY_API_KEY environment variable. Unlike other providers, there's no fallback mechanism to read from environment variables when the config doesn't contain an API key.

2. Header Formatting Issue:
The header transformation logic has a bug in the string manipulation:
```typescript
const headerKey = key.startsWith('portkey')
  ? `x-${key.replace(/^portkey/, 'portkey-').toLowerCase()}`
  : key;
```
This transforms 'portkeyApiKey' to 'x-portkey-apiKey', preserving the camelCase. The Portkey API expects 'x-portkey-api-key' format. The toLowerCase() is applied after the initial replacement, but doesn't handle the camelCase conversion correctly.

Reference to correct header format can be found in Portkey's official documentation:
https://portkey.ai/docs/api-reference/inference-api/portkey-sdk-client#parameters (check "REST Headers" tab)

The documentation specifically shows the header should be 'x-portkey-api-key' in their API reference.

@typpo does it look like something that needs to be prioritised? I can go ahead and create a PR. 

Hi @rajatvijay, have a fix in #2088.  If possible, could you give it a try?

Released in 0.96.0!
Repository: promptfoo/promptfoo
Issue: Support for AWS Bedrock Agents
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

I see in the [documentation](https://www.promptfoo.dev/docs/providers/aws-bedrock/) that AWS Bedrock models can be tested through promptfoo. It does not look that Bedrock Agents can be tested as well. Any thoughts on dealing with Agents in promptfoo?

**Describe the solution you'd like**

Add Bedrock Agents as a promptfoo provider.

**Describe alternatives you've considered**

The alternative might be to not use promptfoo or to implement a [Custom Python](https://www.promptfoo.dev/docs/providers/python/) provider that interacts with AWS Bedrock Agents.


Hi @rvernica,

Sorry for not closing this sooner - AWS Bedrock Agents support was added in **PR #5709** on September 23, 2025, released in version 0.118.9.

You can now use Bedrock Agents with this provider format:

```yaml
providers:
  - bedrock-agent:your-agent-id
    config:
      agentAliasId: your-alias-id
```

The feature includes full UI support with configuration dialogs, validation for required fields, and automatic YAML configuration updates.

Please pull the latest version and give it a try with your Bedrock Agents. If you encounter any issues, feel free to comment or re-open.

See the [AWS Bedrock Agents documentation](https://www.promptfoo.dev/docs/providers/bedrock-agents) for comprehensive configuration options and examples.

Thanks for the feature request!
Repository: promptfoo/promptfoo
Issue: Support AWS Profiles and/or Session Tokens with AWS providers
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

Currently, the only AWS authentication supported is via access key and secret access key. More complex setups use SSO profiles or require a Session Token. Neither is supported by promptfoo.

**Describe the solution you'd like**

Support AWS Profiles and/or AWS Session Tokens for authentication.

**Describe alternatives you've considered**

Do not use promptfoo or use a [Custom Python](https://www.promptfoo.dev/docs/providers/python/) provider that does its own AWS authentication.


Released in 0.96.0!
Repository: promptfoo/promptfoo
Issue: Specifying a grader causes assertions to fail with the error message "Could not extract JSON from llm-rubric response"
Root cause: Configuration Error

**Describe the bug**
When a provider is explicitly set on `defaultTest` or through the command using `--grader`, some random assertions are failing with the error message "Could not extract JSON from llm-rubric response".  If we don't pin a grader and let promptfoo use the default one, the problem doesn't occur.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Either specify a model in the cmd through `--grader` or in the config through
```
defaultTest:
  options:
    provider: openai:gpt-4
```
2. Run promptfoo eval
3. Check the results and look for "Could not extract JSON from llm-rubric response"


**Expected behavior**
We expect not to see random assertion failures due to this bug.

**Screenshots**
If applicable, add screenshots to help explain your problem.


**System information (please complete the following information):**

- Promptfoo version: 0.95.0

**Additional context**
Add any other context about the problem here.


Hi @mouhcinetao, thank you for reporting this issue! To help diagnose, please try running `promptfoo eval --verbose` and check the grading prompt responses. This may give more insight into where the JSON extraction is failing. You could also try enforcing JSON output mode with your grading provider override. Here’s an example `promptfooconfig.yaml`:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - "{{prompt}}"

providers:
  - echo
defaultTest:
  options:
    provider: openai:gpt-4
    config:
      response_format: json_object
tests:
  - vars:
      prompt: Hello!
    assert:
      - type: llm-rubric
        value: Is not a greeting
      - type: llm-rubric
        value: Is a greeting
```

If, after reviewing your grader output, you'd like to discuss further, feel free to book a 15-minute slot on my calendar [here](https://cal.com/michael-dangelo/15min). Let me know how it goes!

The root cause here seems to be that the model you're using to override the grader isn't producing valid JSON.  Which model are you using?

@mouhcinetao Were you able to resolve your issue?
Repository: promptfoo/promptfoo
Issue: Cannot set `response_format` with external file for Azure OpenAI provider
Root cause: Configuration Error

**Describe the bug**

For Azure OpenAI provider, when setting the `response_format` config setting linking to an external file like so;

```
providers:
  - id: azureopenai:chat:xxx-gpt4o
    label: gpt-4o
    config:
      apiHost: "xxx.openai.azure.com"
      apiVersion: 2024-08-01-preview
      response_format: file://schema.json
```

I get the following error;

```
[FAIL] API response error: null 400: There was an error parsing the body 
```

It does not even matter what path I set, it could be to an non-existant file and still reproduce the same error message.

But adding the response format manually such as;

```
      response_format:
        type: json_schema
        json_schema:
          name: reasoning_schema
          strict: true
          schema:
            type: object
            properties:
              reasoning_steps:
                type: array
                items:
                  type: string
                description: The reasoning steps leading to the final conclusion.
              answer:
                type: string
                description: The final answer, taking into account the reasoning steps.
            required:
              - reasoning_steps
              - answer
            additionalProperties: false
```

works like a charm, as long as I remember to set the latest API version (`apiVersion: 2024-08-01-preview`) in the `gpt-4o` provider.

I would expect it to be possible to use external JSON or YAML schema instead of inline, as indicated by [this issue](https://github.com/promptfoo/promptfoo/issues/1710).

- Promptfoo version: 0.95


Thanks for flagging - #2092 should fix

Released in 0.96.0!