zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: parsing string score in `llmRubric` output
Root cause: Validation Gap

**Describe the bug**
Occasionally, the llm grading model used in `llmRubric` assertion may return a json of invalid format. Specifically, the `"score"` property may contain non-numerical data (in most cases, a string with a score, like `"00.75"`), which will lead to a  `c.toFixed is not a function` exception on the eval result page, entirely breaking it and preventing users from seeing any grading results.

**To Reproduce**

1. Create following `promptfooconfig.yaml`:
    ```yaml
    description: "LLM rubric score bug demo"
    
    prompts:
      - This is a test sentence.
    
    providers:
      - bedrock:amazon.nova-micro-v1:0
    
    tests:
      - assert:
          - type: llm-rubric
            value: something
            rubricPrompt:
              - role: user
                content: |
                  STRICTLY follow my instruction. Repeat the following EXACTLY, NOTHING MORE THAN A SINGLE JSON OBJECT
                  { "score": "0.75", "reason": "Something" }
            provider: bedrock:amazon.nova-micro-v1:0
    ```
2. run `promptfoo eval`
3. run `promptfoo view`
4. Navigate to http://localhost:15500/eval
3. See the error
    


**Expected behavior**
The `llmRubric` should try to parse the output score as a number instead of optimistically believe that it has a valid type.


**Screenshots**
<img width="734" alt="Image" src="https://github.com/user-attachments/assets/d1941a2c-ecfe-4e15-82b2-80904dc37186" />

**System information:**

- Promptfoo version: 0.103.19

UPD: this behaviour has changed in https://github.com/promptfoo/promptfoo/commit/864406f9630184ec5c059a3ec338d4b4f5323637 so now the invalid score would be turned to `0.0` or `1.0` based on the value of `pass`. I would still suggest to try to parse the `score` as number as in some cases it provides a valid value but just in the wrong type.


Thus it's no longer a bug, but rather a feature request
Repository: promptfoo/promptfoo
Issue: support aliyun china
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
Hardcoding the API_BASE_URL in Alibaba's provider leads to an inability to support the use of Alibaba Cloud China.

![Image](https://github.com/user-attachments/assets/e448fdb6-5677-4213-8f06-54bc0d66f677)

**Describe the solution you'd like**
change apiBaseUrl to bellow

```
super(modelName, {
      ...options,
      config: {
        ...options.config,
        apiBaseUrl: options.config.API_BASE_URL || API_BASE_URL,
        apiKeyEnvar: 'DASHSCOPE_API_KEY',
      },
    });
```

thank you

@Baozi2 this was fixed in https://github.com/promptfoo/promptfoo/pull/3040 and released! Please pull the latest version of promptfoo. Would also love any notes you have on making the aliyun provider better. Feel free to comment or re-open the issue!
Repository: promptfoo/promptfoo
Issue: "--no-progress-bar" for redteam run/eval
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The `eval` cli has a "--no-progress-bar" flag to prevent showing the progress bar. The redteam does not expose this flag0

**Describe the solution you'd like**
Surface the `"--no-progress-bar"` in the redteam cli commands


Implemented in #3043 !

Wow great turnaround!
Repository: promptfoo/promptfoo
Issue: Importing custom python assertion from csv
Root cause: Documentation Deficiency

Hello,

I have the following yaml that I use custom python assertion. Inside this script, I get "groundtruthValues" from context and use them as groundtruth values.

yaml file:

```
description:` Test

prompts: [prompts.txt]
   
providers:
  - ../../provider.js

defaultTest:
  options:
    provider:
        id: gpt
        config:
          apiHost: azure.com
          temperature: 0.2
tests:
  - vars:
      question: What is LLM?
    assert:
      - type: llm-rubric
        value: "A large language model (LLM) is a type of machine learning"
      - type: python
        value: file://../../custom_assertion.py
        config:
            groundtruthValues:
            - name: testName
               value: testValue
           

```

So, if I want to import tests from csv as I showed below, how can I get "groundtruthValues"? Would it be possible?

```
description: Test

prompts: [prompts.txt]
   
providers:
  - ../../provider.js

defaultTest:
  options:
    provider:
        id: gpt
        config:
          apiHost: azure.com
          temperature: 0.2
tests: file://tests.csv
           
```

tests.csv:

```
question,__expected1,__expected2
What is LLM?, llm-rubric: A large language model (LLM) is a type of machine learning, python: file://../../custom_assertion.py
```


You can add another column / other columns that contains the ground truth values.

Example CSV:
```
question,__expected1,__expected2,groundtruthValue
What is LLM?,"llm-rubric: A large language model (LLM) is a type of machine learning","python: file://assert_with_config.py:get_assert","some value"
```

and to access it:
```
def get_assert(output, context):
    # Parse the context to get groundtruth values
    vars = context.get('vars', {})
    groundtruth_values = vars.get('groundtruthValue', [])
```
Repository: promptfoo/promptfoo
Issue: Bug: Google Sheets Public-Access does not work
Root cause: Documentation Deficiency

Tried loading tests from a public sheets with view access (and tried with edit access),
and have been facing API auth errors. 


My config:
```
prompts:
  - '{{input}}'
providers:
  - openai:gpt-4o-mini
  
tests: <gsheet link>
```

Error:
```
{"error":{"message":"You do not have access to the project tied to the API key.","type":"invalid_request_error","param":null,"code":"invalid_project"}}
```

@sert121 Are you able to access that sheet outside of promptfoo?  I just verified that our unauthenticated example works and updated it: https://github.com/promptfoo/promptfoo/pull/3104 

If you can't get this working please feel free to throw some time on my calendar: https://cal.com/michael-dangelo/15min

Hi @mldangelo , it works now! thanks
Repository: promptfoo/promptfoo
Issue: Prompt injection guardrail false positives
Root cause: Algorithmic Error

**Describe the bug**
Almost every simple message gets flagged as a prompt injection attack.


**Screenshots**

<img width="1177" alt="Image" src="https://github.com/user-attachments/assets/54dc1797-2dda-4262-ab77-1ebfb98ff3ba" />


**System information:**

```
{
  "version": "0.104.0",
  "platform": {
    "os": "darwin",
    "release": "23.6.0",
    "arch": "arm64",
    "nodeVersion": "v23.7.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}

```

Not sure, but it looks like the guardrails API isn’t openly available or can't be self-hostable, which makes it difficult to fully use or trust the project. 

It’d be helpful to clarify if the guardrails can be run locally, audited in any way, and which models (if OS) are being used behind, etc.

https://github.com/promptfoo/promptfoo/issues/3689
Repository: promptfoo/promptfoo
Issue: Model can inject arbitrary HTML into my results page
Root cause: Validation Gap

[The results table seems to deliberately inject text as HTML ...?](https://github.com/promptfoo/promptfoo/blob/b24edb65be1bf7d0e8383ce1337d1d405633f6cc/src/web/nextui/src/app/eval/ResultsTable.tsx#L72-L89)

The model's output is arbitrary. I don't want the model to be able to inject random scripts into my results table. At best, this is annoying, and at worst, it's a security vuln.

IMO, input/output in the results page should be shown as plaintext.

Introduced in https://github.com/promptfoo/promptfoo/pull/55 - but not sure why

Thanks for pointing this out.  #309 should fix

Amazingly fast response! Thanks! ❤️ 

No problem!  This fix was included in 0.30.0
Repository: promptfoo/promptfoo
Issue: Access Models configured in LiteLLM Proxy Server
Root cause: Documentation Deficiency

In my litellm, I have configured the below bedrock models.  In promptfoo, when I the model name like this "openai:chat:bedrock:amazon-nova-lite-v1:0" i am getting the error not found at LiteLLM side. What is the correct way to litellm models?

```yaml
model_list:
  - model_name: amazon-nova-lite-v1:0
    litellm_params: 
      model: bedrock/amazon.nova-lite-v1:0 
  - model_name: amazon-nova-micro-v1:0
    litellm_params:
      model: bedrock/amazon.nova-micro-v1:0
  - model_name: amazon-nova-pro-v1:0
    litellm_params:
      model: bedrock/amazon.nova-pro-v1:0
```

Hey @ramnamy, you need to use the exact model_name from your LiteLLM config in promptfoo (without “bedrock:” in the name). In your LiteLLM config you have:

```yaml
model_list:
  - model_name: amazon-nova-lite-v1:0
    litellm_params:
      model: bedrock/amazon.nova-lite-v1:0
```
So in promptfoo, reference it like this:

```yaml title="promptfooconfig.yaml"
providers:
  - id: openai:chat:amazon-nova-lite-v1:0
    config:
      apiBaseUrl: http://localhost:4000
```

Then use `openai:chat:amazon-nova-lite-v1:0` in your tests. LiteLLM will route it internally to `bedrock/amazon.nova-lite-v1:0`. See more information on LiteLLM in our [docs](https://www.promptfoo.dev/docs/providers/litellm/). Please let us know if this works or if you run into any other issues! 

Thanks. All good now.  Guess what my issue was, / at the end of my apiBaseUrl
Repository: promptfoo/promptfoo
Issue: Implementing OpenAI assistants through node package
Root cause: Configuration Error

I'm trying to implement the node package using the OpenAI assistants.
The issue is that I can't manage to get my app working by setting the `apiKey` on my provider configuration.
I don't want to use environment variables so I try to use the `evaluate` function that way but the documentation is not clear enough.

Do you have any idea on how to achieve this ?

Here is the piece of my code :

```
const results = await promptfoo.evaluate(
        {
            env: {
                OPENAI_API_KEY: XXX,
            },
            prompts: prompts,//Assume this variable is set
            providers: ["openai:assistant:" + XXX],
        },
        {
            maxConcurrency: 1,
            progressCallback: (progress, total) => {
                console.log(`PROGRESS: ${progress} / ${total}`);
            },
        }
    );
```

And here is the error I get when executing this :
`Error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add `apiKey` to the provider config.`

Thanks for the report!  It looks like the current Assistants implementation is not correctly using the `apiKey` properties on the provider or env.  This is fixed in 6a5aab72f2b00fe650feeb1db7c41122e9958da9

The fix will be published in the next release.  Until then, you'd have to use environment variable (or clone & build the promptfoo repo manually).

Sorry to bother you again, thanks again for your work on the previous issue I had.

I'm trying to implement the "similar" assert type on my program but it cannot work for the same reason as my previous issue.
On line 539 of file `/src/providers/openai.ts`, you put this line of code : `export const DefaultEmbeddingProvider = new OpenAiEmbeddingProvider('text-embedding-ada-002');`.

The issue is that I have this error : "OpenAI API key must be set for similarity comparison" even though I filled the api key through the `env.config` variable before. I assume it takes the environment variable which I want to avoid. Is there a possibility for you to add a custom config for the `DefaultEmbeddingProvider` so it retrieve the key set in the `config` variable declared previously ?

You could override the provider on the test itself:

```yaml
  assert:
    - type: similar
      value: Hello world
      provider:
        id: openai:embeddings:text-embedding-ada-002
        config:
          apiKey: 'abc123'
```
Repository: promptfoo/promptfoo
Issue: Similarity based assertions with transformations
Root cause: Interface Contract Mismatch

I've been trying to run Python transformations on top my lm output before evaluating them but they just freeze.

The output contains a json inside <json> tags which I want to process and return content from a single key.

Here's a sample output:
```
  <some_tag>...</some_tag>
  <json>{my_key: "abc" }</json>
```

Here's my transformation fn:

```
  def get_transform(output, context):
      pattern = r"<json>(.*?)</json>"
      match = re.search(pattern, output, re.DOTALL)
  
      json_content, transformed_output = "", ""
      if match:
          json_content = match.group(1).strip()
  
      transformed_output = json.loads(json_content).get("my_key")
  
      return str(transformed_output)
```

Running the evaluation remains stuck for a while:

<img width="1209" alt="Image" src="https://github.com/user-attachments/assets/a76611d0-b77f-44f8-a589-5a26ebfce7c4" />

After a while I can see the transformation worked and the content was extracted fine but the test case errored out with:
```
API call error: TypeError: Cannot read properties of undefined (reading 'message'). Output:
undefined
```

And this only happens with assertion that  checks for similarity; all other assertions (deterministic or llm-based) work fine.

Hi @manankalra I am having trouble replicating this. Can you have a look at https://github.com/promptfoo/promptfoo/compare/fix/issue-3100 and let me know if this is an accurate re-construction of your issue? If it's easier, feel free to grab a time on my calendar: https://cal.com/michael-dangelo/15min

Thanks for looking at this closely, Michael. I'll try running the config you shared today and update here.
Repository: promptfoo/promptfoo
Issue: Allow to remove variables from the output
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I am doing a vision test for base64 images. I am providing the base64 content and mime type to the prompt template via dynamic js vars. However, I don't want to see this base64 in the output - it takes to much space, makes the browser page slow and results cannot be shared (the content is too long).

**Describe the solution you'd like**
Some kind of an option that would allow me to do not output specific var into the end result

**Describe alternatives you've considered**
I tried to use output transformation to delete variables from the context, but it also affected subsequent prompt executions, resulting in failures.
```js
cleanupOutput: (output, context) => {
    // Remove base64 image content to not pollute the outputs
    delete context.vars.content;
    delete context.vars.type;
    return output;
  },
```

I tried to play with dynamic tests/input transforms to see if I'll able to workaround this, but failed

**Additional context**
-


You can solve this by using the `PROMPTFOO_STRIP_TEST_VARS` environment variable, which will remove test variables from the output while still allowing them to be used during test execution.

```bash
export PROMPTFOO_STRIP_TEST_VARS=true
promptfoo eval
```

This will keep your base64 content and mime type variables available for test execution, remove these variables from the final output, and not affect subsequent test executions.

If you're also concerned about memory usage due to large outputs, you can combine this with other output stripping options. See https://www.promptfoo.dev/docs/usage/troubleshooting/ for more information. 

As for selectively removing variables - you can implement a custom provider where you keep a reference (such as a file path or a uuid) in your vars instead of the base64 image and load the base64 image only in the provider code. I would also love to explore features to selectively strip vars from evals but likely wont get to it soon.
Repository: promptfoo/promptfoo
Issue: Performance as metric
Root cause: Unimplemented Feature Gap

Hi,

I see that there is a `latencyMs` field on the yaml results file, but I see no obvious way to compare performance (be it tokens/s, time to answer, multiple queries and whatnot).

The use-case is to be able to understand if a number of inference servers is enough for the demand, or if one needs to scale up things. In fact, right now I don't even know how fast my servers run or how much load I can throw at them.

Do you think that makes sense for promptfoo? 

Thanks!

Hi @surak,

Timings are surfaced in the webview, although it is a bit subtle:

![llm latency benchmark](https://github.com/promptfoo/promptfoo/assets/310310/a992e38f-2453-4c4d-92a5-7e00224ec131)

When you run `promptfoo eval`, be sure to use the `--no-cache` argument (timings aren't shown when the outputs are pulled from cache)

I agree it'd help if latency were more of a first-class metric - displayed more prominently, able to be used in assertions, etc.  As well as additional metrics such as tokens/second.

So, the code already measures and stores it somehow, that's already great!

Now it "just"* needs to be a metric for comparison just like the others!

* "just" is never so simple when you are the one writing the code for it. 

So do you think this would make sense to be added to promptfoo?

Yup, definitely.  I plan to follow up on this :)  

Hi @surak,

Pull request #344 adds a `latency` assertion type.  You can use it to fail any test case that takes longer than a given amount of time.  For example:

```yaml
assert:
  - type: latency
    threshold: 4000  # anything slower than 4 seconds fails
```

Would this be useful to you?

Added in version 0.33.0 !
Repository: promptfoo/promptfoo
Issue: Question on Default Behavior for Script based loads without Function Names
Root cause: Documentation Deficiency

I am curious what the default behavior should be when no function is provided, particularly for python files.

For Providers, `main()` is run and any printed output is collected.
For dynamic TestCases (new feature: https://github.com/promptfoo/promptfoo/pull/2993) the default is `generate_tests` but only currently for Python and not for JS or TS: https://github.com/promptfoo/promptfoo/blob/12374f38c3c058bad6baadf823a4126a08c260af/src/util/testCaseReader.ts#L103

I was going to add `generate_tests` as the default func name for TS (and JS)  here, following the Python example: https://github.com/promptfoo/promptfoo/blob/12374f38c3c058bad6baadf823a4126a08c260af/src/util/testCaseReader.ts#L110

But then I realize there is a bit of an inconsistency with the Provider behavior. I stumbled on this as a new user because I assumed `main()` would work with TestCases and it didn't. I was wondering if there's opportunity to unify this, given that the dynamic Test Cases is new (would be a breaking change if you change default behavior from `generate_tests()` to `main()`.

I see that the "execute `main()`" idea is labeled as legacy though, so maybe you don't want to enable this behavior moving forward? I can understand that to avoid an unnecessary print statement (and potential weird escaping issues with long prompts, I think I ran into something like this).
https://github.com/promptfoo/promptfoo/blob/26e10e9547139d51c41aadeb2642e58b59bfbeec/src/prompts/processors/python.ts#L97

For other people following along - we spoke offline and agreed to move away from executing main for Python. @dhodun’s work in [#3139](https://github.com/promptfoo/promptfoo/pull/3139) has made great progress toward loading assets from any json/yaml/js/python/etc script and we'll likely work towards making everything this way in the future. Closing the issue for now. Thank you so much for your contributions @dhodun
Repository: promptfoo/promptfoo
Issue: Google provider broken with OpenAI-compatible prompt format containing multiple content parts
Root cause: Interface Contract Mismatch

**Describe the bug**

In the `google` provider with (at least) Gemini models, when using OpenAI compatible prompt format with multiple `content` elements in the user role (whether or not there is actually more than one part), the following error occurs:

```
API did not return any candidate responses: {"error":{"code":400,"message":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field.\nInvalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field.","status":"INVALID_ARGUMENT","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"contents[0]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[0]': Cannot find field."},{"field":"contents[1]","description":"Invalid JSON payload received. Unknown name \"content\" at 'contents[1]': Cannot find field."}]}]}}
```

This was previously mentioned in [issue 1876](https://github.com/promptfoo/promptfoo/issues/1876), but I don't think the PR that's mentioned at the end of the linked issue was actually relevant to that issue because that change was to the Vertex provider, and the issue was about the Google provider. This is, however, the same error that was reported there.

This format works:

```
- role: system
  content: |-
    You are a helpful AI assistant.
- role: user
  content: |-
    What is {{thing}}?
```

But this one doesn't:

```
- role: system
  content: |-
    You are a helpful AI assistant.
- role: user
  content:
    - type: text
      text: |-
        What is {{thing}}?
```

(Of course, in reality, there would be no reason to use this form unless there were multiple content parts; I've shown it with one to be as minimal as possible.)

This second format works fine with other providers, like OpenAI and Anthropic – it's only the Google provider that has an issue with it.

**To Reproduce**

Failing `promptfooconfig.yaml`:
```
description: test
prompts:
    - file://prompt.yaml
providers:
    - google:gemini-1.5-flash-002
    - google:gemini-2.0-flash
tests:
    - vars:
        thing: 2 + 2
```

Add bad prompt and `promptfooconfig.yaml` files and run promptfoo on the config.


**Expected behavior**

The request would return a completion, rather than a 400 Bad Request.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:  0.104.4

```
{
  "version": "0.104.4",
  "platform": {
    "os": "linux",
    "release": "5.10.233-1-MANJARO",
    "arch": "x64",
    "nodeVersion": "v18.17.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

Thanks for flagging!  #3206 should fix in the next release

Sweet, thanks! Probably won't have a chance to check this today but I'll let you know if I run into any issues.