zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Eval fails when a prompt string ends with a period "."
Root cause: Validation Gap

Version 0.68.2

Take the example promptfooconfig.yaml that is generated by the `init` command and modify the prompts by adding a "." at the end like this:

```
# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "My eval"

prompts:
  - "Write a tweet about {{topic}}."
  - "Write a concise, funny tweet about {{topic}}."
  

providers:
  - "openai:gpt-3.5-turbo"
  - "openai:gpt-4o"
  

tests:
  - vars:
      topic: bananas

  - vars:
      topic: avocado toast
    assert:
      # For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs

      # Make sure output contains the word "avocado"
      - type: icontains
        value: avocado

      # Prefer shorter outputs
      - type: javascript
        value: 1 / (output.length + 1)

  - vars:
      topic: new york city
    assert:
      # For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
      - type: llm-rubric
        value: ensure that the output is funny
```

Running `eval` will fail with error "There are no prompts in "Write a tweet about {{topic}}."

Thanks for the easy repro.  Fix in #1084 

@lacom a fix for this was deployed in [0.68.3](https://github.com/promptfoo/promptfoo/releases/tag/0.68.3) Please let us know if you run into any other issues and thanks again for reporting it!

@mldangelo 👊 awesome! Thanks for the fix!
Repository: promptfoo/promptfoo
Issue: Error running example classifier assertion
Root cause: Validation Gap

Version 0.68.2

Take the example config generated by `init` and add a classifier type assertion to the test that uses a HuggingFace model. This classifier example is taken straight from the docs:

```
# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "My eval"

prompts:
  - "Write a tweet about {{topic}}"
  - "Write a concise, funny tweet about {{topic}}"
  

providers:
  - "openai:gpt-3.5-turbo"
  - "openai:gpt-4o"
  

tests:
  - vars:
      topic: new york city
    assert:
      # For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
      - type: llm-rubric
        value: ensure that the output is funny

      - type: classifier
        provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
        value: nothate
        threshold: 0.5
```

Running `eval` fails with error:
`TypeError: Cannot assign to read only property 'options' of object '#<Object>' at runAssertion...`

Thanks for flagging!  #1083 should fix this

@lacom Thanks for reporting this one too! A fix was also deployed in [0.68.3](https://github.com/promptfoo/promptfoo/releases/tag/0.68.3) 

Please let us know if you run into any other issues!

@mldangelo excellent, appreciate you fixing!
Repository: promptfoo/promptfoo
Issue: Group tests into categories
Root cause: Unimplemented Feature Gap

I currently have ~30 test cases over 8 test files referenced from one promptfooconfig.yaml. In the web view, the test cases are presented as one long, flat table. I understand I can create multiple promptfooconfig files, eval each one separately and browse through them via the `Eval run` drop down. However, I'd like to keep all test cases on one page (the prompt is shared between all) and group them into user-defined categories. I'm thinking to display one ResultsTable per category and make these tables collapsible. Currently I'm using the test description to 'group' tests from the same file; but the presentation isn't very clear. There could be a drop-down to display/hide test categories and derivedMetrics could be calculated per category and globally over all (selected) categories. Ideally the eval's output to shell should also show the categories.

In terms of promptfooconfig syntax, it could look as follows:
```yaml
tests:
  - id: tests/cars.yaml
    category: Cars
  - id: tests/planes.yaml
    category: Airplanes
```

Are there/have there been any plans for test grouping? I could look into implementing this. If so, any feedback on what to watch out for? I found [this issue](https://github.com/promptfoo/promptfoo/issues/992), but that's related to test repeats.

I opened the other issue and a general purpose "group by" feature would solve the problem for me too. I add a variable to uniquely identify each test so one collapsible table per group, grouped by a given variable/description, would work for the other ticket too.
Repository: promptfoo/promptfoo
Issue: Evalulate Multiple Config Files Separately Instead of Combining them
Root cause: Unimplemented Feature Gap

I'm looking for a way to run `promptfoo eval` on multiple config files but without combining them together.

I noticed that when I ran the following (referring to [this issue](https://github.com/promptfoo/promptfoo/issues/329)):

`promptfoo eval -c myconfig1.yaml myconfig2.yaml`

This ended up combining the two config files which was not what I wanted.

I tried the following too:

`promptfoo eval -c configs/*`

But this ended up combining all the config files within the `configs` directory. I'm looking for a way to run `promptfoo eval` separately on all the config files within a directory but not combine them all together into a single config file. Is this possible?

What is the desired behavior in this situation?  Are you expecting multiple evals that you can review in the end, e.g. a single eval with different sections?

Yes, each config file in my scenario is running a various number of tests (these tests are different in every config file) on the prompt (this prompt is also different in every file) specified within that config file, and the provider specified also varies for each config file. The existing behaviour after running: `promptfoo eval -c configs/*` results in all of these being combined. 

Creating a separate directory for each config file and then running `promptfoo eval` on each config file separately is a time-consuming task. Also, I cannot view all the results in a consolidated place that way, I would have to check each one of them separately. What I'm looking for is to have different sections corresponding to each eval (which has been run separately/independently for each file) within a single report. I hope this clears out my intention. Please let me know if you require further details. 

Was this solved? Facing a similar issue right now!

@cheungjulia this is not currently supported in promtpfoo, but you can use this one-liner to evaluate each config file separately:

```sh
find . -maxdepth 1 -name "*.yaml" | xargs -I {} sh -c 'npx promptfoo eval -c "{}"'
```

To make it reusable, add this as an alias to your .bashrc or .zshrc:

```sh
alias pfm='find . -maxdepth 1 -name "*.yaml" | xargs -I {} sh -c '\''npx promptfoo eval -c "{}"'\'''
```
An LLM can be really helpful in modifying the command to suite your exact use case (subdirectories, other args, etc). 

Thank you! Works perfectly :)

Linking to #2453 which describes a similar use case with a detailed repro. I posted a workaround there using a single placeholder provider with per-test `provider:` overrides.
Repository: promptfoo/promptfoo
Issue: ReDoS Vulnerability in Async Dependency in promptfoo Library
Root cause: External Dependency Breakage

I ran tests using a Docker container, and the vulnerability scanner for images gave me this message. Is there a possibility to update the Async library to a stable version?

The promptfoo library uses the Async <= 2.6.4 and <= 3.2.5 dependencies, which are vulnerable to ReDoS (Regular Expression Denial of Service) when parsing the function in autoinject.

**1TH**
Name - async
Installed version / Fixed version
0:3.2.5 / Not available
Package manager
NODEPKG
File paths
usr/local/lib/node_modules/promptfoo/node_modules/async/package.json

**2ND**
Name - async
Installed version / Fixed version
0:3.2.3 / Not available
Package manager
NODEPKG
File paths
usr/local/lib/node_modules/promptfoo/node_modules/cache-manager/node_modules/async/package.json


The maintainers of async consider this to be a false positive CVE, so it is unlikely to be fixed.  https://github.com/caolan/async/issues/1975

One more note on this. The maintainers of async requested a retraction of the CVE. I am going to close this issue as a WONTFIX. We'll update async if the maintainers choose to update it. Thanks @oksanagorbachenko If you want to discuss this further (or explain why this is important to you please feel free to reach out via email - michael@promptfoo.dev)
Repository: promptfoo/promptfoo
Issue: Progress bar not matching number of `tests`
Root cause: Algorithmic Error

I have 248 `tests` in a `promptfooconfig.yaml`. However, running `promptfoo eval` I see 139/310:

```none
[██████████████████░░░░░░░░░░░░░░░░░░░░░░] 44% | ETA: 659s | 139/310 | anthropic:messages:claude-3-sonnet-20240229 "Q: {{quest" question=W
```

Where does this 310 come from? I think there might be a bad formula somewhere in `promptfoo`

---

Aside: perhaps it would be useful to add a row count somewhere to `promptfoo view`

Hey @jamesbraza! Thanks for taking the time to report this back in July 2024 – apologies it took us a while to get back to you! Since this issue was opened, we've also made several improvements to the progress bar UX (including fixes for edge cases where bars would exceed 100%), so the experience should be smoother overall.

Closing this as resolved, but please reopen if you have any questions!
Repository: promptfoo/promptfoo
Issue: Is there a way to assert only a part of the LLMresponse ?
Root cause: Documentation Deficiency

Imagine a scenario where you have an SQL LLM Agent that also prefixes each answer with the target app in which to be used:

Example LLM Response:

CoalContext
SELECT *
FROM
..

FarmContext
SELECT *
FROM
..

Is there any way to use  - type: contains-sql or is-sql specifically on the sql portion without the prefix ?


@SysOverdrive check out https://www.promptfoo.dev/docs/configuration/guide/#transforming-outputs 

You should be able to write something like 

```yaml
tests:
  - vars:
      ...
    options:
      transform: |
        const selectIndex = output.indexOf('SELECT');
        if (selectIndex !== -1) {
          return output.substring(selectIndex);
        }
        return output;
 ```
 or in an external file 
 
 ### transform.js
 ```js
 module.exports = (output, context) => {
  const selectIndex = output.indexOf('SELECT');
  // Extract the substring starting from "SELECT"
  if (selectIndex !== -1) {
    return output.substring(selectIndex);
  }
  // Return the original output if "SELECT" is not found
  return output;
};
```
with 

```yaml
defaultTest:
  options:
    transform: file://./transform.js
```
Let us know if you get stuck or need help. 

Thanks very useful will try this and come back to you on this thread.
Repository: promptfoo/promptfoo
Issue: Issue while using message arrays as a prompts with HTTP provider
Root cause: Validation Gap

I was tring to use the HTTP provider with message arrays as a prompts. And nunjucks templating method "renderString" produces invalid JSON string, which is being parsed using `JSON.parse` ( line: [providers/http.ts:46](https://github.com/promptfoo/promptfoo/blob/main/src/providers/http.ts#L46) ) and throws error of `unexpected token`

### Example:
```javascript
const aiServiceProvider = {
    id: "http://localhost/test",
    config: {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: {
        input: {
          messagesArray: "{{ prompt }}",
          modelName: "openai:gpt-4-turbo"
        },
      },
      responseParser: (response) => {
        return response;
      },
   },
  };
  ......
  ......
  ......
  const prompts = [
      [
        {
          role: "system",
          content: "You are speaking to a time traveler. Begin your conversation.",
        },
        {
          role: "user",
          content: "I want to know about switzerland. which places should I visit?",
        },
      ],
    ];
```
### Produced JSON by renderString

```json
{"method":"POST","headers":{"Content-Type":"application/json"},"body":{"input":{"messagesArray":"[
  {
    "role": "system",
    "content": "You are speaking to a time traveler. Begin your conversation."
  },
  {
    "role": "user",
    "content": "I want to know about switzerland. which places should I visit?"
  }
]","modelName":"openai:gpt-4-turbo"}}}
```

Looks like messagesArray: "{{ prompt }}" not is not resulting with proper quote escaping. 


Please correct me if my implementation is wrong :)
Repository: promptfoo/promptfoo
Issue: Check CLI arguments like --output before executing evals
Root cause: Validation Gap


The CLI argument `--output` expects a filename with a supported extension to know how to write the output. However, it seems the argument is only checked when writing the output, _after_ the evals have been performed.

The arguments should be checked _before_ the (expensive) evals are run.

For example:

```sh
$ npx promptfoo@0.65.2 eval --config promptfooconfig.yaml --output eric/test-one --no-table --filter-first-n 1
[████████████████████████████████████████] 100% | ETA: 0s | 1/1 | Sample full configuration "{{prompt}}" …
Error: Unsupported output file format eric/test-one, please use csv, txt, json, yaml, yml, html.
    at writeOutput (/Users/efung/src/eval/node_modules/promptfoo/dist/src/util.js:409:19)
```



Addressed in https://github.com/promptfoo/promptfoo/pull/1135

Hi @efung, Thank you for highlighting this issue! We've just released an update that addresses it. The CLI now validates the `--output` file extension before running evaluations, as suggested. This fix is included in [0.70.1](https://github.com/promptfoo/promptfoo/releases/tag/0.70.1) Thanks again for your suggestion!
Repository: promptfoo/promptfoo
Issue: Application error when running view or share
Root cause: External Dependency Breakage

![image](https://github.com/promptfoo/promptfoo/assets/13112588/e7e241ab-a000-4542-86bc-bbc2a119e361)


Thanks for the report!  This should be fixed in 0.20.1 - could you please give it a try?

It's working now. Thanks!
Repository: promptfoo/promptfoo
Issue: Request: link in variables
Root cause: Documentation Deficiency

It would be useful to have a clickable URL as a variable in the evaluation web viewer. Currently, the tag <a href=> is being escaped and I'd love a way to be able to pass a working link.

Example of what I'd like to be able to do:
![Screenshot from 2024-07-12 18-23-24](https://github.com/user-attachments/assets/14369542-ab7d-4848-8019-a489c3582d57)



Thanks for the question / feature request. Would you mind sharing an example from your config?

I believe you can do this now as markdown. Try:

```yaml
tests:
  - vars:
      topic: bananas
      url: '[promptfoo](https://www.promptfoo.dev/)'
```

And under table settings click: `render as markdown`. Note the quotes (`'`) around URL. Let us know if you have any other questions / comments / thoughts / feedback. Thanks! 

Yep - I would recommend doing this with the markdown support.

Thank you, that solved it for me
Repository: promptfoo/promptfoo
Issue: Request: streaming into `eval --output`
Root cause: Unimplemented Feature Gap

For long `eval` invocations (10+ minutes), there is a handy progress bar. I also specify `--output` to persist the output to disk.

For those who want to start reviewing the output mid-run, it would be nice if `promptfoo` would stream to `--output` as results become available.

I think this would require some substantial reworking, as some result-level statistics are post-computed (as well as assertions like `select-best`)

In a similar concept - I would find it useful to show the first few results in long runs to help diagnose which might be API issues (e.g., auth or quota) instead of having the run complete, finding a simple issue, and having to re-run.

@aisonym, one workaround you can try is running `promptfoo` with the `-n` option followed by a small number. This will run the first `n` test cases, allowing you to inspect the output early. If everything looks good, you can rerun the full set, and those first `n` cases will be cached, so you won’t hit the provider again.

I understand this doesn’t fully address streaming to `--output`, but it might help in diagnosing issues early on. Thanks for the suggestion, and we’ll keep this in mind for future improvements!
Repository: promptfoo/promptfoo
Issue: Script provider is asking me to supply OPEN_AI_KEY
Root cause: Documentation Deficiency

Getting this error when I run the evaluation:

```
[FAIL] Error: OpenAI API key is not set. Set OPENAI_API_KEY environment variable or pass it as an argument to the constructor. Error: OpenAI API key is not set. Set OPENAI_API_KEY environment variable or pass it as an argument to the constructor.
--
```

https://www.promptfoo.dev/docs/configuration/testing-llm-chains#using-a-script-provider

Following the guide above, I set up a custom script provider:

```
prompts:
  - tests/prompt/prompt1.txt
providers:
  - exec:poetry run python tests/prompt/ai_client_provider.py
tests:
  - vars:
      question: What's the weather in New York?
    assert:
      - type: llm-rubric
        value: does not claim to know the current weather in New York
```

How the `ai_client_provider.py` looks like:
```
import sys
from app.clients.ai_client import AIClient
from app.services.intent_service import IntentService

conversation_id = "dummy"
admin_id = "dummy"
ai_client = AIClient()
intent = IntentService(conversation_id, admin_id, ai_client.vectorstores)

chat_history = []
question = sys.argv[1]
response_text = ai_client.get_response(question, chat_history, intent)

print(response_text)
```

`AIClient` internally would load its own azure openai keys and secrets. Not sure why I am getting the errors [from here](https://github.com/promptfoo/promptfoo/blob/main/src/providers/openai.ts#L149-L153) when I've already specified a custom script. 

My config does not appear to be going down the [script completion path](https://github.com/promptfoo/promptfoo/blob/main/src/providers.ts#L73-L78).


`promptfoo --version`
=> 0.20.1

Hi @geekyme-fsmk,

The `llm-rubric` assert type uses GPT-4 by default, which is why it's asking for the OpenAI API key.  You can override the LLM grader using one of these methods: https://promptfoo.dev/docs/configuration/expected-outputs#overriding-the-llm-grader

Note that the grader uses a chat-completion style prompt by default.  If you need to override it or modify the prompt, you can set the `rubricPrompt` property on the test or assertion: https://promptfoo.dev/docs/configuration/expected-outputs#assertion-properties

Hope this helps solve your issue!

hi @typpo , understood - is it possible then to use my Azure Open AI key instead? I do have a subscription already with GPT4 enabled. 

Alright I ended up doing this: 

```
defaultTest:
  options:
    provider: azureopenai:chat:gpt-35-turbo
```

Then specifying my env vars accordingly:

`AZURE_OPENAI_API_HOST=<azure_base> AZURE_OPENAI_API_KEY=<api_key> promptfoo eval -c promptfooconfig.yaml`
Repository: promptfoo/promptfoo
Issue: Does promptfoo accept multimodal input
Root cause: Documentation Deficiency

Hello, my LLM workflow is heavily dependent on images being used along with each text prompts. Does promptfoo support this kind of workflow? I couldn't find any mentions of this in documents so my assumption is NO, but just wanted to check.

TIA!

Yes, it does, and we have several examples:

- [Claude via AWS and Anthropic with a JSON prompt](https://github.com/promptfoo/promptfoo/tree/main/examples/claude-vision)
- [Claude vs GPT-4 with JavaScript and Python prompts](https://github.com/promptfoo/promptfoo/tree/main/examples/claude-vs-gpt-image)
- [Anthropic Image Classification with CSV input](https://github.com/promptfoo/promptfoo/tree/main/examples/image-classification)
- [DALL·E image generation](https://github.com/promptfoo/promptfoo/tree/main/examples/openai-dalle-images) (output not input but possibly interesting)
- Local Models via [LM Studio](https://github.com/promptfoo/promptfoo/tree/main/examples/lm-studio) can easily be adapted for image input too. This example uses Gemma - you need to pass a base64 encoded image in the prompt. 

Let us know if you get stuck or have any more questions! Thanks!


That's fantastic, Michael! Apologies for not reading the docs more thoroughly. Might be better to highlight this as a "feature" in the docs, for n00bs like me.

No problem at all! I'll update the docs. Thanks for the callout. Please let us know if you run into any other issues. 

Hi, does it also work with the Jest integration? Like uploading images to GPT4o and assert on the result?
Repository: promptfoo/promptfoo
Issue: Bug - model-graded-closedqa shows as failed if the reason have some extra explanation 
Root cause: Validation Gap

In some cases, the output of the model-graded-closedqa has additional information. So, the test is detected as a failure. Below is a sample of it. 
![image](https://github.com/user-attachments/assets/4eb9c48a-ad92-48e3-ac34-b6b11f899b9b)


Hi @shahriardn, was this done with the default grading provider (gpt-4o)?

model-graded-closedqa is an OpenAI prompt that requests an reasoning explanation followed by Y or N.  It looks like this one failed because it ends with neither Y nor N.

Hi @typpo 
No, it's GPT 4 


Closing as this is an issue with OpenAI's prompt.  Because `model-graded-closedqa` is an exact replica of their prompt, nothing we can do here.  I recommend that you use `llm-rubric` instead for your use case.
Repository: promptfoo/promptfoo
Issue: Request - Extract the test result as a CSV file that all the metrics and the results of test included. 
Root cause: Unimplemented Feature Gap

One of the things that is very important in measuring LLMs is statistics. 
The CSV of the test result currently provides good information but I need to have exported all the metrics with the scores so for instance I can say in terms of LLM-rubric that the average score for all tests is %40 of all the defined. it would be much better if each result was extracted in a dedicated column in CSV format. 
Many thanks.
Repository: promptfoo/promptfoo
Issue: Display hostname in confirmation message when sharing eval to promptfoo server
GitHub labels: Feature Request

good-first-issue

Using the promptfoo CLI, if I'm sharing the latest eval to a self-hosted server (following [these instructions](https://www.promptfoo.dev/docs/usage/self-hosting#pointing-the-promptfoo-client-to-your-hosted-instance)), it would be nice if the confirmation message included the hostname it was sharing to.

That is, change this message:

```
Create a private shareable URL of your most recent eval?

To proceed, please confirm [Y/n]
```

to something like:

```
Create a private shareable URL of your most recent eval on promptfoo.example.com?

To proceed, please confirm [Y/n]
```

Since the hostname is set via an environment variable, having this confirmation would re-assure someone that they are indeed sending it to their instance, and not the public one.

Good suggestion!  Done in #1183

Thanks so much for this suggestion, @efung, and for all your other helpful suggestions. We released a fix for this in [0.72.1](https://github.com/promptfoo/promptfoo/releases/tag/0.72.1).