zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Web view variables ordering
Root cause: Unimplemented Feature Gap

My issue is that I want my variables in the web view to keep the same ordering as in my CSV, however, they are currently being sorted.

I was going to simply workaround this issue by numbering my vars, but I ran into nunjucks errors when trying to do so. 

Any space or special character in a variable name (For example: {{1 firstVar}} or {{1.firstVar}} throws the following error: 
```
Template render error - (unknown path) [Line 1, Column 1268]
expected variable end
```

Not at all familiar with nunjucks, so this may be expected behaviour. I just wanted to point it out as a potential additional thing to fix. 

The main ask here is to maintain ordering of variables from the dataset.




This is expected behavior from nunjucks!  Variables can't contain spaces and `.` is a special operator.  You could probably name them `1_firstVar`, `2_secondVar`, etc.

Var outputs are sorted because aside from the spreadsheets, there's no guarantee that the var will appear in every testcase.  This means for most implementations (which do not use spreadsheet), the ordering is not well defined.  The way I'd implement this request is probably by adding a `varOrder: [...]` field, or by making the columns reorderable on the frontend.

Sounds good! I'd personally prefer the solution of an optional `varOrder` field, so that the columns don't have to be manually reordered on the front end each time. Additionally, it'd be nice to just not do the sorting in the case the datasource was a spreadsheet.
Repository: promptfoo/promptfoo
Issue: Faithfulness calculation with no verdicts not covered
Root cause: Validation Gap

**Describe the bug**
Faithfulness does not handled answers with no verdicts

**To Reproduce**

when calculating faithfulness score; the case of verdict creation containing no verdicts is not handled. e.g. a content of "i apologize, but i cannot create statements or provide an analysis based on the given context ..." for verdicts will result in 1-1/x = 0 for score = (verdicts.split('verdict: no').length - 1) / statements.length;
and thus always in faithfulness 1.0 after score = 1 - score
https://github.com/promptfoo/promptfoo/blob/3402690ccf5694920de519d3008b105225e925b9/src/matchers.ts#L836


maybe the else should be rewritten to 
else if (verdicts.includes('verdicts: no')):
...
else:
  --> faithfulness could not be 

I also sometimes got a negative score of -0.2 (not sure where this resulted from)
Repository: promptfoo/promptfoo
Issue: `-c some_config.yaml` flag overrides `outputPath`
Root cause: Configuration Error

Async JavaScript doesn't seem to work anymore when loading from file. This might be a regression, as async worked before.

To replicate
```
// this doesn't work and results in ERROR undefined
async function main(output, context) {
  console.log(`raw output:\n\n${output}\n\n`);
  return true;
}
module.exports = main;

// this works

function main(output, context) {
  console.log(`raw output:\n\n${output}\n\n`);
  return true;
}

module.exports = main;
```

Edit: I've traced the origin of the error to the following.
`async` works fine when the `promptfooconfig.yaml` is named exactly as such.
However when using `npx promptfoo@latest -c other_config.yaml` then `async` fails.

A second bug occurs.
Namely, when providing `-c some_config.yaml` then `outputPath` is no longer read from `some_config.yaml` and no output file is written. Providing `-o some_path.json` works, however. This bug does not occur when running `eval` with a default `promptfooconfig.yaml` which then doesn't require providing the `-c` flag.

#232 is not published, so it makes sense that `npx promptfoo@latest` is not correctly supporting async assertions.  I am able to use assertions locally on `main`, even with `-c other_config.yaml`.  I'll bump the version and publish today, hopefully it resolves your issue.

Well that's embarrassing, I had not been running the main branch indeed. Apologies for the mess.

Running from main branch, it does work again. The issue with `-c something.yaml` overriding the `outputPath` persists.

No worries!  I've got a fix for the `outputPath` issue in a038ebd.  Will publish that + the async fix shortly.
Repository: promptfoo/promptfoo
Issue: Should folders of the `outputPath` be created when they don't exist?
Root cause: Validation Gap

Currently, providing an `outputPath` like `output/run1` will fail to execute `eval` when `output/run1` doesn't exist. I propose that the path of directories should be created during `eval` execution.

What are your opinions on this?

Easy enough to do: bc6e07e4aa8e166331b9cd4e046d0deffc6f6fcf
Repository: promptfoo/promptfoo
Issue: `async` JavaScript doesn't work from `tests.csv`
Root cause: Interface Contract Mismatch

When running `async` JavaScript evals from `tests.csv` the tests always fail even though the async function returns `true`.

This issue may be related to https://github.com/promptfoo/promptfoo/issues/247.

To replicate, take an `eval.js`
```
// always results in a test FAILURE
async function main(output, context) {
  console.log(`raw output:\n\n${output}\n\n`);
  return true;
}
module.exports = main;

// results in SUCCESS when returning true
function main(output, context) {
  console.log(`raw output:\n\n${output}\n\n`);
  return true;
}

module.exports = main;
```

The `tests.csv` may look as follows

```
to_implement,__expected
coffee machine,fn:file://compiler_eval.js
```

Note that, related to https://github.com/promptfoo/promptfoo/issues/247, this error also occurs when the config file is named `promptfooconfig.yaml` and no `-c` flag is passed.

This is solved by release [0.25.2](https://github.com/promptfoo/promptfoo/releases/tag/0.25.2)
Repository: promptfoo/promptfoo
Issue: support gemini thinking model
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Google has released gemini thinking model, e.g. gemini-2.0-flash-thinking-exp . The returned results is different than previous gemini models. The following code only prints the thinking process but not the result:
```
response = client.models.generate_content(
    model='gemini-2.0-flash-thinking-exp', contents='Solve 3*x^3-5*x=1'
)

Markdown(response.candidates[0].content.parts[0].text)
```
Currently, prompfoo seems take only candidates[0] which is not as expected.

**Describe the solution you'd like**
I think the returned value should be the final results instead of the thinking process. Not sure if this is candidates[1].




Hi @zhlmmc, #2526 adds support for thinking models.

It looks like the responses may have any number of `candidates` and `parts`.  Right now we just concatenate all of them, so at least you will have all the output in your eval.  

Unfortunately, this means that it is difficult to programmatically distinguish thought vs final output.  Google has an alpha API that attaches a `thought` property to each `part`.  Since it's alpha and not documented (except in the alpha python SDK), I think it's too early to switch to that API, but we can certainly switch once it is GA (or at least beta).

This is available in the latest release
Repository: promptfoo/promptfoo
Issue: Feature Request: UI Metrics include Token Usage/Latency Summary
Root cause: Unimplemented Feature Gap

HI Really like this cool project!! Wondering if we can add more informative Token Usage/Latency Summary as part of the UI. Currently, we can only see the token usage/ latency by hovering on the each row.
<img width="1768" alt="image" src="https://github.com/promptfoo/promptfoo/assets/93152173/cba17d51-83c1-452d-ad07-dedeb5abbe82">

Thank u 


Hi @JansonYeTao,

What kind of additional metrics would you like to see?  Would it be helpful to have aggregate metrics for each column?

#258 adds latency and token usage to the prompt column header

![latency and token usage metrics](https://user-images.githubusercontent.com/310310/279482780-1a9df9b9-9ad9-410f-8824-864b2247126a.png)

Yes thats correct. Thanks for taking care of this
Repository: promptfoo/promptfoo
Issue: [Feature Request] Multiple Consecutive Grading Prompts
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
No

**Describe the solution you'd like**
Add the ability to run multiple grading prompts sequentially to allow for Error Instance Identification, Rating, and overall Scoring for meeting summaries.

**Describe alternatives you've considered**
Running a larger single prompt with CoT to feed outputs of the previous step into the next step but I'd like to be able to view and verify intermediate steps.

**Additional context**
See [Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator](https://arxiv.org/pdf/2411.18444) for context and more details. Prompt templates are in the appendices.
Repository: promptfoo/promptfoo
Issue: shareable output folder
Root cause: Documentation Deficiency

_TLDR;_
is it possible to instruct somehow the `view` command to read the results from a different folder ?

--

I've configured the results to be saved on disk in a specific folder in my repository that will be committed so everyone in my team can have access to the test results.
But I also want them to be able to `view` the test results in the UI

so my question is it possible to instruct somehow the `view` command to read the results from a different folder ?


#257 adds support for the `PROMPTFOO_CONFIG_DIR` environment variable, which overrides the `~/.promptfoo` default.

It also adds a `promptfoo view [directory]` CLI param, which you can use to override the viewer directory.

Will be included in the next published release!
Repository: promptfoo/promptfoo
Issue: Error asserting factuality with Bedrock Claude
Root cause: Interface Contract Mismatch

**Describe the bug**
When asserting with type factuality and overriding the test provider to `bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0`, we get the error:

```
ValidationException: Malformed input request: #: subject must not be valid against schema {"required":["messages"]}#/messages: expected minimum item count: 1, found: 0, please reformat your input and try again. 
```

**To Reproduce**
1. Create a test query that is asserted on factuality. Override the test provider to `bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0`
2. Run Promptfoo Eval
3. See error

**Expected behavior**
The evaluation should not fail.

- Promptfoo version: 0.103.3

**Additional context**
We recently upgraded to latest from 0.92.2 and started seeing this issue. We had no issues on the older version.

Thanks @dliuupgrade for the report. I wasn't able to replicate the error you are getting in version 0.103.3. Do you mind sharing your config.yaml?

Setup below works for me

```
prompts:
  - 'What is the capital of {{state}}?'
providers:
  - bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0
tests:
  - vars:
      state: California
    assert:
      - type: factuality
        value: Sacramento is the capital of California
  - vars:
      state: New York
    assert:
      - type: factuality
        value: Albany is the capital city of New York state
```


@dliuupgrade if you can't get it working feel free to throw some time on my calendar: https://cal.com/michael-dangelo/15min 

Hey @MrFlounder, below is my config:

```
prompts: ['prompt.txt']
providers:
  - id: 'python:script.py'
    label: 'Test Sonnet 3-5v2 - cohere deduped - rerank, increase top k, operations pages added'
    config:
      model_name: anthropic.claude-3-5-sonnet-20241022-v2:0
      index_path: "some_path_to_index"
      top_k: 30
      ranking_model: "cohere.rerank-v3-5:0"

defaultTest:
  options:
    provider: bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0

tests:
  - description: Intake Marketing Opt Out
    vars:
      query: "How do I intake a marketing opt out request?"
    assert:
      - type: factuality
        value: >
            Answer to the question
```

The script.py is code that runs the `call_api` function through our RAG pipeline. Let me know if you want to see that as well. 

@mldangelo if it can't be resolved here, happy to schedule something thanks!

@dliuupgrade I'm able to replicate this. I'll get a fix released tomorrow! Thanks for the additional context! 

hey @dliuupgrade! I released a fix for the bedrock anthropic model error. Please update to the latest version ([0.103.9](https://github.com/promptfoo/promptfoo/releases/tag/0.103.9)) and let me know if it resolves your issue. I'll close this now, but feel free to reopen or open a new issue if you run into anything else. Thanks so much for your help in making promptfoo better. 

Thanks @mldangelo I can confirm the fix worked. Thanks for the quick response!
Repository: promptfoo/promptfoo
Issue: Caching Documentation

The Caching feature is briefly described here https://www.promptfoo.dev/docs/configuration/caching/

In order to make efficient use of this feature, it would be helpful to have more information on how the caching works.
* What key used to cache the prompts? 
* Are successful prompts cached the same way a the failed or error prompts? 
* What are the criteria that would invalidate a cache entry?

Hi @rvernica,  thanks again for your feedback! I updated the caching documentation in [#2605](https://github.com/promptfoo/promptfoo/pull/2605). You can view the latest version [here](https://www.promptfoo.dev/docs/configuration/caching). Please keep the suggestions coming. Your input has been invaluable for improving promptfoo.
Repository: promptfoo/promptfoo
Issue: [Question] Generate prompt Dynamically by using the response from previous conversation.
Root cause: Documentation Deficiency

Do we have such functionality to generate prompt dynamically to support conversation usecase? eg. populating fields of input payload by using the reply coming from previous timestamp.


user_request = ["i need help", "need help with my passport"]

For example, 

At timestamp=0, user_request="i need help", and history is an empty list, and input_payload will be:
```
{
    "query": "i need help",
    "history: [],
}
```
And then lets say, we send the input payload and get the result from LLM eg. response payload: `{"reply": "hello, what can i do for u"} ` I want to append the result into history field


then at timestamp=1, input payload will be

```
{
    "query": "need help with my passport",
    "history: [  {"reply": "hello, what can i do for u"}   ],
}
```

and we send it into LLM again



Currently, im trying something like below, but the history field is always an empty []. Also im new to nunjucks so not sure if it is supported. Thank u

```
{% set new_history = [] %}

{% if _conversation.length > 1 %}
  {% for completion in _conversation%}
    {% set new_history = new_history.push(completion.output) %}
  {% endfor %}
{% else %}

{% endif %}

{
  \"query\": \"{{ request }}\",
  \"history\": {{ new_history | dump }},
}

```




It's definitely a little tricky with nunjucks.  Here's how I would do it:

```nunjucks
{% set input %}
{
    "query": "{{ query }}",
    "history": [
      {% for completion in _conversation %}
        {"reply": "{{ completion.output }}"} {% if not loop.last %},{% endif %}
      {% endfor %}
    ]
}
{% endset %}

[{
  "role": "user",
  "content": {{ input | trim | dump }}
}]
```

Here's my config:
```yaml
prompts: [prompt.json]
providers: [openai:gpt-3.5-turbo-0613]
tests:
  - vars:
      query: how you doing
  - vars:
      query: need help with my passport
```

This has the effect of including the conversation history _within_ the prompt content, which I think is what you're going for.  Here's the prompt input for the second test case:

```json
[{
  "role": "user",
  "content": "{\n    \"query\": \"how you doing\",\n    \"history\": [\n      \n    ]\n}"
}]
```

Example output: https://app.promptfoo.dev/eval/f:feb78e53-8b15-45f7-b1ec-804813430dfa

WOW this works like a charm. Thank u so much @typpo
Repository: promptfoo/promptfoo
Issue: Disable _conversation through command line and config
Root cause: Unimplemented Feature Gap

I see that _conversation can be disabled through env: https://github.com/promptfoo/promptfoo/issues/236 . But how to disable it through command line or config like disable cache. Thanks!

This PR https://github.com/promptfoo/promptfoo/pull/282 should take care of the underlying need here. Now the `_conversation` var will only be included if it's actually used.
Repository: promptfoo/promptfoo
Issue: Update README with developer docs for Web API
Root cause: Documentation Deficiency

Can the readme be udpated to include developer documentation for how to run the web server? 
I was able to successfully run the evaluation commands locally, but 'npm run local view' *says* that the server is running on localhost15500, yet when opened in the browser every route returns 404. 


I am using WSL2 on Windows 11, and everything is going swimmingly but the UI. 

Hi @chartsNgraphs,

As of 83065a9c214681bd0d4ea507c7a384539b9456b5, to develop the web UI locally, run:
```
npm run local:web
```

This hosts a web frontend at http://localhost:3000 that talks to the `promptfoo view` web server.  The :3000 endpoint has hot reloading and other benefits of Next.js's developer environment.  (You'll still need to run `promptfoo view` or equivalent to host the CLI's API endpoints)

Hope this helps!

Thanks, this definitely helps. I'm able to get the web UI to load now. 

However, the /eval/ page doesn't show my local evals, which I would expect to see if this is meant to be the equivalent of Promptfoo View. Instead, it is asking for Login/Signup. 

What I'm looking to do is to replicate the Promptfoo View routine from my local repo. Is this possible? Or is the Web app here exclusively for database-backed deployments, not local exploration of evaluations? 


Yes, the local web app supports offline exploration of your evaluations and can run without database-backed deployments.  I use this setup for development.

Can you try the following:
- Make sure that the `NEXT_PUBLIC_PROMPTFOO_BUILD_STANDALONE_SERVER` is NOT set
- Set dummy values for supabase keys, `NEXT_PUBLIC_SUPABASE_URL=http://` and `NEXT_PUBLIC_SUPABASE_ANON_KEY=abc`

Thanks, I've given that a try and it has helped. 

with NEXT_PUBLIC_PROMPTFOO_BUILD_STANDALONE_SERVER not set, the default nextconfig value for output is "export" which results in a nextjs exception of "Middleware cannot be used with "output: export"" exception during server startup, but as far as I can tell the server is starting successfully. Appreciate the help! I might try to get the readme contribution section updated on a future PR after I dig through this and make sure I am well educated. 

Have been using the CLI for prompt testing for a while now and highly appreciative of the work and thought by all the contributors so far!
Repository: promptfoo/promptfoo
Issue: Anthropic Provider - 
Root cause: Documentation Deficiency

After defining the **ANTHROPIC_API_KEY** in `promptconfig.yaml` as well as an environment variable, the system still defaults to openapi's key:
```[FAIL] API response error: TypeError: Cannot read properties of undefined (reading '0'): {"error":{"message":"Incorrect API key provided: sk-or-v1*************************************************************7034. You can find your API key at https://platform.openai.com/account/api-keys.","type":"invalid_request_error","param":null,"code":"invalid_api_key"}}```

I've set providers as anthropic:

```yaml
providers: [anthropic:completion:claude-instant-1-100k]
```

version: 0.26.1


Are you using a `similar`, `llm-rubric`, or `model-graded-*` assertion type?  If so, you'll have to [set the provider](https://promptfoo.dev/docs/configuration/expected-outputs/similar/#overriding-the-provider) for these test cases as well.

Thank you! This resolved the issue
Repository: promptfoo/promptfoo
Issue: Support fine-tuned OpenAI models
Root cause: Validation Gap

Fine-tuned models may carry identifies such as `ft:gpt-3.5-turbo-0613:company-name::ID`. Currently, promptfoo doesn't seem to parse these identifiers correctly.

I've attempted the following variations.
```
  - openai:ft:gpt-3.5-turbo-0613:company-name::ID:
      id: gpt-3.5-custom
  - openai:chat:ft:gpt-3.5-turbo-0613:company-name::ID:
      id: gpt-3.5-custom
  - openai:completion:ft:gpt-3.5-turbo-0613:company-name::ID:
      id: gpt-3.5-custom
```

An example of an error message.
```
[FAIL] API response error: TypeError: Cannot read properties of undefined (reading '0'): {"error":{"message":"The model `ft` does not exist","type":"invalid_request_error","param":null,"code":"model_not_found"}}
```

It seems that name parsing stops after "ft".

Thanks for the report.  b52de61c6e1fd0a9e67d2476a9f3f9153084ad61 should fix this.  

The correct format would be `openai:chat:ft:gpt-3.5-turbo-0613:company-name::ID:` or `openai:completion:ft:gpt-3.5-turbo-0613:company-name::ID:`, depending on whether the model uses the `chat`-format or `completion`-format prompt.

Fixed in 0.26.2 release
Repository: promptfoo/promptfoo
Issue: Support o1 model in Azure provider
Root cause: Unimplemented Feature Gap

Just got o1 access in Azure and noticed the Azure provider does not support o1 models, but the OpenAI provider do.

For now to make it work quickly I just changed body variable from const to `let` and added the snippet below after the body assignment at <https://github.com/promptfoo/promptfoo/blob/191576f26048b24b8a1691e68b6944bff10a82db/src/providers/azure.ts#L575-L604>

```ts
        if (this.config.o1) {
            delete body["max_tokens"];
            delete body["temperature"];
            body = { ...body, max_completion_tokens: this.config.max_completion_tokens };
        }
```

Then I just set `o1: true` and `max_completion_tokens: xxx` in my o1 provider config yaml. 

Just posting this quick hack for others looking for the same solution, but I look forward to real o1 support in promptfoo for Azure provider!

@typpo saw you were working on pr #2710 please look at <strike>#2743</strike>#2817 and consider it for supporting o1 fully in azure and openai
Repository: promptfoo/promptfoo
Issue: Assertions with error information in the web UI.
Root cause: Documentation Deficiency

Currently JavaScript assertions allow boolean and integer return types.

![image](https://github.com/promptfoo/promptfoo/assets/39304339/fdfcb829-b314-4c3d-9add-77d3e6dd9f90)


I propose that a string should be passable with an error message alongside a boolean or int type.
At least in the case "false" such a message should be choosable in the UI to be displayed.

One example use-case could be compiler-feedback when compiling model-generated code.

Hey @Wehzie, try returning an object with the following structure:

```typescript
interface GradingResult {
  pass: boolean;
  score: number;
  reason: string;
}
```

For example:
```js
{
  pass: false,
  score: 0,
  reason: 'Assertion failed...',
}
```

This is vaguely documented [here](https://promptfoo.dev/docs/configuration/expected-outputs/javascript#multiline-functions), but can definitely be improved.

Updated the docs!  Closing this one out
Repository: promptfoo/promptfoo
Issue: Update docs: "Load assertions from CSV" has invalid information
Root cause: Documentation Deficiency

I was following the instructions here to setup a custom file-based JS extension.

https://www.promptfoo.dev/docs/configuration/expected-outputs/#load-assertions-from-csv
![screenshot 2025-01-15 at 09 38 48@2x](https://github.com/user-attachments/assets/8f86798b-1acf-4ac4-b720-91c6d0ce6315)

If you specify file:// it will not work unless it's a python file. See the code here:
It'll end up being evaluated as `equals` 

See https://github.com/promptfoo/promptfoo/blob/31a262d00169209fb69da39a0619f0a717110c48/src/csv.ts#L54