zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Set max dimensions for images rendered with markdown
Root cause: Unimplemented Feature Gap

Trying to use promptfoo as an eval platform for visual language models.

I'm using markdown to render the images, which works okay! One problem is if there's a large discrepancy in image size:

<img width="1917" alt="Screenshot 2024-05-22 at 7 50 20 PM" src="https://github.com/promptfoo/promptfoo/assets/17098959/7828f052-77bd-452f-851a-97c14cf9750e">

Wondering if we can set a max image size?

### Potential Solution
1. Custom component to handle image resizing
2. Wrap it in a `MarkdownRenderer` component
3. Use `MarkdownRenderer` instead of `ReactMarkdown` component


<details>
  <summary>Example code</summary>
  
  ```typescript
// components/CustomImage.tsx
import Image from 'next/image';
import React from 'react';

interface CustomImageProps {
  src: string;
  alt: string;
  title?: string;
}

const CustomImage: React.FC<CustomImageProps> = ({ src, alt, title }) => {
  const width = 300; // Adjust this value as needed
  return (
    <Image 
      src={src} 
      alt={alt} 
      title={title} 
      width={width} 
      height="auto"
      style={{ maxWidth: '100%', height: 'auto' }} // Ensure responsiveness
    />
  );
};

export default CustomImage;
```

```typescript
// components/MarkdownRenderer.tsx
import React from 'react';
import ReactMarkdown from 'react-markdown';
import remarkGfm from 'remark-gfm';
import CustomImage from './CustomImage';

interface MarkdownRendererProps {
  content: string;
}

const MarkdownRenderer: React.FC<MarkdownRendererProps> = ({ content }) => {
  return (
    <ReactMarkdown
      remarkPlugins={[remarkGfm]}
      components={{
        img: ({ node, ...props }) => <CustomImage {...props} />
      }}
    >
      {content}
    </ReactMarkdown>
  );
};

export default MarkdownRenderer;
```

```typescript
import MarkdownRenderer from '../components/MarkdownRenderer';

const markdownContent = `
  # Hello World

  ![Alt text](https://via.placeholder.com/600x400)
`;

...
<MarkdownRenderer content={markdownContent} />
...
```
</details>


Makes sense!  Given that you've already outlined the solution, would you like to PR it, or want me to go ahead?

I'd love to give it a shot @typpo :) 

Per discussion on #812, we're good now!
Repository: promptfoo/promptfoo
Issue: Markdown appearing when manually failing test case
Root cause: Algorithmic Error

When rendering with markdown in the web viewer, if I manually fail the test case (i.e. thumbs down), the markdown appears _and_ is rendered:

https://github.com/promptfoo/promptfoo/assets/17098959/a80c214c-511b-449e-aea8-eb45bab074cc


Is this intended? Would be nice if we _only_ rendered the markdown, similar to the success case

https://github.com/promptfoo/promptfoo/assets/17098959/db302cbd-5810-4dd6-8e39-62697aee9afa

This no longer occurs as of version 0.61.0, closing!

@typpo looks like this is issue is re-occurring in 0.62.0

Can we open up this issue again? 🙏 

Hi @chrismaltais, this has been fixed in #5872 (v0.118.13). Thanks for reporting and following up when it regressed.

The root cause was fail reasons being prepended to output text, which then got rendered as markdown (causing the double display you saw). That's now been removed. Please feel free to open another issue or comment here if you run into anything else!
Repository: promptfoo/promptfoo
Issue: `n` parameters does not work with openai as provider 
Root cause: Unimplemented Feature Gap

Openai documentation: https://platform.openai.com/docs/api-reference/chat/create#chat-create-n


Curious what would be the expectation here? Multiple outputs within the same table two? Multiple table rows ?

In general, if you want to generate multiple you can use the --repeat argument. This works for all providers including OpenAI.
Repository: promptfoo/promptfoo
Issue: Allow overriding provider config in prompt or test case
Root cause: Documentation Deficiency

Currently, all provider config must be done at the top-level. It's not possible to run the same provider with different configuration. It should be possible to override the provider config at the test-case or prompt level (or both).

Hi @tomquist,

Good news - this feature is now fully supported!

You can run the same provider with different configurations in two ways:

**1. Define the same provider multiple times with different labels:**
```yaml
providers:
  - id: openai:gpt-4
    label: low-temp
    config:
      temperature: 0.1
  - id: openai:gpt-4
    label: high-temp
    config:
      temperature: 1.5
```

**2. Override provider config at the test-case level:**
```yaml
providers:
  - openai:gpt-4

tests:
  - vars:
      question: "Test question"
    provider:
      id: openai:gpt-4
      config:
        temperature: 1.5
```

See the [Configuration Reference](https://www.promptfoo.dev/docs/configuration/reference#provideroptions) for complete details on ProviderOptions.

Since this issue is from May 2024 and the feature is now working, I'm closing it. If you have any follow-up questions, feel free to comment or re-open!

Thanks!
Repository: promptfoo/promptfoo
Issue: Using vars on rubricPrompt lacks serialization, leading to the failure of the entire assertion.
Root cause: Validation Gap

I intend to use LLM2 to assist in evaluating the response of another LLM1.
Therefore, the vars for the first request will be passed to LLM2 again.
However, if the vars contain special characters (such as `\n`), it will lead to the failure of the entire llm-rubric assertion.

In the following example for a translation request.
An error occurs due to the `\n` in the `vars.input`. (`... families are all alike. \nAnd every ...`)
And removing this character will make the evaluation be successful.


example.yaml
```
prompts: >-
  [{
    "role": "system",
    "content": "Translate text into {{ language }}."
  }, {
    "role": "user",
    "content": "{{ input }}"
  }]
providers: [openai:gpt-3.5-turbo]
defaultTest:
  options:
    rubricPrompt:
      - role: system
        content: >-
          Evaluate the quality of the translation provided by an AI assistant to the user input displayed below.
          Score the response on a scale of 0 to 10.
          Output your response in the following JSON format: {pass: bool, score: number, reason: string}
      - role: user
        content: >-
          [Input Start]
          {{ input }}
          [Input End]

          [Translation Start]
          {{ output }}
          [Translation End]

tests:
  - vars:
      input: "Happy families are all alike. \nAnd every unhappy family is unhappy in its own way."
      language: "Chinese"
    assert:
      - type: llm-rubric
```

Will get the result:
![image](https://github.com/promptfoo/promptfoo/assets/11830576/f1a47044-5204-4668-8755-054356b96a4e)

Due to the unserialized `input`:
`...\"content\":\"[Input Start] Happy families are all alike. \n And every unhappy family is unhappy in its own way. [Input End]\\n[Translation Start]...`

Is there any workaround or maybe a better way to do this, please?

Related to #759 

Hi typpo,

In my personal opinion, this doesn't seem to be related to issue #759.
The key difference is only about whether the `tests.vars.input` string contains special characters, which has nothing to do with the config generation.

Meanwhile, the problem I see is in the implementation of `src/matchers.ts#fromVars`.
Currently, it only stringifies object values, but the same treatment should be applied to string values as well.

Did you agree with this perspective?
If you do, I would be happy to submit a request to help fix it. (You can also fix it yourself directly.)

Thank you for looking into this.  I agree and would gladly accept a PR @YiFanChen99 !
Repository: promptfoo/promptfoo
Issue: Mistral Instruct prompt erroneously considered as JSON
Root cause: Validation Gap

Trying to use promptfoo on mistral-instruct 7b.

Here is a basic non-working example taken from [mistral prompting guides ](https://www.promptingguide.ai/models/mistral-7b#chat-template-for-mistral-7b-instruct):
```

 You are a helpful code assistant. Your task is to generate a valid JSON object based on the given information:
name: John
lastname: Smith
Just generate the JSON object without explanations:


```

When using this prompt directly inside the YAML configuration or inside a dedicated file `.txt`, we are welcomed with the following error :
`[FAIL] Error: Chat Completion : prompt is not a valid JSON string: SyntaxError: Unexpected token 'I', "
 You "... is not valid JSON`

As far as I understand, promptfoo wrongly consider this "text" block as a JSON because of the first `[`. 

Is there any way to get around this issue and force this prompt as "plain-text" even if it starts with a bracket ?

Thanks for reporting, this is indeed a bug. 

As a temporary workaround, you can set options.prefix and options.suffix to 
 and 
 respectively. https://promptfoo.dev/docs/configuration/reference/#test-case. Then your text file can just contain the prompt text. 

To avoid repetition you can use the defaultTest property to apply this to all tests. 

Actually, which provider are you using for mistral 7b instruct?  I think there is a deeper issue here where you're using a chat completion provider for a non-chat API.  This means it expects a JSON formatted string and the error is intended.

We're using a locally hosted Ollama inference server with [mistral:instruct model](https://ollama.com/library/mistral:instruct)

Perhaps you should be prefixing the provider with ollama:completion instead of ollama:chat

Let me know if that doesn't work!
Repository: promptfoo/promptfoo
Issue: Not working using text prompt with Palm2 Google Vertex API
Root cause: Unimplemented Feature Gap

Hi, 

I tried to use this tool using the provider `vertex:text-bison@002` as the config file below:
```
# Learn more: https://promptfoo.dev/docs/configuration/guide
description: 'Testing'

prompts:
  - 'Convert this English to {{language}}: {{input}}'

providers:
  - id: vertex:text-bison@002
    config:
      generationConfig:
        temperature: 0.4
        maxOutputTokens: 1024
        topP: 0.95

tests:
  - vars:
      language: Portuguese
      input: Hello my old friend!
```


It returned the error:
```
"status":400,"code":400,"errors":[{"message":"Invalid instance: input has an instance that's missing a prompt field","domain":"global","reason":"badRequest"}]
```

Checking the source code, it looks like this tool only works for the Palm2 Chat API. 

Is it intentional, or could I open a PR to fix/add it?

Edit:
it looks like it is a simple change https://cloud.google.com/vertex-ai/generative-ai/docs/text/test-text-prompts.

Not intentional, just not yet implemented! Would definitely accept a PR to handle text completions. 

The model `text-bison` will be discontinued on Oct 9, 2024. https://cloud.google.com/vertex-ai/generative-ai/docs/legacy/legacy-models#legacy_model_discontinuation_date

I am closing this issue since it is not worth implementing it anymore.
Repository: promptfoo/promptfoo
Issue: promptfoo 0.60.0 --share not working
Root cause: External Dependency Breakage

I am upgrading to promptfoo 0.60.0 and run my test again with `--share` flag.

<img width="806" alt="image" src="https://github.com/promptfoo/promptfoo/assets/6728224/3f075f54-07d1-4889-90ad-407cc00b1259">
<img width="1920" alt="image" src="https://github.com/promptfoo/promptfoo/assets/6728224/25ea3995-81a7-4a0b-acbe-0ffd7eb65117">

It failed both in CI/CD and local. Looks like a bug.
Could you reproduce this on your side?

Thanks for the report.  This should be fixed now.  Could you please confirm on your side?

@typpo Thanks!! It works!
Repository: promptfoo/promptfoo
Issue: Support `systemInstruction` for Gemini (PALM)
Root cause: Unimplemented Feature Gap

So that users can put there system prompt here.

Ref. to the [document](https://ai.google.dev/api/rest/v1beta/models/streamGenerateContent#request-body).
Note that the `systemInstruction` types `Content`, in form of `{ parts: [ { text: ... } ] }`, like the regular user message.

And there may be Nunjucks variables in the system prompt?

Thanks for flagging, https://github.com/promptfoo/promptfoo/pull/841 adds support

Hi typpo,

I understand that #841 is supported on the Vertex.
Would PALM also support in the near future?

And the implementation on #841 seems not support using `vars` for `systemInstruction` explicitly.
Has this feature been implemented and currently supported?


`var` support is added in https://github.com/promptfoo/promptfoo/pull/902 for gemini system instructions. I don't think PaLM models support system instruction?

Released in 0.63.0

Thank you for the fast supporting.

I apologize for the confusion my previous statement have caused.
When I mentioned PaLM models, I was referring to Google AI Studio models (which is implemented by the file `palm.ts`).
I have confirmed that certain models, such as `google:gemini-1.5-pro-latest` and `google:gemini-1.5-flash-latest`, do support system instructions.
As I do not have a VERTEX_API_KEY, I conducted all of my tests using Google AI Studio models. 

Hi @typpo, should this be reopened for the Google AI Studio model?

Yes, thank you @YiFanChen99.  It is on my to do list :)

Hi @EdwardSanglayGarcia and everyone who participated in this discussion! Excellent news - `systemInstruction` is now fully supported for Gemini providers! 

You can use it like this:
```yaml
providers:
  - id: google:gemini-pro
    config:
      systemInstruction: "You are a helpful assistant..."
```

Please feel free to send me an email if you would like some swag! Thanks for reporting this.
Repository: promptfoo/promptfoo
Issue: Allow options to avoid using special characters

Hi, I have a need to invoke a shell script to install and run ```promptfoo eval``` in Java code and host it in a K8s Ubuntu based container. The promptfoo commands are executed in a separate process and I wired the output to stdout and stderr. However, since promtpfoo can output lots of special characters, like progress  bar, green check mark, yellow triangle question mark, colorful bold fonts, somehow it is not printable in my java program. 

For instance, when I run ```prompt share```, nothing shows up as the url is in bold green and there is a green check mark.

I verified the LANG env var. to be UTF8, font setting and terminal emulator, all look good. 

Is there a way to avoid using special character but only output ASCII? 

FYI
node -v: **v20.6.1**
promptfoo --version: **0.57.1**

To disable terminal colors for printed outputs, set `FORCE_COLOR=0` (this is supported by the [chalk](https://github.com/chalk/chalk) library).  You will also have to disable the progress bar and table separately:

```
FORCE_COLOR=0 promptfoo eval --no-progress-bar --no-table
```

Hope this helps!

Thanks for the prompt response, is there a way to disable the green check mark and yellow triangle mark as well?

The green check is a single unicode character, I believe its coloration is just dependent on your how your OS or terminal renders those characters (for example, on my Ubuntu it is just a plain black and white checkmark)

Closing this out, let me know if you run into any issues!
Repository: promptfoo/promptfoo
Issue: How can I set threshold for avg. of test cases scores with csv?
Root cause: Unimplemented Feature Gap

I currently set test cases with csv using `__expected{num}` well. But I cannot set `threshold` for avg. score of `__expected{num}`'s results.
I want to get `PASS` if the avg. of the scores exceed the threshold I set.

example) If I set `0.5` for threshold

__expected1 : `1.0`
__expected2 : `1.0`
__expected3 : `1.0`
__expected4 : `0`

0.75 > 0.5 : `PASS`

Can you tell me how I set the threshold for tests cases with csv? 
If there is no way, Can you support this function such as `__threshold`?

Related : #606, #238

`__threshold` support added in #903!

Released in 0.63.0
Repository: promptfoo/promptfoo
Issue: Types for `promptfoo.evaluate` broken when using TS
Root cause: Interface Contract Mismatch

Using the node api in a TS environment is broken due to overlapping & incompatible types.

Using the `node-package` example (but with Typescript) you can see the errors.
<img width="1035" alt="Screenshot 2024-06-03 at 14 38 57" src="https://github.com/promptfoo/promptfoo/assets/1196098/afb6e089-bebb-408d-a236-079fd305a106">
<img width="1031" alt="Screenshot 2024-06-03 at 14 39 26" src="https://github.com/promptfoo/promptfoo/assets/1196098/6bb480e9-94ac-483f-8222-edebc072910f">
<img width="772" alt="Screenshot 2024-06-03 at 14 39 31" src="https://github.com/promptfoo/promptfoo/assets/1196098/faef202f-2175-4fe6-ba86-b3edfd4648f9">

I've began addressing the issue with #854

However the bug runs deep into the "writefile" territory... I would appreciate some guidance on what can/cant be changed here as passing functions into `JSON.stringify` doesn't always do what we want.
Repository: promptfoo/promptfoo
Issue: Add ability to bust cache from the web UI
Root cause: Unimplemented Feature Gap

Our team interacts with promptfoo strictly through the web ui.

Right now, there isn't a way to bust the cache from the web ui, making it unusable.

Thinking a solution could either be:
- A button/setting in the web ui to clear the cache
- Supporting `cache: false` in the yaml file specified in the web ui

Or both? @typpo any opinions?
Repository: promptfoo/promptfoo
Issue: When Python provider raises exception, details are no longer recorded
Root cause: Validation Gap

In #833 by @mldangelo , individual Python assertions that raise an exception now have their exception messages recorded as the `reason` in the `GradingResult` which is then visible as part of the set of `gradingResults` recorded for every test case when requesting JSON output.

However, an undesirable side effect of this is that if a Python **provider** raises an exception, that message is no longer visible.

Previously, say, in 0.59.1, if a provider raised an exception (e.g. incorrect OpenAI token), the exception and stack trace get dumped to the command-line.

Now, the command-line doesn't show anything, but the JSON test results now include an `error` value which begins:

```
Error: The Python script `call_api` function must return a dict with an `output` string/object or `error` string

Error: The Python script `call_api` function must return a dict with an `output` string/object or `error` string
    at PythonProvider.callApi
```

It would be nice if the exception message and stack trace were recorded in the `error` field.

Fix landed in #863 

Thanks for the fixes. When will this be released? 

This change is released in 0.62.0
Repository: promptfoo/promptfoo
Issue: Ensure percent complete accurately reflects test suite
Root cause: Algorithmic Error

I have the following promptfoo config file:
```yaml
description: 'Segmenter'
prompts:
- " "
providers:
- id: 'python:src/segmenter/provider.py'
  label: 'Staging'
  config:
    env: 'staging'
    storage_folder: 'test/staging'
- id: 'python:src/segmenter/provider.py'
  label: 'Production'
  config:
    env: 'production'
    storage_folder: 'test/production'
tests: src/segmenter/tests.yaml
evaluateOptions:
  maxConcurrency: 2
  delay: 500
  showProgressBar: true
  cache: false
```

Where the `src/segmenter/tests.yaml` file looks like:
```yaml
- description: Apparel and Accessories
  vars:
    originalImage: "![image](https://example.com/some_image.png)"
  assert:
    - type: latency
      threshold: 2000
- description: Apparel and Accessories
  vars:
    originalImage: "![image](https://example.com/some_image.png)"
  assert:
    - type: latency
      threshold: 2000
- description: Apparel and Accessories
  vars:
    originalImage: "![image](https://example.com/some_image.png)"
  assert:
    - type: latency
      threshold: 2000
```

When I run these tests via the web ui, I'm expecting the results to go 0 -> 33% -> 66% -> 100% (i.e. updating based on the number of variables).

However, the web ui stays at 0% until all the tests are completed, then redirects me to the evals page:
https://github.com/promptfoo/promptfoo/assets/17098959/9457aac4-a284-4246-ba76-0dc7d9509cf5
Repository: promptfoo/promptfoo
Issue: Support for loading test cases from directory
Root cause: Unimplemented Feature Gap

## Description

To facilitate easier management, discussion, and understanding of individual test cases, it would be beneficial to allow defining and loading tests (consisting of variables and assertions pairs) from a specified directory.

## Proposed Implementation

Consider a scenario where the test definitions could be mentioned in a configuration file in the following way:

promptfooconfig.yaml
```yaml
prompts: [prompts/prompt1.yaml]
providers: [openai:gpt-3.5-turbo]
tests: './tests/
```

Where the test directory would look like this:

```
tests
├── 1.yaml
├── 2.yaml
└── 3.yaml
```

Each yaml will then hold the defintion of the input vars and the assertions for one specific test case:

```yaml
vars:
  text: Some input text for the test
assert:
  - type: not-contains-any
    value:
      - "diagnose"
  - type: contains-all
    value:
      - "enigmatic creatures"
  - type: similar
    value: Some input text for the testing
    threshold: 0.95
```

## Potential Benefits:

**1. Granular Test Case Management:** This structure enables users to work with each test case as a separate unit, alleviating the necessity of maintaining all test cases within a single file.

**2. Facilitates Discussion:** A numerical test case system allows teams to easily refer and discuss specific test cases.

**3. Grouping Tests:** This structure would enable users to categorize their tests based on directory names (for example, tests/accuracy/ and tests/creativity/), enabling easier comparison of prompt performance across various performance categories, rather than on an aggregate measure across all tests.

Currently, while tests.csv is used, it does not allow defining multiple assertions for the same variable. This proposed change will be especially useful when the number of assertions for each test case is significantly large, making their management in separate files more efficient.

I'm open to contributing towards the implementation of this feature. Please let me know if any additional information is required.

Great suggestion!  I appreciate the offer of contributing to the implementation.  Turns out I was already working on this and just opened a PR at https://github.com/promptfoo/promptfoo/pull/88

Still work in progress but probably ready to land tonight or tomorrow - let me know what you think.