zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: --interactive-providers usage

Can the '--interactive-providers' directive provide an option to automatically test the next model after a model has been tested without waiting for the user to press 'Enter'?

Probably makes sense to remove this.  Curious why you ask?  We recently refactored evaluations to do providers 1 at a time, so if you set concurrency to 1 (`promptfoo eval -j 1`) it will ensure that only 1 provider is used at a time.  This is useful for local use cases like ollama.

I discovered this feature from the Ollama docs, and wondered exactly the same, if there could be an auto yes option.

The problem with `-j 1` is that it slows down evaluation significantly. Ollama supports parallel requests (OLLAMA_NUM_PARALLEL). So, there may be enough VRAM to server parallel requests for the same model but not enough VRAM to serve multiple models.

Current version promptfoo 0.113.4 does not run one provider after another with default `-j 4`: I have all 10 small models loaded in the middle of evaluation (10 models, 20 tests total).

Could you please add an option to run providers sequentially but tests for the same provider in parallel?

UPD: I have found a workaround: you can run `promptfoo eval` with `--filter-providers <id>` multiple times and then merge results in the UI.

Thanks for the feedback here, especially @Radrik5's detailed analysis of the performance tradeoff with `-j 1`.

We agree that there's a gap: users need **provider-level sequential execution** (one model in VRAM at a time) with **test-level parallel execution** (multiple concurrent requests per model).

This is being tracked as part of a larger "Eval Orchestrator" feature: **[ENG-1356](https://linear.app/promptfooo/issue/ENG-1356)**

The proposed solution adds:
- `--sequential-providers` or `--provider-concurrency 1` flag
- Ability to run `promptfoo eval --sequential-providers -j 4` (one provider at a time, 4 parallel tests per provider)

For now, the workaround @Radrik5 mentioned works:
```bash
promptfoo eval --filter-providers <model1>
promptfoo eval --filter-providers <model2>
# Results merge in the UI
```

Leaving this open to track interest.
Repository: promptfoo/promptfoo
Issue: Pipe Character Splitting Prompt 
Root cause: Documentation Deficiency

I am using a prompt which has multiple `|` symbols used in it. Somehow, promptfoo splits the prompt into multiple prompts. Is there a fix or workaround for this without removing the pipe character?

Example config?

This is probably due to the behavior of pipe in YAML, which can be escaped.

The config file is as follows:
prompts: 
  - ../../../../prompts/myPrompt.txt

providers: 
  - *insert provider data here, using openai:gpt-4o at the moment.*

tests:
  - assert:
    - type: icontains-all
      value: 
        - 'someValue'
        
 
 The myPrompt.txt looks like the following:
 Some very long text here blah blah blah | Something written here | Something written here too | Some more very long text here.
 
 
 This gets split up like this, which is not what I wanted: 
 
![image](https://github.com/user-attachments/assets/e1a075f5-081e-46c0-8f9c-2a58aacb8fcd)


Unfortunately I'm unable to repro with the following.  If you could share an exact example or a zip, that could help.

promptfooconfig.yaml:
```
prompts:
  - prompt.txt
providers:
  - openai:gpt-4o-mini

tests:
  - assert:
    - type: icontains-all
      value:
        - 'someValue'
```
prompt.txt:
```
Some very long text here blah blah blah | Something written here | Something written here too | Some more very long text here.
```

Ah apologies, I just noticed that there was the `---` separator between the pipe characters, which is why it was being split. Didn't notice these before. 

Got it.  One workaround here would be to override the separator to something that you're not using, with the `PROMPTFOO_PROMPT_SEPARATOR` environment variable.

For example
```
PROMPTFOO_PROMPT_SEPARATOR=xxxxxxxxxx promptfoo eval
```

Thanks! Closing this.
Repository: promptfoo/promptfoo
Issue: `{#` causes `Template render error`
Root cause: Documentation Deficiency

I have a test case that contain `{#` in a variable.
I systematically get the following error:

    /usr/local/share/npm-global/lib/node_modules/promptfoo/node_modules/nunjucks/src/lib.js:32
        err = new Error(old.message);
              ^
    
    Template render error: (unknown path)
      Error: expected end of comment, got end of file
        at Object._prettifyError (/usr/local/share/npm-global/lib/node_modules/promptfoo/node_modules/nunjucks/src/lib.js:32:11)
        at Template.render (/usr/local/share/npm-global/lib/node_modules/promptfoo/node_modules/nunjucks/src/environment.js:442:21)
        at Environment.renderString (/usr/local/share/npm-global/lib/node_modules/promptfoo/node_modules/nunjucks/src/environment.js:313:17)
        at renderPrompt (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/evaluator.js:232:25)
        at async Evaluator.runEval (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/evaluator.js:267:32)
        at async processEvalStep (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/evaluator.js:627:25)
    
    Node.js v20.15.1

I’m pretty sure it’s because `{#` indicate the begining of a comment in Nunjucks, but couldn’t the variables be sanitized?

The issues occurs when the prompt come from Python or JS, e.g.

promptfooconfig.yaml:

    prompts:
      - prompt.js
    
    providers:
      - "ollama:phi3"
    
    tests:
      - vars:
          topic: ban{#anas

prompt.js:

    module.exports = async function ({ vars, provider }) {
        return [
          {
            role: 'user',
            content: `Write a tweet about ${vars.topic}`,
          },
        ];
      };


@mrBen This is a great question about a sharp edge in promptfoo and is something our documentation currently does not explain well.

You can escape it as `{{'ban{#anas'}}`.

For example:

```yaml
prompts:
  - "Write a tweet about {{topic}}"
providers:
  - "openai:gpt-4o-mini"
tests:
  - vars:
      # does not need to be escaped
      topic: ban{#anas
    assert:
      - type: contains
        # needs to be escaped with {{'escaped content'}}
        value: "{{'ban{#anas'}}"
```

I wrote a more detailed answer to a similar question yesterday: https://github.com/promptfoo/promptfoo/issues/1258#issuecomment-2247933607 

I hope this helps! 

Feel free to re-open if this does not solve your issue.
Repository: promptfoo/promptfoo
Issue: Supporting PROMPTFOO_DELAY_MS for Gemini
Root cause: Documentation Deficiency

Gemini has really strict quotas for Gemini 1.5 Pro (at least for those just signing up through normal flow - 60 requests per minute).

Can you support PROMPTFOO_DELAY_MS like you do for OpenAI? (In case it's supported and I'm just not doing it right - code below that doesn't throw and error but also seems to have no impact).

```
  - id: vertex:gemini-1.5-pro
    config:
      PROMPTFOO_DELAY_MS: 1000
```

P.S. Congrats on the funding.



Hmm, `PROMPTFOO_DELAY_MS` is actually implemented on the evaluator level, so it should work for everything.

However, it needs to be set as an environmental variable:
```
PROMPTFOO_DELAY_MS=1000 promptfoo eval
```

You can also use it as a CLI arg like this:
```
promptfoo eval --delay 1000
```

Pardon my ignorance.

Thank you.

@aisonym don't worry! It was a good question. Let us know if you encounter any other issues. 

If I were to be nitpicky - I'd point out that it's only mentioned on the OpenAI page... (https://www.promptfoo.dev/docs/providers/openai/) although to be fair it is mentioned in the "supported environment variables" section.
Repository: promptfoo/promptfoo
Issue: The test progress when using nodejs package
Root cause: Documentation Deficiency

Is there a way to get the testing progress when using NodeJS package? When using CLI, I can see a progress bar and I can see how many cases are done of total cases. But when using NodeJS package, I can only see three functions to call:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/731d2357-44f0-415d-a1e8-6964b3f80f86)

Not sure if the case is supported. Thanks!

There is!  The `evaluate` function takes a callback: 

```ts
promptfoo.evaluate(config, {
  progressCallback: (progress: number, total: number) {
    console.log(`${progress} of ${total} complete`);
  }
});
```
Repository: promptfoo/promptfoo
Issue: Selecting something on the ui for copy paste doesn't work
Root cause: Validation Gap

**Describe the bug**
On the UI, when you select some text in the output tab, the selection disappears and the state of the cell toggles.

**To Reproduce**
run an evaluation and then go to the table view on web.

**Expected behavior**
When you select the UI, the state of the output should not toggle.

**Screenshots**
![image](https://github.com/user-attachments/assets/29538c12-9537-4ba5-8075-5fd1e79e81bf)

When you select "The first question you asked", the selection disappears and the state of the output toggles. this makes it impossible to copy paste parts of the output.

**System information (please complete the following information):**

- Promptfoo version: 0.73.6



Thanks @rishimalhotra2.  #1295 should help with this.

@rishimalhotra2 this was recently deployed. You can test it out with `npx promptfoo@latest ...`. Thank you!
Repository: promptfoo/promptfoo
Issue: Improve Documentation regarding embedding providers
Root cause: Documentation Deficiency

I needed to implement a custom embedding provider due to the API utilized by the company i work for, for specific embedding models. It took quite some time to discover that I required the method call EmbeddingApi instead of simply callApi in my custom JavaScript provider. I believe this should be noted in Providers -> Custom Javascript. If desired, I would be pleased to create a pull request with an example and the necessary documentation.

Sorry about that, @Luca-Hackl.  A PR would be much appreciated!

Thanks so much @Luca-Hackl for making the docs better!
Repository: promptfoo/promptfoo
Issue: Output JSON format issue
Root cause: Validation Gap

Based on my test, I found when there is only one provider, the output JSON will not show the provider name in results:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/fdb0f73c-3854-42b9-93f2-97a0c08daae3)

I'm not sure why it is designed like this. I think it is more reliable to show the provider all the times.
![image](https://github.com/promptfoo/promptfoo/assets/1726527/ba16f5c3-dd13-40f3-a7d8-e8da7ff94fd2)


If an eval is run on a single provider, I think the information is redundant atop every single column.  Is there a reason you want it displayed?

> If an eval is run on a single provider, I think the information is redundant atop every single column. Is there a reason you want it displayed?

When parse the output, I have the following code:
```TypeScript
/**
 * Find all results of provided llm and extract the scores
 */
function getLLMScores(llm_id: string, results) {
    var scores: TestScore[] = [];
    for (const result of results) {
        // currently, when there is only one provider, promptfoo's return result will not display provider id
        // TODO
        if (result.prompt.display.includes(llm_id)) {
            var score: TestScore = {
                test_name: result.vars.name,
                score: result.score
            }
            scores.push(score);
        }
    }
    return scores;
}
```
As there is no field in results showing which provider generated this result, I can only get the information from field "display" which is a workaround. Do you think it is necessary to add provider field in each of the result?

I see.  Change b6104e9a65c417e4b83a0ea74919d3c77dfa24eb will include the provider id in `result.provider.id`, which should be a cleaner way to accomplish what you're trying to do!

This change is published in 0.23.0.
Repository: promptfoo/promptfoo
Issue: Provider "prompts" filter does not work when importing provider from a file
Root cause: Configuration Error

**Describe the bug**
The filter does not work, here is seen behavior :
- phi3 model X phi3 template prompt
- phi3 model X gemma template prompt
- gemma model  X phi3 template prompt
- gemma model  X gemma template prompt

**To Reproduce**

**config.yml**
```yaml
# We define the models prompt templates, used to structure the whole prompt.
# The tool iterate over those, and for each one, will add iteration depending on the variables / providers to use.
prompts:
  - id: "{{phi3_template_prompt}}"
    label: phi3_template_prompt
  - id: "{{gemma_template_prompt}}"
    label: gemma_template_prompt

# Providers are the models that will be used to answer the prompts.
providers:
  - file://./providers/phi3.t_0.tokens_1024.yml
  - file://./providers/gemma.t_0.tokens_1024.yml

# Global variables and assertions that are used in all tests.
defaultTest:
  vars:
    # Template prompts source files, then used in the `prompts` section.
    phi3_template_prompt: file://./prompts/model-templates/phi3.txt
    gemma_template_prompt: file://./prompts/model-templates/gemma.txt
    # System prompts source files, then used in the `providers` section.
    system_prompt_list: [
      file://./prompts/agent-system-prompts/api-event-expert/phi3.v001.txt,
      file://./prompts/agent-system-prompts/dumb.txt
    ]
```

**gemma.t_0.tokens_1024.yml**
```yaml
id: ollama:gemma:2b
label: "[gemma:2b] | temperature 0 | tokens 1024"
prompts:
  - gemma_template_prompt
config:
  temperature: 0
  max_tokens: 1024
```

**phi3.t_0.tokens_1024.yml**
```yaml
id: ollama:chat:phi3:mini
label: "[phi3:mini] | temperature 0 | tokens 1024"
prompts:
  - phi3_template_prompt
config:
  temperature: 0
  max_tokens: 1024
```

**Expected behavior**
- phi3 model X phi3 template prompt
- gemma model  X gemma template prompt

**System information (please complete the following information):**

- Promptfoo version: 0.73.7

**Additional context**
Discussed on Discord [here](https://discord.com/channels/1153767083381903389/1265076695765880916)

The label is correctly loaded and i do see this in the logs : 
```
2024-07-25 11:47:29 Loaded provider ollama:gemma:2b from ./providers/gemma.t_0.tokens_1024.yml
2024-07-25 11:47:29 Loaded provider ollama:gemma:2b from ./providers/gemma.t_0-50.tokens_1024.yml
```
 



I have the  same question

rewrite the provider of test
Repository: promptfoo/promptfoo
Issue: Can't import tools in provider from json file
Root cause: Unimplemented Feature Gap

**Describe the bug**
When trying to import a tool from a json file in the provider i get an error.

**To Reproduce**

Conf 1 : 
```yml
...
tools:
   - file://./get_current_weather.json
```

Conf 2 : 
```yaml
...
tools: [
   file://./get_current_weather.json
]

```

It does work when putting the json directly.

Tool content : 
```yaml
{
  "type": "function",
  "function": {
    "name": "get_current_weather",
    "description": "Get the current weather in a given location",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {
          "type": "string",
          "description": "The city and state, e.g. San Francisco, CA"
        },
        "unit": {
          "type": "string",
          "enum": [
            "celsius",
            "fahrenheit"
          ]
        }
      },
      "required": [
        "location"
      ]
    }
  }
}
```

Error : 
```
API error: Invalid type for 'tools[0]': expected an object, but got a string instead., Type: invalid_request_error, Code: invalid_type

{
  "error": {
    "message": "Invalid type for 'tools[0]': expected an object, but got a string instead.",
    "type": "invalid_request_error",
    "param": "tools[0]",
    "code": "invalid_type"
  }
}
```

**Expected behavior**
A provider correctly configured with the tool.

**System information (please complete the following information):**

- Promptfoo version: 0.73.7



Implementing in #1342 and should be in the next release!

Released in 0.76.0
Repository: promptfoo/promptfoo
Issue: Citation

Hello,

As a fan of promptfoo, I'd appreciate if you could add a CITATION.cff file to the repository. This would help users like me properly credit your work.For guidance, see: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files

Thank you for considering! 😁



Thanks for the suggestion @Sbkubuii! This just went into main. Please let us know if there's anything else we can do to accelerate your research!
Repository: promptfoo/promptfoo
Issue: cache when using Node.js package
Root cause: Documentation Deficiency

I'm wondering if there is cache mechanism when using Node.js package? From the documents, it seems that no cache when using Node.js package. But from my tests, I found the speed of the same tests varies a lot. If cache is supported in Node.js package, how do I disable it?

Also, I found some config options are not available in Node.js package:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/20327d7b-2085-46d3-a938-66398e9bea4c)


To force-disable the cache, you can set the environment variable `PROMPTFOO_CACHE_ENABLED=0`

`evaluateOptions` are passed as a second argument to the `evaluate()` function in the node package.  Its function signature is:

```ts
function evaluate(testSuite: EvaluateTestSuite, options?: EvaluateOptions): Promise<EvaluateSummary>
```

So you can call it like so:

```ts
evaluate({
  // main config here...
}, {
  maxConcurrency: 2,
  repeat: 2,
  delay: 0,
})
```

I've updated the node-package example to show this and will also update the docs to be clearer on this point.  To your original question, I think that there should be a way to disable the cache in this function call besides using the environment variable.
Repository: promptfoo/promptfoo
Issue: Login page in the dashboard
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Basically if we host this as a service, then we need authentication and authorization feature

**Describe the solution you'd like**
A login page with authentication. Resource authorization based on groups/ldap etc

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


Thanks for the suggestion, @dmockingbird551! We now have a managed version of Promptfoo that includes login pages, authentication, and private sharing. If you're interested in a trial or want more information, feel free to get in touch with us here: https://www.promptfoo.dev/contact/.

If you prefer to self-host, there are many options for securing your instance using OAuth2 Proxy, Authelia, Keycloak, or Nginx with basic auth.

Let me know if you have any other questions!
Repository: promptfoo/promptfoo
Issue: Missing deployment docs

are there any docs i can follow to deploy promptfoo to the cloud (specifically azure would be nice)? if not, is there a way i can get `promptfoo share` to work with my custom domain for security purposes? I couldn't find docs for this anywhere.

Hi Rishi, have you reviewed the self-hosting docs?  They'll probably be
helpful here https://www.promptfoo.dev/docs/usage/self-hosting/

On Tue, Jul 30, 2024 at 9:52 AM rishimalhotra2 ***@***.***>
wrote:

> are there any docs i can follow to deploy promptfoo to the cloud
> (specifically azure would be nice)? if not, is there a way i can get promptfoo
> share to work with my custom domain for security purposes? I couldn't
> find docs for this anywhere.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1329>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJWMLM5NGWJMARRJTFDZO7AD7AVCNFSM6AAAAABLWYPOA6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQZTQMRXGMZTENA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: [Feature request] Support Huggingface Inference API
Root cause: Unimplemented Feature Gap

Supporting the Hugginface Inference API allows users to easily test models available on the Huggingface Hub without the need to set them up locally. This can be desireable when evaluating a range of models for further customization.

See their [documentation](https://huggingface.co/docs/api-inference/index#hosted-inference-api).

Hi @Wehzie,

Thanks for the suggestion.  Out of curiosity, which models are you interested in running?  Because every model has a different output format, we likely need to add some generic `postprocess` directive for Providers in order to meaningfully support a wide range of models from HuggingFace.

To add my two cents, I would use https://huggingface.co/skeskinen/ggml/blob/main/all-MiniLM-L6-v2/ggml-model-q4_0.bin as a stand-in grader in `similar` checks, as opposed to OpenAI's `text-embedding-ada-002`.

Currently for this technique, one has to stand up LocalAI, both locally and in GitHub Actions.  With HF Inference API, could skip that step in both places

Support for Huggingface as landed, for text generation and feature extraction (embeddings).  Here's an example:

```yaml
providers:
  - id: huggingface:text-generation:mistralai/Mistral-7B-v0.1
    config:
      temperature: 0.1
```

An example embeddings provider: `huggingface:feature-extraction:sentence-transformers/all-MiniLM-L6-v2`

The request will be sent with the `HF_API_TOKEN` environment variable, if specified.

The timing of this PR is perfect.  My 16 GB Mac apparently can't run two 9 GB models...

could this be enhanced to also support a local setup of 🤗's [TGI](https://github.com/huggingface/text-generation-inference)?

The API/usage should be the same

@simon376 https://github.com/promptfoo/promptfoo/pull/290 adds support for the local setup.  It will be in the next release :)
Repository: promptfoo/promptfoo
Issue: Pre-poulate the config.yaml on UI 
Root cause: Unimplemented Feature Gap

If the `config.yaml ` is located on the file path in the promptfoo container, can we have it available on the UI by default when we run promptfoo container so we can run the evals. 

One scenario could be building a custom image for promptfoo using the promptfoo base image where the `config.yaml `and required other file are already present on the container.