zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Issue: Difficulty in Configuring provider Property for Test Assertions
Root cause: Validation Gap

### Description:
According to the documentation, all test assertion types support the provider property. However, I'm facing challenges when configuring it to use the Azure OpenAI API for extracting embeddings.

### Configuration:

```
assert:
      - type: similar
        provider: azureopenai:completion:<deploymentName>
        value: "{{variable_in_tests}}"
```
### Error:
Received API response:
```
API response error: Error: No embedding returned: {"error":{"message":"Incorrect API key provided: adf8b367********************6d0e. You can find your API key at https://platform.openai.com/account/api-keys.","type":"invalid_request_error","param":null,"code":"invalid_api_key"}}
```
Any guidance on this would be appreciated.


### Potential Root Cause & Suggested Fix

Upon inspecting the source code, I noticed that the class AzureOpenAiEmbeddingProvider https://github.com/promptfoo/promptfoo/blob/84d937c9d0713559cf18d60851a1d3dc341ad7c5/src/providers/azureopenai.ts#L64 is not being utilised anywhere.

A possible remedy could involve extending the conditional statement in [poviders.ts](https://github.com/promptfoo/promptfoo/blob/84d937c9d0713559cf18d60851a1d3dc341ad7c5/src/providers.ts#L137C7-L138C81) to incorporate the following:
```
else if (modelType === 'embedding') {
      return new AzureOpenAiEmbeddingProvider(deploymentName, providerOptions);
}
```
This might help in correctly initialising the AzureOpenAiEmbeddingProvider when the model type is 'embedding'.

Thanks for the catch, you're right that the `similar` assertion is not correctly using the provider from the test or assert object.  I'll work on a fix for this.
Repository: promptfoo/promptfoo
Issue: Promptfoo cannot decode special characters
Root cause: Environment Incompatibility

**Describe the bug**
When the input message contains the hungarian `Á` or `Ő` characters I get the following error

```
Python code execution failed: Error running Python script: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 25: character maps to <undefined>
Stack Trace: Error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 25: character maps to <undefined>
    at PythonShell.parseError (C:\Users\bdomonkos\AppData\Roaming\nvm\v20.9.0\node_modules\promptfoo\node_modules\python-shell\index.js:303:21)
    at terminateIfNeeded (C:\Users\bdomonkos\AppData\Roaming\nvm\v20.9.0\node_modules\promptfoo\node_modules\python-shell\index.js:193:32)
    at ChildProcess.<anonymous> (C:\Users\bdomonkos\AppData\Roaming\nvm\v20.9.0\node_modules\promptfoo\node_modules\python-shell\index.js:185:13)
    at ChildProcess.emit (node:events:514:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
    --Python Traceback: --
    File "C:\Users\bdomonkos\AppData\Roaming\nvm\v20.9.0\node_modules\promptfoo\dist\src\python\wrapper.py", line 33, in <module>
      data = json.load(fp)
             ^^^^^^^^^^^^^
    File "C:\Users\bdomonkos\miniconda3\envs\pythonservice\Lib\json\__init__.py", line 293, in load
      return loads(fp.read(),
                   ^^^^^^^^^
    File "C:\Users\bdomonkos\miniconda3\envs\pythonservice\Lib\encodings\cp1250.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

**To Reproduce**
Run promptfoo with user prompts containing the hungarian `Á` or `Ő` characters.

**Expected behavior**
Promptfoo should handle special characters or provide the opportunity to change the encoding.

**System information (please complete the following information):**

- Promptfoo version: 0.80.2

Hi @balintdom, 

Thank you for reporting this issue. We’ve just issued a release in [v0.80.3](https://github.com/promptfoo/promptfoo/releases/tag/0.80.3) that addresses this issue by ensuring that UTF-8 encoding is used for Python IPC. This should resolve the UnicodeDecodeError you encountered. Please update to the latest version and let us know if you continue to experience any problems. Thanks again and please don't hesitate to reach out.
Repository: promptfoo/promptfoo
Issue: [Feature Request] Implement G-EVAL Inspired Weighted Sum Scoring for LLM Grading
Root cause: Documentation Deficiency

### Potential concern:
In the [self-grading example](https://github.com/promptfoo/promptfoo/blob/main/examples/custom-grading-prompt/promptfooconfig.yaml), you utilise LLMs to judge text quality using a form-based method. I've spotted two problems with this scoring approach:

1. Often, one number, like 3 on a 1-5 scale, seems to appear a lot more in the scores. This can make scores too similar and not match well with human opinions.
2. Even if the prompt asks for decimals, LLMs tend to give whole number scores. This results in a lot of identical scores, missing the tiny differences between texts.

### Feature Description:
Integrate a [G-EVAL](https://github.com/nlpyang/geval) inspired grading system that addresses these issues, by using the
probabilities of output tokens from LLMs to normalise the scores and take their weighted summation as the final score.

### Expected Benefits:

- Improved Correlation: As observed in the G-EVAL paper, this approach has a higher correlation with human evaluations compared to other NLG evaluators.
- Refined Grading: Using the probabilities of output rating tokens ensures a more nuanced and detailed grading system.


Hi @jamesbraza, closing out old issues. Thanks for suggesting G-EVAL weighted sum scoring! Great news - this feature has been implemented and is available in the current version. You can use G-EVAL scoring by using the `geval` assertion type. Check out the implementation in `src/assertions/geval.ts` and tests in `test/assertions/geval.test.ts` for examples.

Documentation: https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#g-eval

Thanks again for helping make promptfoo better! Feel free to open a new issue if you have any questions about using this feature.

Also please feel to drop me a note / email if you have more thoughts / comments / etc. on promptfoo. Your feedback has been invaluable. Thank you!
Repository: promptfoo/promptfoo
Issue: TypeError: Cannot read properties of undefined (reading 'indexOf') appearing randomly when trying to generate test dataset
Root cause: Validation Gap

**Describe the bug**
TypeError while trying to generate dataset. 

I have tried various different backends (Mistral-Large, GPT4-mini, Claude 3.5). The error appears randomly. Perhaps it has something to do with the way promptfoo handles empty responses -- perhaps it does not retry them when generating test datasets? No idea. 

If anybody is willing to assist me with this bug, I think I could try to work something out. I'm not a pro JS dev...

**To Reproduce**

```
[promptfoo-based-on-docs] npx promptfoo@latest generate dataset -w                                                                                                                          Cache is disabled.
 ██████████████████░░░░░░░░░░░░░░░░░░░░░░ 43% | ETA: 11s | 7/16/Users/mpasternak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/util/index.js:893
    let openBracket = str.indexOf('{');
                          ^

TypeError: Cannot read properties of undefined (reading 'indexOf')
    at extractJsonObjects (/Users/mpasternak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/util/index.js:893:27)
    at synthesize (/Users/mpasternak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/testCases.js:327:70)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async doGenerateDataset (/Users/mpasternak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/commands/generate/dataset.js:63:21)

Node.js v18.19.1
```

Hi @mpasternak,

Thank you for reporting this issue. For some reason (possibly related to your existing test cases or prompts) the models you tried are not outputting JSON. I just opened PR #1544 to attempt to address your problem. If you can, please try to check out and run the branch. And to help me investigate further, could you please provide

1. Your promptfoo configuration file (with sensitive information redacted)
2. The version of promptfoo you're using (`npx promptfoo --version`)
3. Any specific prompt or configuration that consistently reproduces the issue.

Thank you for your patience and for reporting an issue!

Hi @mpasternak,

I believe this issue has been addressed in the recent updates. I'll go ahead and close this for now. However, if the issue persists, please feel free to reopen it, and I'll look at it again. 

Thanks again for reporting it!
Repository: promptfoo/promptfoo
Issue: Is there a way to import a environment variable that is not predefined in promptfoo ?
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently, it seems that only certain predefined environment variables are supported, and there isn't a straightforward way to import custom environment variables that are specific to my workflow or setup.

**Describe the solution you'd like**
I would like the ability to import any environment variable, even if it's not predefined in `promptfoo`. 



By "import an environment variable", do you mean referencing an environment
variable in a prompt template?

On Thu, Aug 29, 2024, 1:41 AM Amine Djeghri ***@***.***>
wrote:

> *Is your feature request related to a problem? Please describe.*
> Currently, it seems that only certain predefined environment variables are
> supported, and there isn't a straightforward way to import custom
> environment variables that are specific to my workflow or setup.
>
> *Describe the solution you'd like*
> I would like the ability to import any environment variable, even if it's
> not predefined in promptfoo.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1552>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJQOHHJMLCL6H5LEJ6DZT3NCXAVCNFSM6AAAAABNJ5KRBKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ4TGOBRGI2TSNI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Yes!

@AmineDjeghri I'd either use something to generate YAML configuration file... or check out this answer: https://stackoverflow.com/questions/52287881/how-do-access-node-env-in-nunjucks-template

Thanks to @AmineDjeghri and @mpasternak for raising and discussing this issue. @typpo has shipped a fix, and you can check it out by running `npx promptfoo@latest`. Here's a quick example:

You can access environment variables in your templates using the `env` global:

```yaml
tests:
  - vars:
      headline: 'Latest news in {{ env.COUNTRY }}'
```

Please let us know if you encounter any issues or have more ideas for improving the product.

Hello again,
I am trying to use it in the providers, but it seems that it doesn't work. Promptfoo version is 0.90.3

```
providers:
  - id: azureopenai:chat:gpt-4o-mini # env variables are in .env
  - id: file://../configs/config_json.py
    label: '{{ env.AZURE_OPENAI_DEPLOYMENT_NAME }}'
```

I'm trying to use an env var to pass an api key in the header but it's not working, is the env vars only available in specific places in the configuration? Can it be extended to headers we well? Thanks!

`headers:
                'apikey': {{ env.MY_API_KEY }}`

@zpedrol @mldangelo  seems to be a bug: https://github.com/promptfoo/promptfoo/issues/3331
Repository: promptfoo/promptfoo
Issue: Skip assertion decision
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

I am testing a large number of test cases using a specified python metric set.
In the case of some test cases, there is a dynamic situation in which it is not possible to determine whether to pass/fail because it does not meet the conditions.

for example, there is a test case which can determined by a specific var, but the var has not exist. I don't want to set the result as both of pass/fail. It should be set to just`skipped`.

**Describe the solution you'd like**

Need to way to set `skipped` and show `1 Skipped` on web ui.

For example
```python
@dataclass
class GradingResult:
    pass_: bool  # 'pass' is a reserved keyword in Python
    score: float
    reason: str
    component_results: Optional[List['GradingResult']] = None
    named_scores: Optional[Dict[str, float]] = None  # Appear as metrics in the UI
    skipped : bool
```
**Describe alternatives you've considered**
Until now, fail has been set in python assertion of my project for ambiguous cases.

Hi @sjsinju,

Thank you for your suggestion! We currently don't have built-in support for marking test cases as "skipped" directly. However, you can achieve similar functionality by implementing custom assertions using Python or JavaScript within your test cases.

If you're interested, I'd be happy to brainstorm adding first-class support for skipping tests into Promptfoo. However, we would need to approach this carefully, as our current system runs all assertions in parallel and they aren't dependent on each other.

Another option you might consider is using a sentinel value, like a score of 0.5, to indicate that a test should be treated as "skipped."

Let me know if you'd like to discuss this further or need help implementing the workaround!
Repository: promptfoo/promptfoo
Issue: Streaming is used when using Ollama as provider
Root cause: Configuration Error

**Describe the bug**
I am using Ollama to evaluate several models using promptfoo. When running the evals using the verbose flag, I get to see several individual tokens being sent from the Ollama server.

Checking the docs from Ollama, you can suppress streaming setting the stream parameter to false. (Related issue: https://github.com/ollama/ollama/pull/639).  I have used this configuration but the output is streamed (making no difference).

```yaml
providers:
  - id: ollama:chat:llama3.1:8b-instruct-q8_0
    label: llama31_8B_Q8
    config:
      temperature: 0
      stream: false
      system: "You are an expert in foobar test generation ..."
```

Setting stream to false makes no difference and all requests are processed as streaming. Note that this also applies when using Ollama for the grader LLM.

Sample output:

```
Ollama generate API response: {"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.348475547Z","message":{"role":"assistant","content":"No"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.36468283Z","message":{"role":"assistant","content":" tengo"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.38080782Z","message":{"role":"assistant","content":" información"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.396949072Z","message":{"role":"assistant","content":" sobre"},"done":false}
{"model":"llama3.1:8b-instruct-q8_0","created_at":"2024-09-02T12:49:16.413265688Z","message":{"role":"assistant","content":" las"},"done":false}
```


**To Reproduce**
Use the previous config snippet in a yaml file as a provider and call the evaluation as `promptfoo eval --verbose` to visualize the output.

**Expected behavior**
Non interactive evaluations should not use stream as it might be less efficient.

**System information (please complete the following information):**

- Promptfoo version: 0.78.3



@elsatch [8be6447](https://github.com/promptfoo/promptfoo/commit/8be644786f51a5bf301b74b84959584ad2a81927) disables streaming
Repository: promptfoo/promptfoo
Issue: Use the api from the browser
Root cause: Architectural Constraint

I would like to use the api from a browser frontend. I tried to use the node-package example in the browser and got an error: 

> ReferenceError: global is not defined

I guess this is because it is meant to be used in aNode env. Would it be possible to have a browser compatibility for this package? 

promptfoo is not bundled for the browser and unfortunately relies on node primitives and the filesystem for things like custom functions and caching.  So this is unlikely to happen soon :(
Repository: promptfoo/promptfoo
Issue: Absence of AZURE_OPENAI_API_HOST in AZURE provider
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**

When I try to use the AZURE_OPENAI_API_HOST environment variable, it doesn't seem to work.  Additionally, this environment variable isn't mentioned in the documentation,. 


Hey @AmineDjeghri, thanks for flagging this! Our docs were missing some key info about the Azure environment variables - I just pushed an update to fix this. Check out [this](https://github.com/promptfoo/promptfoo/commit/535bc61d93408653a658451b5c9742984feb9ec0) commit. I added a new section on environment variables, including `AZURE_OPENAI_API_HOST`, `AZURE_OPENAI_API_BASE_URL`, and `AZURE_OPENAI_BASE_URL`. It should clear things up about which ones you need and in what order they're used.

You can see the updated docs [here](https://www.promptfoo.dev/docs/providers/azure/). Give it a look and let me know if it helps or if anything's still unclear. We're always looking to improve our docs, so any feedback is welcome.

If you're still having trouble after checking the new docs - we can hop on a call and figure it out together. Always happy to help! Thanks again for bringing this up. It's helpful feedback like this that makes promptfoo better for everyone.

Hello @mldangelo 
i just tested it and it works.

Thank you for your quick updates
Repository: promptfoo/promptfoo
Issue: Unable to delete all evals
Root cause: Validation Gap

**Describe the bug**
When running `npx promptfoo delete eval all` a foreign key contraint error is thrown. Making it impossible to delete all evals in one go.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run a few evals with `npx promptfoo eval`
2. Make sure evals are in the database with `npx promptfoo list evals`
3. Try to delete them all with `npx promptfoo delete eval all`
4. See the following error:

```
? Are you sure you want to delete all stored evaluations? This action cannot be undone. yes

C:\dev\promptfoo\node_modules\drizzle-orm\better-sqlite3\session.cjs:88
    return this.stmt.run(...params);
                     ^
SqliteError: FOREIGN KEY constraint failed
    at PreparedQuery.run (C:\dev\promptfoo\node_modules\drizzle-orm\better-sqlite3\session.cjs:88:22)
    at QueryPromise.run (C:\dev\promptfoo\node_modules\drizzle-orm\sqlite-core\query-builders\delete.cjs:97:28)
    at deleteAllEvals (C:\dev\promptfoo\node_modules\promptfoo\dist\src\util\index.js:812:37)
    at handleEvalDeleteAll (C:\dev\promptfoo\node_modules\promptfoo\dist\src\commands\delete.js:28:37)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Command.<anonymous> (C:\dev\promptfoo\node_modules\promptfoo\dist\src\commands\delete.js:70:13) {
  code: 'SQLITE_CONSTRAINT_FOREIGNKEY'
}

Node.js v20.13.1
```

**Expected behavior**
From my understanding of the CLI docs, `delete eval all` should delete all evals.

**System information (please complete the following information):**

- Promptfoo version: 0.83.2

I am running this on Windows using Powershell

**Additional context**
I have already tried to completely remove the .promptfoo folder, but even running a single eval and trying to delete all causes the issue.


This is fixed in the latest release!  Thanks for flagging

Tested and working. Thanks for the quick fix!
Repository: promptfoo/promptfoo
Issue: Multilingual Redteam Strategy only uses OpenAI as a provider when generating translations 
Root cause: Architectural Constraint

**Description**
The multilingual redteam strategy will never use a custom provided provider for generations and translations. 

**To Reproduce**
1. Run promptfoo init and choose a multilingual strategy 
2. Modify the promptfooconfig.yaml file to host your own redteam provider 
3. Run export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true 
4. promptfoo redteam generate

**Expected behavior**
The multilingual strategy should use the red team provider to generate its translations

- Promptfoo version: 0.83.2

**Additional context**
The source of the bug is contained in `src/redteam/strategies/multilingual.ts` in the `translate` function.

```typescript
export async function translate(text: string, lang: string): Promise<string> {
  const redteamProvider = new OpenAiChatCompletionProvider(ATTACKER_MODEL, {
    config: {
      temperature: TEMPERATURE,
      response_format: { type: 'json_object' },
    },
  });

  const result = await redteamProvider.callApi(
    dedent`Translate the <Text> below to language '${lang}'. Respond with JSON object containing {translation: string}.
    
    <Text>
    ${text}
    </Text>`,
  );
  return (JSON.parse(result.output) as { translation: string }).translation;
}
```


Thanks for the report, @Marsh61.  #1586 should fix this, and will be included in the next release.
Repository: promptfoo/promptfoo
Issue: Question: Python prompt function returning object

From the docs on custom prompt functions: https://promptfoo.dev/docs/configuration/parameters#prompt-functions

> The function should return a string or an object that represents the prompt.

Can you elaborate on what was meant by "object"?  Like what types or behaviors should be present in a non-`str` object?

Hi James!  Technically it will take any Javascript object and send it to whichever API you're using, assuming the provider supports this.

In practice, the main reason to use this is to return a chat-style object like:

```ts
[
    {
      role: 'system',
      content: `You're an angry pirate. Be concise and stay in character.`,
    },
    {
      role: 'user',
      content: `Tell me about ${vars.topic}`,
    },
  ]
```

Okay I follow.  So the general signature for a Python prompt function would be this, correct?

```python
def generate_prompt(context: dict[str, str | dict]) -> str | list[dict[str, str]]:
    ...
```

Right now yes, but it's possible for provider endpoints to add support for new formats anytime.

Thank you!  Closing this out as resolved, appreciated.
Repository: promptfoo/promptfoo
Issue: Fail Message for icontains-all Assertion Should Mark Which String is Missing
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I'm currently using a lot of deterministic assertions in my tests. I mostly use `icontains-all` or `contains-all` and when I have a large number of tests which fail, I have to manually check and see which string was not found in the response from the LLM.

**Describe the solution you'd like**
I would like the test results to point out exactly which strings are missing when I use something like `icontains-all` so that I don't have to manually check all the failing tests.

**Describe alternatives you've considered**
The alternative is to check the assertion manually and see exactly what string makes it fail.

**Additional context**
Used v0.82.0

Good suggestion!  Added in #1596 

Hi @mrafnadeem-apimatic, we’ve just released [0.85.0](https://github.com/promptfoo/promptfoo/releases/tag/0.85.0), which includes improvements for icontains-all and contains-all error messages, now pointing out exactly which string is missing in the test results. Let us know if you run into any issues or have further feedback. Thanks for the great suggestion! 

Hey @typpo and @mldangelo, thanks for this release. You guys are life savers. Keep up the great work.