Repository: promptfoo/promptfoo
Issue: Calculate true/false positive rate
Root cause: Validation Gap

**Is your feature request related to a problem? Please describe.**
I have a boolean classification task that I'm performing with an LLM. I want to know what my true and false positive rate is

**Describe the solution you'd like**
I'd like metrics that outputs the true/false positive rate, and maybe other things like F1 score

It would be cool if I could define my own aggregate metric functions, and if there were some built in

Thanks!

Hi @brendanator,

Promptfoo already supports custom aggregate metrics through [derived metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/#creating-derived-metrics), which can be used to calculate metrics like true/false positive rates and F1 scores. Here's a quick example which *should* work (CC @kaiktl) 

1. Set up your assertions to capture the necessary data for your classification task. For example:

```yaml
tests:
  - vars:
      query: "Your classification input here"
    assert:
      - type: javascript
        value: output.category === 'expected_category'
        metric: CategoryAccuracy
      # Add other relevant assertions
```

2. Then, in your configuration file, define derived metrics to calculate the rates you're interested in:

```yaml:examples/openai-structured-output/promptfooconfig.yaml
derivedMetrics:
  - name: 'OverallAccuracy'
    value: '(CategoryAccuracy + OtherRelevantMetrics) / TotalMetrics'
  - name: 'TruePositiveRate'
    value: 'TruePositives / (TruePositives + FalseNegatives)'
  - name: 'FalsePositiveRate'
    value: 'FalsePositives / (FalsePositives + TrueNegatives)'
  - name: 'F1Score'
    value: '2 * (Precision * Recall) / (Precision + Recall)'
```

You can define any custom metric calculation you need using mathematical expressions. I would love to build in first class support for these types of metrics (similar to how [select-best ](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#examples-comparison) works). I'll add this to the backlog. I would also welcome contributions here 

Thanks for the suggestion @mldangelo 

When I tried this promptfoo view crashes - I wonder if this a divide by zero error?

![image](https://github.com/user-attachments/assets/b3b65a32-00c8-41c3-92c3-85c7ed2b5ccd)

For reference, here's the config I used
```yaml
defaultTest:
  threshold: 1
  assert:
  - metric: accuracy
    weight: 1
    type: javascript
    value: |
      output.should_link == (context.vars.is_linked == "True")

  - metric: count_true_positives
    weight: 0
    type: javascript
    value: |
      output.should_link && (context.vars.is_linked == "True")
  - metric: count_true_negatives
    weight: 0
    type: javascript
    value: |
      !output.should_link && (context.vars.is_linked == "False")
  - metric: count_false_positives
    weight: 0
    type: javascript
    value: |
      output.should_link && (context.vars.is_linked == "False")
  - metric: count_false_negatives
    weight: 0
    type: javascript
    value: |
      !output.should_link && (context.vars.is_linked == "True")
  - metric: count_positives
    weight: 0
    type: javascript
    value: |
      context.vars.is_linked == "True"
  - metric: count_negatives
    weight: 0
    type: javascript
    value: |
      context.vars.is_linked == "False"

derivedMetrics:
  - name: true_positive_rate
    value: |
      count_true_positives / (count_true_positives + count_false_negatives)
      ```

#2135 should fix!