Repository: promptfoo/promptfoo
Issue: Severity not aligned with Attack Success Rate
Root cause: Documentation Deficiency

**Describe the bug**
Severity stats in report are static and does not depend on the result of the check.

**To Reproduce**
Run several red teaming on different models and see the severity stats in the report are always:
Critical: 3, High: 9, Medium: 12, Low: 15


**Expected behavior**

Account properly severities based on checks results. In the below screenshots we can see that for an attack success rate of 10% is rated as "Medium" and "0%" as "High". So it seems the calculus between Attack Success Rate & Severity is not done properly.


**Screenshots**

<img width="1230" height="167" alt="Image" src="https://github.com/user-attachments/assets/d5f5442b-ae25-4f41-aae1-5bd65a970cf4" />

<img width="827" height="552" alt="Image" src="https://github.com/user-attachments/assets/7e586e2e-bf59-4a87-aab5-ed6f8ec772ab" />

If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: **0.117.2**



Hey @gelim, I'm not able to reproduce this- the stats not static as far as I see. Could you share the configs for the runs you did? or `promptfoo share` your evals/reports? 

Sure, here is the `promptfooconfig.yaml` for a Meta Llama4 redteaming on Azure:

``` yaml
description: "Red team Llama4"

targets:
  - id: azure:chat:Meta-Llama-4-Maverick
    config:
      apiHost: 'MASKED.cognitiveservices.azure.com'
      apiVersion: '2024-05-01-preview'
      deploymentName: 'Llama-4-Maverick-17B-128E-Instruct-FP8'
      apiKey: 'MASKED'

prompts:
  - '{{prompt}}'

redteam:
  purpose: "REPLACEME"

  numTests: 5

  plugins:
    - id: bias:age  # Age-related bias detection
    - id: bias:disability  # Disability-related bias detection
    - id: bias:gender  # Gender-related bias detection
    - id: bias:race  # Race-related bias detection
    - id: contracts  # Tests for unauthorized contractual commitments and legal exposure
    - id: excessive-agency  # Tests for unauthorized actions beyond defined system boundaries
    - id: hallucination  # Tests for fabrication of false or misleading information
    - id: harmful:chemical-biological-weapons  # Tests handling of WMD-related content
    - id: harmful:child-exploitation  # Tests handling of child exploitation content
    - id: harmful:copyright-violations  # Tests handling of intellectual property violations
    - id: harmful:cybercrime  # Tests handling of cybercrime-related content
    - id: harmful:cybercrime:malicious-code  # Tests creation of malicious code
    - id: harmful:graphic-content  # Tests handling of graphic or violent content
    - id: harmful:harassment-bullying  # Tests handling of harassment and bullying content
    - id: harmful:hate  # Tests handling of hate speech and discriminatory content
    - id: harmful:illegal-activities  # Tests handling of general illegal activities
    - id: harmful:illegal-drugs  # Tests handling of illegal drug-related content
    - id: harmful:illegal-drugs:meth  # Tests handling of methamphetamine-related content
    - id: harmful:indiscriminate-weapons  # Tests handling of weapons-related content
    - id: harmful:insults  # Tests handling of personal attacks and insults
    - id: harmful:intellectual-property  # Tests handling of IP theft and violations
    - id: harmful:misinformation-disinformation  # Tests handling of false information campaigns
    - id: harmful:non-violent-crime  # Tests handling of non-violent criminal content
    - id: harmful:privacy  # Tests handling of privacy violation attempts
    - id: harmful:profanity  # Tests handling of profane or inappropriate language
    - id: harmful:radicalization  # Tests handling of extremist content and recruitment
    - id: harmful:self-harm  # Tests handling of self-harm and suicide content
    - id: harmful:sex-crime  # Tests handling of sexual crime content
    - id: harmful:sexual-content  # Tests handling of explicit sexual content
    - id: harmful:specialized-advice  # Tests handling of unauthorized professional advice
    - id: harmful:unsafe-practices  # Tests handling of dangerous activity instructions
    - id: harmful:violent-crime  # Tests handling of violent crime content
    - id: harmful:weapons:ied  # Tests handling of Improvised Explosive Devices
    - id: hijacking  # Tests for unauthorized resource usage and purpose deviation
    - id: pii:api-db  # Tests for PII exposure via API/database access
    - id: pii:direct  # Tests for direct PII exposure vulnerabilities
    - id: pii:session  # Tests for PII exposure in session data
    - id: pii:social  # Tests for PII exposure via social engineering
    - id: politics  # Tests handling of political content and bias
  strategies:
    - id: jailbreak  # Single-shot optimization of safety bypass techniques
    - id: jailbreak:composite  # Combines multiple jailbreak techniques for enhanced effectiveness
    - id: goat  # Dynamic multi-turn attack generation using adversarial techniques
      config:
        stateful: true

```

By doing same for a DeepSeek V3 endpoint or whatever will be the same as can be seen in my screenshots on OP.

@gelim Severities are a representation of the risk associated with the vulnerability category, they are not related to the attack success rate.  https://github.com/promptfoo/promptfoo/blob/main/src/redteam/constants/metadata.ts#L288

Oh well ok thanks for the pointer.

When you compare N LLMs and want to show their respective strengths and weaknesses, you assess them against the same checks/strategies. This Severity stat will be then the same and very misleading for the reader.

I would have expected a mapping like CVSS when you assign a severity "High", "Medium", "Low" to a number (between 0 & 10.0).  Wouldn't it make more sense here?

The CVSS numeric score is not based on attack success rate, it's based on constant aspects of a vulnerability regardless of the target, so using ASR would not be the right approach. Having a consistent severity is more similar to how CVSS does it.

The CVSS comparison has its limit, but I thought it will be clear to highlight the weakness of doing a static mapping with a check and its severity. CVSS has multiple parameters that are involved to get the final score, it's not relying on a single static one.

My expectations are different on this apparently, if I'm the only one feeling this is not right as it is shown today, then fine.

I know this isn't what you're asking for, but in case it helps @gelim, you can adjust the static severity mapping to your preferred (constant) values: https://www.promptfoo.dev/docs/red-team/configuration/#severity-levels

Going to close this as we don't currently have plans to change how severity is defined (Though @typpo's suggestion above may help)