Repository: promptfoo/promptfoo
Issue: Template render error
Root cause: Validation Gap


Getting this error

 err = new Error(old.message);
          ^

Template render error: (unknown path) [Line 1, Column 131]
  tag name expected
    at Object._prettifyError (/usr/local/lib/node_modules/promptfoo/node_module/
    at Template.render (/usr/local/lib/node_modules/promptfoo/node_modules/nunj/
    at Environment.renderString (/usr/local/lib/node_modules/promptfoo/node_mod/
    at Object.renderString (/usr/local/lib/node_modules/promptfoo/node_modules//
    at renderVarsInObject (/usr/local/lib/node_modules/promptfoo/dist/src/util/i
    at renderPrompt (/usr/local/lib/node_modules/promptfoo/dist/src/evaluatorHel
    at runEval (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:143:
    at processEvalStep (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator
    at async /usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:858:13


Can you please provide a full repro?  Hard to know what's wrong with just this stack trace.

promptfoo redteam generate

Test Generation Report:
`
┌─────┬──────────┬────────────────────────────────────────┬────────────┬────────────┬──────────────┐
│ #   │ Type     │ ID                                     │ Requested  │ Generated  │ Status       │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 1   │ Plugin   │ contracts                              │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 2   │ Plugin   │ excessive-agency                       │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 3   │ Plugin   │ hallucination                          │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 4   │ Plugin   │ harmful:chemical-biological-weapons    │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 5   │ Plugin   │ harmful:child-exploitation             │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 6   │ Plugin   │ harmful:copyright-violations           │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 7   │ Plugin   │ harmful:cybercrime                     │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 8   │ Plugin   │ harmful:cybercrime:malicious-code      │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 9   │ Plugin   │ harmful:graphic-content                │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 10  │ Plugin   │ harmful:harassment-bullying            │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 11  │ Plugin   │ harmful:hate                           │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 12  │ Plugin   │ harmful:illegal-activities             │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 13  │ Plugin   │ harmful:illegal-drugs                  │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 14  │ Plugin   │ harmful:illegal-drugs:meth             │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 15  │ Plugin   │ harmful:indiscriminate-weapons         │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 16  │ Plugin   │ harmful:insults                        │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 17  │ Plugin   │ harmful:intellectual-property          │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 18  │ Plugin   │ harmful:misinformation-disinformation  │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 19  │ Plugin   │ harmful:non-violent-crime              │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 20  │ Plugin   │ harmful:privacy                        │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 21  │ Plugin   │ harmful:profanity                      │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 22  │ Plugin   │ harmful:radicalization                 │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 23  │ Plugin   │ harmful:self-harm                      │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 24  │ Plugin   │ harmful:sex-crime                      │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 25  │ Plugin   │ harmful:sexual-content                 │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 26  │ Plugin   │ harmful:specialized-advice             │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 27  │ Plugin   │ harmful:unsafe-practices               │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 28  │ Plugin   │ harmful:violent-crime                  │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 29  │ Plugin   │ harmful:weapons:ied                    │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 30  │ Plugin   │ hijacking                              │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 31  │ Plugin   │ pii:api-db                             │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 32  │ Plugin   │ pii:direct                             │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 33  │ Plugin   │ pii:session                            │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 34  │ Plugin   │ pii:social                             │ 5          │ 0          │ Failed       │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 35  │ Plugin   │ politics                               │ 5          │ 5          │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 36  │ Strategy │ gcg                                    │ 170        │ 170        │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 37  │ Strategy │ hex                                    │ 170        │ 170        │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 38  │ Strategy │ jailbreak                              │ 170        │ 170        │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 39  │ Strategy │ jailbreak:composite                    │ 170        │ 850        │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 40  │ Strategy │ jailbreak:likert                       │ 170        │ 170        │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 41  │ Strategy │ leetspeak                              │ 170        │ 170        │ Success      │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 42  │ Strategy │ pandamonium                            │ 170        │ 1          │ Partial      │
└─────┴──────────┴────────────────────────────────────────┴────────────┴────────────┴──────────────┘`

Wrote 1871 test cases to redteam.yaml


promptfoo eval -c redteam.yaml


after few minutes the error occurs 

Can you share the `promptfooconfig.yaml` and/or the prompt file (if applicable)?

@typpo  


```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: My Red Team Configuration
targets:
  - id: http
    config:
      request: |-
        POST /api/chat/send-message HTTP/2
        Host: xxx.com
        Cookie: _ga=xxxxxx
        User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0
        Accept: */*
        Accept-Language: en-US,en;q=0.5
        Accept-Encoding: gzip, deflate, br
        Referer: xxxxx
        Content-Length: 422
        Origin: xxxx
        Sec-Fetch-Dest: empty
        Sec-Fetch-Mode: cors
        Sec-Fetch-Site: same-origin
        Priority: u=0
        Te: trailers

        {"alternate_assistant_id":0,"chat_session_id":"d4911d6d-981d-4e2c-89e9-eb5607fed2ac","parent_message_id":2221,"message":"{{prompt}}","prompt_id":0,"search_doc_ids":null,"file_descriptors":[],"regenerate":false,"retrieval_options":{"run_search":"auto","real_time":true,"filters":{"source_type":null,"document_set":null,"time_cutoff":null,"tags":[]}},"prompt_override":null,"llm_override":null}
      validateStatus: ''
    label: test
prompts:
  - '{{prompt}}'
redteam:
  purpose: ''
  plugins:
    - id: contracts  # Tests for unauthorized contractual commitments and legal exposure
    - id: excessive-agency  # Tests for unauthorized actions beyond defined system boundaries
    - id: hallucination  # Tests for fabrication of false or misleading information
    - id: harmful:chemical-biological-weapons  # Tests handling of WMD-related content
    - id: harmful:child-exploitation  # Tests handling of child exploitation content
    - id: harmful:copyright-violations  # Tests handling of intellectual property violations
    - id: harmful:cybercrime  # Tests handling of cybercrime-related content
    - id: harmful:cybercrime:malicious-code  # Tests creation of malicious code
    - id: harmful:graphic-content  # Tests handling of graphic or violent content
    - id: harmful:harassment-bullying  # Tests handling of harassment and bullying content
    - id: harmful:hate  # Tests handling of hate speech and discriminatory content
    - id: harmful:illegal-activities  # Tests handling of general illegal activities
    - id: harmful:illegal-drugs  # Tests handling of illegal drug-related content
    - id: harmful:illegal-drugs:meth  # Tests handling of methamphetamine-related content
    - id: harmful:indiscriminate-weapons  # Tests handling of weapons-related content
    - id: harmful:insults  # Tests handling of personal attacks and insults
    - id: harmful:intellectual-property  # Tests handling of IP theft and violations
    - id: harmful:misinformation-disinformation  # Tests handling of false information campaigns
    - id: harmful:non-violent-crime  # Tests handling of non-violent criminal content
    - id: harmful:privacy  # Tests handling of privacy violation attempts
    - id: harmful:profanity  # Tests handling of profane or inappropriate language
    - id: harmful:radicalization  # Tests handling of extremist content and recruitment
    - id: harmful:self-harm  # Tests handling of self-harm and suicide content
    - id: harmful:sex-crime  # Tests handling of sexual crime content
    - id: harmful:sexual-content  # Tests handling of explicit sexual content
    - id: harmful:specialized-advice  # Tests handling of unauthorized professional advice
    - id: harmful:unsafe-practices  # Tests handling of dangerous activity instructions
    - id: harmful:violent-crime  # Tests handling of violent crime content
    - id: harmful:weapons:ied  # Tests handling of Improvised Explosive Devices
    - id: hijacking  # Tests for unauthorized resource usage and purpose deviation
    - id: pii:api-db  # Tests for PII exposure via API/database access
    - id: pii:direct  # Tests for direct PII exposure vulnerabilities
    - id: pii:session  # Tests for PII exposure in session data
    - id: pii:social  # Tests for PII exposure via social engineering
    - id: politics  # Tests handling of political content and bias
  strategies:
    - id: jailbreak  # Single-shot optimization of safety bypass techniques
    - id: jailbreak:composite  # Combines multiple jailbreak techniques for enhanced effectiveness
    - id: jailbreak:likert  # Uses Likert scale-based prompts to bypass content filters
    - id: hex  # Tests handling of hex-encoded malicious payloads
    - id: gcg  # Greedy Coordinate Gradient adversarial suffix attack
    - id: leetspeak  # Tests handling of leetspeak-encoded malicious content
    - id: pandamonium  # [Experimental] Promptfoo's blackbox jailbreak. The latest and greatest jailbreak techniques, expertly tuned and crafted by Promptfoo's AI team.
defaultTest:
  options:
    transformVars: '{ ...vars, sessionId: context.uuid }'
```

After updating to the latest version I am getting this error 

`/usr/local/lib/node_modules/promptfoo/node_modules/nunjucks/src/lib.js:32swer "
    err = new Error(old.message);
          ^

Template render error: (unknown path) [Line 1, Column 156]
  unknown block tag: lean
    at Object._prettifyError (/usr/local/lib/node_modules/promptfoo/node_module/
    at Template.render (/usr/local/lib/node_modules/promptfoo/node_modules/nunj/
    at Environment.renderString (/usr/local/lib/node_modules/promptfoo/node_mod/
    at Object.renderString (/usr/local/lib/node_modules/promptfoo/node_modules//
    at renderVarsInObject (/usr/local/lib/node_modules/promptfoo/dist/src/util/i
    at renderPrompt (/usr/local/lib/node_modules/promptfoo/dist/src/evaluatorHel
    at runEval (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:143:
    at processEvalStep (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator
    at async /usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:858:13`

@d1pakda5 please throw some time on my calendar if you would like help debugging https://cal.com/michael-dangelo/15min