Repository: promptfoo/promptfoo
Issue: [BUG] crescendo redteam provider doesn't transform query correctly
Root cause: Documentation Deficiency

**Describe the bug**
I'm trying to configure my red-teaming config to send requests to an openai-compatible vLLM server hosting a llamaguard model, based on the [red-teaming configuration](https://www.promptfoo.dev/docs/red-team/configuration/) docs, and the [HTTP provider documents](https://www.promptfoo.dev/docs/providers/http/). I'm sending in a JSON request via my request body - but it appears that it isn't expected by promptfoo.  
Here's my red-teaming configuration: 

```
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: My Red Team Configuration
targets:
  - id: http
    config:
      url: http://0.0.0.0:6969/v1/chat/completions
      method: post
      Content-Type: application/json
      body:
        model: /checkpoints/postsft_v2_beavertails-rlhf-dpo-filtered_8_epoch/
        messages: '{{prompt}}'
        max_tokens: 50
        temperature: 0.7
        top_p: 0.9
        'n': 1
        stream: false
        stop:
          - |+

      transformResponse: json.choices[0].message.content
    label: '''amzn-prize'''
redteam:
  provider:
    id: http
    config:
      url: http://0.0.0.0:1420/v1/chat/completions
      method: POST
      headers:
        Content-Type: application/json
      body:
        model: meta-llama/Llama-Guard-3-8b
        messages:
          - role: user
            content: '{{prompt}}'
        max_tokens: 50
        temperature: 0.7
        top_p: 0.9
        'n': 1
        stream: false
        stop:
          - |+

      transformRequest: 'file://request.js'
      transformResponse: (json, text) => json.choices[0].message.content

  plugins:
    - id: beavertails
  purpose: Evaluate and test language model responses to various prompts
  strategies:
    - id: crescendo
      config:
        maxTurns: 2
```

However, promptfoo throws me this error: 
```
Invalid options:
  Invalid output from extraction. Got: (json, text) => json.choices[0].message.content
```
Any guidance on what I'm doing wrong/what can I do would be appreciated!  

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. create a config as above and save it as `redteam.yaml` in the CWD.
2. run `promptfoo redteam run -c redteam.yaml --verbose`


**Expected behavior**
The redteaming strategy should query my hosted instance based on the config


**System information:**

- Promptfoo version: 0.103.14



It also appears that the jailbreak and default strategies completely ignores the `redteam.provider` field. I'm not sure why.

Hi @AetherPrior, thanks for reaching out!  

There are two issues with your setup:  

1. **Llama Guard is a classifier**, not a generative model, so it may not be suitable as a red-team provider since it doesn’t generate adversarial inputs.  
2. **Your `transformResponse` should work** if you update it to:  
   ```yaml
   transformResponse: json.choices[0].message.content

I'll update our documentation to reflect this and add support for the format you used today.

Regarding your second question, could you share more details about your setup? I'd be happy to take a look.

Thanks!

Thanks for getting back to me and for providing pointers for the provider! 
Re my second question -  I realized what I was doing wrong here - I'm actually trying to modify the LLM scorer function in the Red-teaming setup. I would like to use LlamaGuard as my classifier as I'd prefer my evaluations be done with open-source models which I would wish to host on my own compute. I couldn't find how to do this in the documentation or while browsing the source so pointers here would be super helpful for me! 

I don't think we support using custom graders at the moment. You can use the LlamaGuard through [moderation](https://www.promptfoo.dev/docs/configuration/expected-outputs/moderation/) though!

thanks for answering my questions. That's unfortunate that custom graders aren't supported, as I do not have access to OpenAI APIs and would like an Opensource alternative. Thanks though for getting back to me. 

Just to clarify, you can definitely do this by overriding the red team provider. This will work for most strategies such as Crescendo. 
 https://www.promptfoo.dev/docs/red-team/configuration/#changing-the-model

We just don't have a way to edit and customize the prompts that get sent to the provider.

Hi, this is where I'm confused - I'm looking to change the custom scorer, not the adversarial generator. It doesn't look like that's handled in the red-team provider? 

The redteam provider should also determine the custom scorer https://github.com/promptfoo/promptfoo/blob/main/src/redteam/plugins/base.ts#L298-L303

If you'd like to update just the scorer and not the adversarial generator, you can try setting a `defaultTest` key like this: https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader

```yaml
defaultTest:
  options:
    provider:
      text:
        id: ollama:...
```

That will override the custom grader.  I'm not totally sure offhand how this will interact with strategies (which use their own custom grader), so I would recommend just setting `redteam.provider`.