Repository: promptfoo/promptfoo
Issue: Custom HTTP grader gets empty string in request body for llm-rubric eval
Root cause: Configuration Error

**Describe the bug**
I want to use a custom HTTP provider as a grader in an LLM-rubric evaluation, the endpoint is successfully called, however the `{{rubric}}` variable is resolved as an empty string. If I understood the documentation correctly, then the llm-rubric type assert's value is the `{{rubric}}` parameter.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:
I defined the provider under `defaultTest`
```yaml
defaultTest:
  options:
    provider:
      id: https
      config:
        url: 'https://my-llm.com/endpoint'
        method: POST
        headers:
          'Content-Type': 'application/json'
        body:
          messages:
            - role: user
              content:
                - type: text
                  content: '{{rubric}}'
  assert:
    - type: llm-rubric
      value: Rate the output on a scale of -1 to 1 based on how funny it is

prompts:
  - Write a joke about cats.
```
1. Run 'promptfoo eval -v'
2. In the logs I see that that the `content` is empty:
`[http.js:"590] [HTTP Provider]: Calling .... "messages": [{"role":"user", "content":[{"type": "text", "content":""}] ....`

**Expected behavior**
The `{{rubric}}` should be resolved in the request and sent to the grader

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

```
[http.js:600] [HTTP Provider]: Response: {"error": "An error occurred (ValidationException) when calling the InvokeModel operation: messages.0.content.0.text.text: Field required"}
Error parsing response from provider: TypeError: Cannot read properties of undefined (reading '0')
  at eval (eval at <anonymus> (/usr/local/lib/node_modules/promptfoo/dist/src/providers/http.js:216:37, <anonymus>:14:31)
  at eval (eval at <anonymus> (/usr/local/lib/node_modules/promptfoo/dist/src/providers/http.js:216:37, <anonymus>:14:3)
  at /usr/local/lib/node_modules/promptfoo/dist/src/providers/http.js:221:28
  at HttpProvider.callApi (/usr/local/lib/node_modules/promptfoo/dist/src/providers/http.js:632:60)
  at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
  at async matchesLlmRubric (/usr/local/lib/node_modules/promptfoo/dist/src/matchers.js:328:18)
  at async runAssertion (/usr/local/lib/node_modules/promptfoo/dist/src/assertions/index.js:227:24)
  at async /usr/local/lib/node_modules/promptfoo/dist/src/assertions/index.js:227:24
```

- MacOS version: 
Sequoia 15.4
- Promptfoo version:
0.110.0

**Additional context**
Add any other context about the problem here.
I can't share the exact provider due to company policy, but it should be possible to reproduce with any LLM exposed though e.g. an API Gateway.

I also tried overriding the `rubricPrompt`, or passing the `{{output}}` and other vars from the test, but none of them got properly resolved in my request.

Hi, do you solve the problem?

Hi, same problem here! Custom LLM grader doesn't work for me. 

I believe I'm having having the same issue! Anyone else discover any work arounds?

Hey @vira-s, @OCEANOUXIN, @gaborerdosi76, and @dbouquin! 👋

Thanks for reporting this issue and for your patience. I did a deep dive into this and have good news!

## TL;DR

This issue appears to be **fixed** in recent versions of promptfoo. The `{{rubric}}` and `{{output}}` variables now correctly resolve when using a custom HTTP provider as a grader.

## Investigation Summary

I traced through the entire code path and ran end-to-end tests with a real evaluation. Here's what I found:

**Test config using HTTP provider as llm-rubric grader:**

```yaml
defaultTest:
  options:
    provider:
      id: https://api.openai.com/v1/chat/completions
      config:
        method: POST
        headers:
          Content-Type: application/json
          Authorization: 'Bearer {{env.OPENAI_API_KEY}}'
        body:
          model: gpt-4o-mini
          messages:
            - role: system
              content: You are a grading assistant. Evaluate the output based on the rubric.
            - role: user
              content: |
                RUBRIC: {{rubric}}
                OUTPUT TO EVALUATE: {{output}}
        responseParser: 'json.choices[0].message.content'

tests:
  - vars:
      topic: cats
    assert:
      - type: llm-rubric
        value: 'The response should be a joke about cats.'
```

**Result:** Both `{{rubric}}` and `{{output}}` were correctly substituted in the HTTP request body. The eval passed with a 100% pass rate.

## What to Do

If you're still experiencing this issue, please:

1. **Upgrade to the latest version:** `npm install -g promptfoo@latest`
2. **Run with `--no-cache`** to ensure fresh results
3. **Try the config above** as a starting point

If the issue persists after upgrading, please let us know:
- Your promptfoo version (`promptfoo --version`)
- Your full config (redact any secrets)
- Verbose output (`promptfoo eval --verbose`)

We're happy to reopen this if needed!

---

Thanks again for taking the time to report this. Bug reports like yours help make promptfoo better for everyone.


Thank you @mldangelo for the update and good news! I'll test it in the following days and come back if anything is amiss.