Repository: promptfoo/promptfoo
Issue: Silent failure by promptRubric not imputing variables or ignoring `value`
Root cause: Configuration Error

**Describe the bug**
The promptRubric is ignored or the imputed parameters are not inserted. I can't tell because the logs (with `--verbose`) are insufficient.

To reduce likelihood of my error, I searched for existing issues and copied `promptRubric` from an issue that is resolved (#823). Doing eval on the same YAML (only difference is provider), it silently fails (doesn't produce expected result) and from LLM output I hypothesize that the LLM didn't receive the full prompt. This can be reproduced even with simpler prompts, but I wanted to be sure that the YAML is 100% correct.

**To Reproduce**
Take this YAML as from #823 (modified only the provider):
```
prompts: >-
  [{
    "role": "system",
    "content": "Translate text into {{ language }}."
  }, {
    "role": "user",
    "content": "{{ input }}"
  }]
providers: vertex:gemini-1.5-flash
defaultTest:
  options:
    rubricPrompt:
      - role: system
        content: >-
          Evaluate the quality of the translation provided by an AI assistant to the user input displayed below.
          Score the response on a scale of 0 to 10.
          Output your response in the following JSON format: {pass: bool, score: number, reason: string}
      - role: user
        content: >-
          [Input Start]
          {{ input }}
          [Input End]

          [Translation Start]
          {{ output }}
          [Translation End]

tests:
  - vars:
      input: "Happy families are all alike. \nAnd every unhappy family is unhappy in its own way."
      language: "Chinese"
    assert:
      - type: llm-rubric
```
and run `promptfoo eval` on it.

**Expected behavior**
The promptRubric should rate the translation quality. Instead, it responds: `{"pass":false,"reason":"No rubric was provided","score":0,"tokensUsed":{"total":201,"prompt":179,"completion":22,"cached":0}}`

**Screenshots**
```
Reading prompts from ["[{\n  \"role\": \"system\",\n  \"content\": \"Translate text into {{ language }}.\"\n}, {\n  \"role\": \"user\",\n  \"content\": \"{{ input }}\"\n}]"]
Inserting prompt 3675bb3253267004907c89fcb4eaa51cbc0f6e2d963b914b11fa7e1497969feb
Inserting dataset 14998238815e9383d52ac592abda36278c157fa1fe9ce420837809599f62d532
Coerced JSON prompt to Gemini format: [{"role":"system","content":"Translate text into Chinese."},{"role":"user","content":"Happy families are all alike. \nAnd every unhappy family is unhappy in its own way."}]
Preparing to call Google Vertex API (Gemini) with body: {"contents":{"role":"user","parts":{"text":"Translate text into Chinese.Happy families are all alike. \nAnd every unhappy family is unhappy in its own way."}},"generationConfig":{}}
Gemini API response: [{"candidates":[{"content":{"role":"model","parts":[{"text":"幸福"}]}}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":"的家庭都一样。\n不幸的家庭各有各的不幸。 \n"}]},"safetyRatings":[{"caY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.17480469,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.083984375},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.040283203,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.02368164},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.25585938,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.111328125},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.087402344,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.04272461}]}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"avgLogprobs":"NaN"}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":24,"candidatesTokenCount":17,"totalTokenCount":41},"modelVersion":"gemini-1.5-flash"}]
Gemini API response: [{"candidates":[{"content":{"role":"model","parts":[{"text":"幸福"}]}}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":"的家庭都一样。\n不幸的家庭各有各的不幸。 \n"}]},"safetyRatings":[{"caY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.17480469,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.083984375},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.040283203,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.02368164},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.25585938,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.111328125},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.087402344,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.04272461}]}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"avgLogprobs":"NaN"}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":24,"candidatesTokenCount":17,"totalTokenCount":41},"modelVersion":"gemini-1.5-flash"}]
Performing remote grading: {"task":"llm-rubric","rubric":"","output":"幸福的家庭都一样。\n不幸的家庭各有各的不幸。 \n","vars":{"input":"Happy families are all alike. \nAnd every unhappy family is unhappy in its own way.","language":"Chinese"}}
Got remote grading result: {"pass":false,"reason":"No rubric was provided","score":0,"tokensUsed":{"total":201,"prompt":179,"completion":22,"cached":0}}
```

**System information:**

- Promptfoo version: 0.92.2

Hi @strentom, definitely a bug. Just clarifying, are you running into this error with a redteam config?  Based on the logs, you are hitting a redteam codepath which is different from the example provided and different from the behavior when I run it locally.  

Hi @typpo . I’m running ‘promptfoo eval -c script_above.yaml —verbose’ . I have other files and configs in the folder (incl. redteam related) but these should be ignored. 

Got it, this is helpful thanks.  #1877 should fix the immediate issue.  There is another issue of your redteam config being picked up when it shouldn't.

@typpo, Thank you for the commits. I re-ran the yaml with 0.93.0. The main issue is fixed. However, I still see two possible problems. The test runs, the input is translated into Chinese but LLM rubric fails with `[FAIL] Could not extract JSON from llm-rubric response`. 

(1) Firstly, 
```
Returning cached response: {"cached":false,"output":"```json\n{\n\"pass\": true,\n\"score\": 9,\n\"reason\": \"The translation is accurate and natural-sounding. It captures the essence of the original English text, which is a famous opening line from Anna Karenina by Leo Tolstoy. The translator correctly identifies and conveys the nuances of the original text, including the use of parallelism and the contrast between \"alike\" and \"in its own way.\" \n\"\n}\n``` \n","tokenUsage":{"total":204,"prompt":108,"completion":96}}
```
It looks like JSON to me; maybe the parsing got confused by ` ```json ` at the beginning. One option is to use [Constrained generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output) in vertexAI which guarantees JSON output. In python for example, Langchain can deserialize json into a native class. Another quick fix is to use a [json-repair](https://pypi.org/project/json-repair/) package (exists for js too).

(2) Secondly, I re-ran the test (`--no-cache`). This time, LLM generated correct JSON output but I'm confused by the logs. It seems instead of using Gemini-1.5-Flash as specified in the YAML, Gemini-1.5-Pro was used for evaluation!
```
Gemini API response: [{"candidates":[{"content":{"role":"model","parts":[{"text":"幸福"}]}}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":"的家庭都一样，不幸的家庭各有各的不幸。 \n"}]},"safetyRatings":[{"cateHATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.20410156,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.09667969},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.04272461,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.028442383},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.29296875,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.123535156},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.09033203,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.04272461}]}],"modelVersion":"gemini-1.5-flash"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":24,"candidatesTokenCount":16,"totalTokenCount":40},"modelVersion":"gemini-1.5-flash"}]
Using Google default providers
Coerced JSON prompt to Gemini format: [{"role":"system","content":"Evaluate the quality of the translation provided by an AI assistant to the user input displayed below. Score the response on a scale of 0 to 10. Output your response in the following JSON format: {pass: bool, score: number, reason: string}"},{"role":"user","content":"[Input Start] Happy families are all alike. \nAnd every unhappy family is unhappy in its own way. [Input End]\n[Translation Start] 幸福的家庭都一样，不幸的家庭各有各的不幸。 \n [Translation End]"}]
Preparing to call Google Vertex API (Gemini) with body: {"contents":{"role":"user","parts":{"text":"Evaluate the quality of the translation provided by an AI assistant to the user input displayed below. Score the response on a scale of 0 to 10. Output your response in the following JSON format: {pass: bool, score: number, reason: string}[Input Start] Happy families are all alike. \nAnd every unhappy family is unhappy in its own way. [Input End]\n[Translation Start] 幸福的家庭都一样，不幸的家庭各有各的不幸。 \},"generationConfig":{}}
Gemini API response: [{"candidates":[{"content":{"role":"model","parts":[{"text":"```"}]}}],"modelVersion":"gemini-1.5-pro"},{"candidates":[{"content":{"role":"model","parts":[{"text":"json\n{\n  \"pass\": true,\n  \"score\": "}]},"safetyRatings":[{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.111328125,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.09667969},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.080566406,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.06347656},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.15820313,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.053466797},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.04736328,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.05029297}]}],"modelVersion":"gemini-1.5-pro"},{"candidates":[{"content":{"role":"model","parts":[{"text":"9,\n  \"reason\": \"This is an accurate and natural-sounding"}]},"safetyRatings":[{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.10498047,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.10253906},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.05493164,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.03955078},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.14160156,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.061035156},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.03955078,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.06347656}]}],"modelVersion":"gemini-1.5-pro"},{"candidates":[{"content":{"role":"model","parts":[{"text":" translation of the famous opening lines of Anna Karenina. The translator has correctly captured the nuances of the original meaning and expressed it fluently in Chinese. The use"}]},"safetyRatings":[{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.072753906,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.07470703},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.037353516,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.03466797},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.080566406,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.057373047},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.07714844,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.10839844}]}],"modelVersion":"gemini-1.5-pro"},{"candidates":[{"content":{"role":"model","parts":[{"text":" of  '都一样or 'all alike' and '各有各的' for 'in its own way' is particularly effective.\"\n}\n```"}]},"safetyRatings":[{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.071777344,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.072753906},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE","probabilityScore":0.026000977,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.03112793},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.07080078,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.061035156},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.05834961,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.071777344}]}],"modelVersion":"gemini-1.5-pro"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"avgLogprobs":"NaN"}],"modelVersion":"gemini-1.5-pro"},{"candidates":[{"content":{"role":"model","parts":[{"text":""}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":105,"candidatesTokenCount":97,"totalTokenCount":202},"modelVersion":"gemini-1.5-pro"}]
```
Where does the `pro` come from?

Hi @strentom, the main issue (rubricPrompt being ignored with "No rubric was provided") was fixed in #1877 (v0.93.0).

The follow-up issues you mentioned in your last comment are separate:
1. **JSON parsing with markdown code blocks** - LLMs sometimes wrap JSON in backticks. This can be addressed by using `responseSchema` for structured output or by improving prompt instructions.
2. **Wrong model for grading** - The default grading provider may differ from your evaluation provider. You can explicitly set the grading provider via `defaultTest.options.provider`.

Since the original issue is resolved, I'm closing this. Feel free to open new issues for the follow-up concerns if they're still affecting you.