Repository: promptfoo/promptfoo
Issue: Failing to use custom LLM rubric provider (error: Cannot read properties of undefined 'startsWith')
Root cause: Validation Gap

Thank you for building a fantastic tool!

**Describe the bug**
I'm trying to use LLM rubric with a custom provider and it errors with a cryptic message "Cannot read properties of undefined 'startsWith')"

**To Reproduce**

I have the following promptfoo config
```yaml
# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "My eval"

prompts:
  - file://prompt.py:code_reviewer

providers:
  - "exec: node complete.js claude-3.5-sonnet"
  # - "exec: node complete.js gemini-1.5-pro"
  - "exec: node complete.js gpt-4o"

defaultTest:
  options:
    provider: "exec: node complete.js claude-3.5-sonnet"
  assert:
    - description: Evaluate output using LLM
      assert:
        - type: llm-rubric
          value: Is concise
tests:
  - vars:
      dir: test-dir/javascript
  - vars:
      dir: test-dir/java


```

When I run `pnpm exec promptfoo eval`, I get the following stacktrace
```
TypeError: Cannot read properties of undefined (reading 'startsWith')

TypeError: Cannot read properties of undefined (reading 'startsWith')
    at /Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/promptfoo@0.91.1_@aws-sdk+client-bedrock-runtime@3.658.1_@azure+identity@4.4.1_@azure+openai-_3fdhvawzhz4grxwyrvnbbxjoim/node_modules/promptfoo/dist/src/assertions.js:1151:28
    at /Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/async@3.2.6/node_modules/async/dist/async.js:151:38
    at replenish (/Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/async@3.2.6/node_modules/async/dist/async.js:447:21)
    at /Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/async@3.2.6/node_modules/async/dist/async.js:452:13
    at Object.eachOfLimit (/Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/async@3.2.6/node_modules/async/dist/async.js:478:36)
    at /Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/async@3.2.6/node_modules/async/dist/async.js:220:25
    at new Promise (<anonymous>)
    at Object.awaitable (/Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/async@3.2.6/node_modules/async/dist/async.js:215:20)
    at runAssertions (/Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/promptfoo@0.91.1_@aws-sdk+client-bedrock-runtime@3.658.1_@azure+identity@4.4.1_@azure+openai-_3fdhvawzhz4grxwyrvnbbxjoim/node_modules/promptfoo/dist/src/assertions.js:1150:27)
    at Evaluator.runEval (/Users/olafurpg/dev/sourcegraph/openapi/examples/prompt-evals/node_modules/.pnpm/promptfoo@0.91.1_@aws-sdk+client-bedrock-runtime@3.658.1_@azure+identity@4.4.1_@azure+openai-_3fdhvawzhz4grxwyrvnbbxjoim/node_modules/promptfoo/dist/src/evaluator.js:244:74)
```

**Expected behavior**

I expected promptoo to use the customized provider.

**System information (please complete the following information):**

- Promptfoo version: 0.91.1
- OS: macOS
- Node: v20.4.0

**Additional context**

The docs don't mention that you can customize LLM rubrics with `exec` providers so feel free to close this as wontfix if it's just not a supported feature. For compliance reasons, I'm unable to use the off-the-shelf providers so this limitation would prevent me from using LLM rubric.




Hi @olafurpg,

Thank you for the kind words and for taking the time to report this! Luckily, I believe we already support this. It looks like your issue stems from a YAML syntax error that we are not catching in our schema validator. Specifically, the `llm-rubric` assertion needs to be structured slightly differently in your configuration. Could you try updating it like this:

```yaml
defaultTest:
  options:
    provider: "exec: node complete.js claude-3.5-sonnet"
  assert:
    - description: Evaluate output using LLM
      type: llm-rubric
      value: Is concise
```

Let me know if this works or if you encounter any other issues. Thanks again for bringing this to our attention! I will add improving the validation logic to our backlog.

Hi @olafurpg, I’m going to close this issue. If you're still experiencing the problem or have any additional questions, feel free to comment or re-open. Thanks again for your feedback and for using Promptfoo!