Repository: promptfoo/promptfoo
Issue: Cannot test complex tools that use $refs and $defs
Root cause: Validation Gap

When trying to test a non-trivial tool, YAML parsing of `promptfooconfig.yaml` fails, because anchors inside the tool definition are parsed as anchors inside the YAML file itself.

For example, the following file cannot be parsed:

```
prompts:
  - file://prompts/*.txt

providers:
  - openai:gpt-4:
      config:
        tools:
          [
            {
              "type": "function",
              "function":
                {
                  "name": "kubectl_describe",
                  "description": "Run kubectl describe on a Kubernetes resource",
                  "parameters":
                    {
                      "$defs":
                        {
                          "KubernetesResourceKind":
                            {
                              "enum":
                                [
                                  "deployment",
                                  "node",
                                  "pod",
                                  "replicaset",
                                  "daemonset",
                                  "statefulset",
                                  "job",
                                  "service",
                                  "other",
                                ],
                              "title": "KubernetesResourceKind",
                              "type": "string",
                            },
                        },
                      "properties":
                        {
                          "kind": { "$ref": "#/$defs/KubernetesResourceKind" },
                          "namespace":
                            {
                              "anyOf": [{ "type": "string" }, { "type": "null" }],
                              "default": None,
                              "title": "Namespace",
                            },
                          "name": { "title": "Name", "type": "string" },
                        },
                      "required": ["kind", "name"],
                      "title": "KubectlDescribe",
                      "type": "object",
                    },
                },
            },
          ]
```

Running `promptfoo eval` yields an error:

```
/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/pointer.js:81
                throw new errors_js_1.MissingPointerError(token, decodeURI(this.originalPath));
                      ^

MissingPointerError: Token "$defs" does not exist.
    at Pointer.resolve (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/pointer.js:81:23)
    at $Ref.resolve (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/ref.js:82:28)
    at $Refs._resolve (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/refs.js:159:21)
    at dereference$Ref (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:152:27)
    at crawl (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:86:40)
    at crawl (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:98:44)
    at crawl (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:98:44)
    at crawl (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:98:44)
    at crawl (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:98:44)
    at crawl (/usr/local/lib/node_modules/promptfoo/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/dereference.js:98:44) {
  code: 'EUNMATCHEDRESOLVER',
  source: '/Users/natanyellin/Projects/rv2-ai/',
  path: null,
  toJSON: [Function: toJSON],
  [Symbol(nodejs.util.inspect.custom)]: [Function: inspect]
}

Node.js v21.5.0
```

Is there a way to work around this?

Thanks for the catch! #365 prevents dereferencing of tools and functions, and also adds a flag to disable dereferencing behavior entirely.  This fix will be included in the next release.

Awesome, thanks.

Fix is in 0.34.0