Repository: promptfoo/promptfoo
Issue: JSON Schema validation errors for custom fields (assertionTemplates & $ref)
Root cause: Validation Gap

**Describe the bug**
When using promptfooconfig.yaml with the YAML Language Server schema reference (`$schema=https://promptfoo.dev/config-schema.json`), VS Code and other editors display validation errors for:

1. Custom top-level fields like `assertionTemplates`
2. `$ref` syntax used to reference templates

These features are documented in official docs and work perfectly at runtime, but show validation errors in editors.

**To Reproduce**
1. Create a `promptfooconfig.yaml` file with the schema reference:
   ```yaml
   # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
   ```

2. Add the following configuration (from [official documentation](https://promptfoo.dev/docs/configuration/expected-outputs/#reusing-assertions-with-templates)):
   ```yaml
   prompts:
     - "Test {{input}}"

   providers:
     - openai:gpt-4

   # ❌ Editor shows error: "Property assertionTemplates is not allowed"
   assertionTemplates:
     containsPirate:
       type: icontains
       value: arrr

   tests:
     - assert:
         # ❌ Editor shows error: "Property $ref is not allowed"
         - $ref: '#/assertionTemplates/containsPirate'
   ```

3. Observe red squiggly lines under `assertionTemplates` and `$ref` in VS Code

4. Run `promptfoo eval` - it works perfectly with no errors

This also affects the official example: [examples/simple-test/promptfooconfig.yaml](https://github.com/promptfoo/promptfoo/blob/main/examples/simple-test/promptfooconfig.yaml)


**Expected behavior**

- No schema validation errors in editors
- `assertionTemplates` and similar custom fields should be valid
- `$ref` should be recognized as a valid field in assertions and other objects


**Screenshots**

<img width="768" height="224" alt="Image" src="https://github.com/user-attachments/assets/283d3ddc-b40e-4fba-b13f-c2ca3ff4bea1" />

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.118.14

**Additional context**

To fix this issue, the JSON Schema needs to be modified. Two approaches can be implemented via post-processing in `scripts/generateJsonSchema.ts`:

### Option 1: Global `additionalProperties: true`

Allow any field anywhere - maximum flexibility but no typo detection.

Pros: Simple, any field name works
Cons: No typo detection (e.g., `prviders` would be silently ignored)

### Option 2: Add `$ref` as union + limit extension fields with `x-` prefix

Add `$ref` as a valid field everywhere (union type) and allow `x-` prefixed fields at top level.

The `x-` prefix is used in [OpenAPI](https://spec.openapis.org/oas/v3.1.0#specification-extensions) and [Docker Compose](https://docs.docker.com/compose/compose-file/11-extension/) for extension fields.

```typescript
// scripts/generateJsonSchema.ts
const jsonSchema = zodToJsonSchema(UnifiedConfigSchema);

// Add $ref to all object schemas as union
addRefToAllObjects(jsonSchema);

// Allow x- prefix at top level
schema.patternProperties = {
  '^x-': {},
};
schema.additionalProperties = false; // Keep typo detection
```

Pros: Typo detection maintained, clear naming convention
Cons: Requires `x-` prefix (documentation needs update), slightly more complex implementation

### Zod v4 Migration

Currently using Zod v3 + `zod-to-json-schema`.

If Zod v4 migration ([#5692](https://github.com/promptfoo/promptfoo/pull/5692)) is challenging, this fix could be applied now with Zod v3. Alternatively, if migration is planned soon, it might be easier to address this issue after migrating to Zod v4.

---

I'm happy to submit a PR if a direction is agreed upon.