Repository: promptfoo/promptfoo
Issue: Promptfoo Provider Config Parameters Not Being Loaded - Only Environment Variables Work
Root cause: Documentation Deficiency

# Issue: Promptfoo Provider Config Parameters Not Being Loaded - Only Environment Variables Work

## Description
When running Promptfoo evaluations, the provider configuration parameters defined in `promptfooconfig.yaml` are not being properly loaded or applied to API requests. The configuration only works when set via environment variables (e.g., `OPENAI_MAX_TOKENS=10000`), causing tests to fail with empty response arrays `[]` when responses hit the default token limit.

## Environment
- **Promptfoo Version**: 0.119.14
- **Platform**: Linux 5.15.0-136-generic
- **Architecture**: x64
- **Node Version**: v22.14.0

## Command Used
```bash
$ promptfoo eval --no-cache --env-file .env
```

## Problem Summary

### Configuration in YAML (Not Working ❌)
```yaml
providers:
  - id: openai:chat:openai/gpt-oss-120b
    config:
      apiBaseUrl: http://172.18.21.143:8000/v1
      apiKey: empty
      config:
        max_tokens: 10000
        max_completion_tokens: 10000
        reasoning: 
          effort: "high"
```

**Result**: Parameters are ignored, API requests use default `max_tokens: 1024`

### Configuration via Environment Variables (Working ✅)
```bash
OPENAI_MAX_TOKENS=10000
```

**Result**: Parameters are applied correctly

## Root Cause

The nested `config` structure in the YAML file is not being parsed or passed to the OpenAI provider correctly. The API requests show:

```json
{
  "requestBody": {
    "model": "openai/gpt-oss-120b",
    "messages": [...],
    "max_tokens": 1024  // ❌ Should be 10000
  }
}
```

This causes responses to be truncated with:
- `"finish_reason": "length"`
- `"content": null`
- Promptfoo output: `"output": []` (empty array)

## Repository Structure

```
└── sysPrompt/
    ├── .env
    ├── gpt-oss.json
    ├── promptfooconfig.yaml
    └── tests/
        └── test_cases.json
```

## File Contents

### 1. `.env`
```bash
# Workaround: Config parameters only work via environment variables
OPENAI_MAX_TOKENS=10000
```

### 2. `gpt-oss.json`
```json
[
  {
    "role": "system",
    "content": "{{system_message}}"
  },
  {
    "role": "user",
    "content": "{{question}}"
  }
]
```

### 3. `promptfooconfig.yaml`

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

description: "AI Assistant System Prompt - Test Configuration"

prompts:
  - id: ./gpt-oss.json
    label: gpt-oss

# Test LLM
providers:
  - id: openai:chat:openai/gpt-oss-120b
    config:
      apiBaseUrl: http://172.18.21.143:8000/v1
      apiKey: empty
      config:
        max_tokens: 10000
        max_completion_tokens: 10000
        reasoning: 
          effort: "high"        

tests:
  - ./tests/test_cases.json

# Judge LLM
defaultTest:
  options:
    provider:
      id: openai:chat:openai/gpt-oss-120b
      config:
        apiBaseUrl: http://172.18.21.143:8000/v1
        apiKey: empty
        config:
          max_tokens: 10000
          max_completion_tokens: 10000
          reasoning: 
            effort: "high"  
  vars:
      system_message: >-
        You are a helpful AI assistant designed to provide accurate and professional responses.
        
        ## Core Capabilities
        - Answer questions based on your training knowledge
        - Provide structured and well-formatted responses
        - Maintain a professional and neutral tone
        - Use proper citation formats when referencing information
        
        ## Response Guidelines
        - Always output valid Markdown formatting
        - Use headings and lists to improve clarity
        - Be concise but thorough in explanations
        - Acknowledge limitations when appropriate
        
        ## Knowledge Parameters
        - Knowledge Cutoff: October 2023
        - Memory: Stateless (no retention between sessions)
        - Language: Respond in the user's language
        - Encoding: Output characters in raw UTF-8
        
        ## Context Variables
        You have access to the following context:
        - {{CURRENT_DATE}}
        - {{CURRENT_DATETIME}}
        - {{CURRENT_TIME}}
        - {{CURRENT_TIMEZONE}}
        - {{CURRENT_WEEKDAY}}
        - {{USER_NAME}}
```

### 4. `tests/test_cases.json`

```json
[
    {
        "vars": {
            "CURRENT_DATE": "2025-10-12",
            "CURRENT_DATETIME": "2025-10-12T17:25:00+02:00",
            "CURRENT_TIME": "17:25:00",
            "CURRENT_TIMEZONE": "Europe/Rome",
            "CURRENT_WEEKDAY": "Sunday",
            "USER_NAME": "TestUser",
            "question": "What is photosynthesis and why is it important?"
        },
        "assert": [
            {
                "type": "llm-rubric",
                "value": "The response should explain that photosynthesis is the process by which plants convert light energy into chemical energy and mention its importance for oxygen production and the food chain"
            }
        ]
    },
    ....... other test cases
]
```

## Observed Behavior

### Without Environment Variable
```bash
$ promptfoo eval --no-cache --env-file .env
```

**API Request Body**:
```json
{
  "model": "openai/gpt-oss-120b",
  "messages": [...],
  "max_tokens": 1024  // ❌ Default value, ignoring YAML config
}
```

**Result**:
- 6 out of 10 tests fail
- Pass Rate: 40%
- Failed responses show: `"output": []`, `"content": null`, `"finish_reason": "length"`

### With Environment Variable
```bash
# .env file
OPENAI_MAX_TOKENS=10000
```

```bash
$ promptfoo eval --no-cache --env-file .env
```

**API Request Body**:
```json
{
  "model": "openai/gpt-oss-120b",
  "messages": [...],
  "max_tokens": 10000  // ✅ Correctly applied from env var
}
```

**Result**:
- All 10 tests pass
- Pass Rate: 100%
- Responses complete successfully

## Log Evidence

### Failed Request (YAML config ignored)
```
{
  "message": "API request",
  "url": "http://172.18.21.143:8000/v1/chat/completions",
  "method": "POST",
  "requestBody": "{\"model\":\"openai/gpt-oss-120b\",\"messages\":[...],\"max_tokens\":1024}",
  "status": 200,
  "response": {
    "choices": [{
      "message": {
        "content": null,
        "reasoning_content": "..."
      },
      "finish_reason": "length"
    }],
    "usage": {
      "completion_tokens": 1024
    }
  }
}
[logger.js:328] Evaluator response = {"output":[],...}
```

## Configuration Tested

###  (Current - Not Working)
```yaml
providers:
  - id: openai:chat:openai/gpt-oss-120b
    config:
      apiBaseUrl: http://172.18.21.143:8000/v1
      apiKey: empty
      config:
        max_tokens: 10000
        max_completion_tokens: 10000
        reasoning: 
          effort: "high"
```

## Expected Behavior

The `max_tokens`, `max_completion_tokens`, and `reasoning.effort` parameters defined in `promptfooconfig.yaml` should be:
1. Parsed from the YAML configuration
2. Passed to the OpenAI provider
3. Included in the API request body
4. Applied to all test evaluations

## Actual Behavior

- YAML configuration parameters are ignored
- Default values are used instead (`max_tokens: 1024`)
- Only environment variables (`OPENAI_MAX_TOKENS=10000`) successfully override defaults
- This causes test failures due to response truncation

## Workaround (Current)

Add to `.env` file:
```bash
OPENAI_MAX_TOKENS=10000
```

## Questions

1. Is the nested `config.config` structure correct for OpenAI providers?
2. Should model-specific parameters be at a different level in the YAML hierarchy?
3. Is there a bug in how Promptfoo parses provider configuration for custom OpenAI-compatible endpoints?
4. Are there any debug flags to verify which configuration values are actually being loaded?

## Test Results Summary

**Without env var workaround**:
```
Duration: 34s (concurrency: 4)
Successes: 4
Failures: 6
Errors: 0
Pass Rate: 40.00%
```

**With env var workaround**:
```
Duration: 28s (concurrency: 4)
Successes: 10
Failures: 0
Errors: 0
Pass Rate: 100.00%
```

## Additional Context

- The API endpoint (`http://172.18.21.143:8000/v1`) is OpenAI-compatible
- The model `openai/gpt-oss-120b` supports extended token limits
- The issue affects both the test provider and the judge/grading provider
- Debug logs show no errors during config parsing, suggesting silent failure

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1195/promptfoo-provider-config-parameters-not-being-loaded-only-environment">ENG-1195</a></p>

Hi @sir3mat,

Can you unnest the keys defined under `config.config` and let me know if that resolves this issue? Our [OpenAI Provider Documentation](https://www.promptfoo.dev/docs/providers/openai/#configuring-parameters) may be helpful here.

Hi @will-holley ,

Thank you so much for the quick response and for pointing me to the documentation! You were absolutely right - it was my mistake. The double nesting of config.config was causing the issue.

After removing the extra nesting level and placing the parameters directly under config, everything works perfectly now: