Repository: promptfoo/promptfoo
Issue: How to avoid `Template render error` without affecting prompt (using Python to generate tests and prompt)?
Root cause: Architectural Constraint

**Describe the bug**
I have Python code for generating tests, and prompt, and also for provider and metric.
The code that generate tests, will produce JSON with text that may contain special character sequences including `{{`
This issue is related to https://github.com/promptfoo/promptfoo/issues/1283

In the following example I provide the minimal Python code to replicate the problem.

**To Reproduce**
Steps to reproduce the behavior: copy the following files to a directory, and run `promptfoo eval --config promptfooconfig.yaml`

Promptfoo configuration:
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: 'promotfoo bug'

prompts:
  - file://my_prompt.py:get_prompt

providers:
  - id: file://my_provider.py:call_api
    label: my_provider

tests: file://my_tests.py:generate_tests
```

Python code:
```python
# ==> my_metric.py <==
from typing import Dict, Any, Union


def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
    return True
```
```python
# ==> my_prompt.py <==
import logging
from typing import Dict, Any

import promptfoo_utils

logging.basicConfig(
    filename="my_prompt.log",
    encoding="utf-8",
    filemode="a",
    format="{asctime} - {levelname} - {message}",
    style="{",
    datefmt="%Y-%m-%d %H:%M",
)


def get_prompt(context: Dict[str, Any]) -> str:
    variables: Dict[str, Any] = context['vars']
    text: str = variables['text']
    prompt = f"""Extract data in JSON format from the following TEXT.

TEXT
---
{text}
"""
    logging.warning(f'prompt =\n'
                    f'{prompt}')
    return prompt
```

```python
# ==> my_provider.py <==
from typing import Dict, Any

generated_text = """
{
  "p1": 1,
  "p2": "Vacancy",
  "p3": "0.11 Ω"
}
"""


def call_api(prompt, options, context: Dict[str, Any]) -> Dict[str, Any]:
    return {
        'output': generated_text
    }
```

```python
# ==> my_tests.py <==
from typing import List, Dict, Any

import promptfoo_utils

TEXT = """For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be
around 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both
methods (XRD and RCVC) provide identical values."""


def _generate_tests(escape: bool) -> List[Dict[str, Any]]:
    text = promptfoo_utils.escape(TEXT) if escape else TEXT
    return [
        {
            'vars': {
                'unique_id': '32',
                'text': text,
            },
            'assert': [
                {
                    'type': 'python',
                    'value': 'file://my_metric.py:get_assert'
                }
            ]
        }
    ]

def generate_tests() -> List[Dict[str, Any]]:
    return _generate_tests(escape=False)

def generate_tests_escape() -> List[Dict[str, Any]]:
    return _generate_tests(escape=True)
```

```python
# ==> promptfoo_utils.py <==
START_ESCAPE = "{{'"
END_ESCAPE = "'}}"

def escape(text: str) -> str:
    return (START_ESCAPE +
            text +
            END_ESCAPE)


def is_escaped(text: str) -> bool:
    return text.startswith(START_ESCAPE) and text.endswith(END_ESCAPE)


def unescape(text: str) -> str:
    if is_escaped(text):
        return text[len(START_ESCAPE):-len(END_ESCAPE)]
    else:
        return text
```

With the above configuration, I get error: `[ERROR] Template render error: (unknown path) [Line 1,Column 51] expected variable end ...`

**Expected behavior**
I expect the code to run without errors.
I have tried to escape the text, as suggested in https://github.com/promptfoo/promptfoo/issues/1283, and that works without errors. However, that affects the prompt. You can see the behaviour with this configuration
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: 'promotfoo bug'

prompts:
  - file://my_prompt.py:get_prompt

providers:
  - id: file://my_provider.py:call_api
    label: my_provider

tests: file://my_tests.py:generate_tests_escape
```

Now when looking at `my_prompt.log`, produced by `my_prompt_py`, I see the following:
```
2025-06-18 11:29 - WARNING - prompt =
Extract data in JSON format from the following TEXT.

TEXT
---
{{'For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be
around 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both
methods (XRD and RCVC) provide identical values.'}}
```
The prompt contains the escape characters `{{'` and `'}}`, which should not be part of the prompt.

How do avoid this?

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

```
➤ promptfoo debug                                                                                                                                                                  (base)
================================================================================================================================================================================
Promptfoo Debug Information
================================================================================================================================================================================
{
  "version": "0.115.1",
  "platform": {
    "os": "darwin",
    "release": "24.5.0",
    "arch": "arm64",
    "nodeVersion": "v24.2.0"
  },
  "env": {},
  "configInfo": {
    "defaultConfigPath": "/Users/marco/Downloads/promptfoo/promptfoo_bug/promptfooconfig.yaml",
    "configExists": true,
    "configContent": {
      "config": {
        "tags": {},
        "description": "promotfoo bug",
        "prompts": [
          "file:///Users/marco/Downloads/promptfoo/promptfoo_bug/my_prompt.py:get_prompt"
        ],
        "providers": [
          {
            "id": "file://my_provider.py:call_api",
            "label": "my_provider"
          }
        ],
        "tests": [
          {
            "vars": {
              "unique_id": "32",
              "text": "{{'For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be\naround 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both\nmethods (XRD and RCVC) provide identical values.'}}"
            },
            "assert": [
              {
                "type": "python",
                "value": "file://my_metric.py:get_assert"
              }
            ]
          }
        ],
        "scenarios": [],
        "env": {},
        "sharing": true,
        "defaultTest": {
          "vars": {},
          "assert": [],
          "options": {},
          "metadata": {}
        },
        "outputPath": [],
        "extensions": [],
        "metadata": {}
      },
      "testSuite": {
        "description": "promotfoo bug",
        "tags": {},
        "prompts": [
          {
            "raw": "import logging\nfrom typing import Dict, Any\n\nimport promptfoo_utils\n\nlogging.basicConfig(\n    filename=\"my_prompt.log\",\n    encoding=\"utf-8\",\n    filemode=\"a\",\n    format=\"{asctime} - {levelname} - {message}\",\n    style=\"{\",\n    datefmt=\"%Y-%m-%d %H:%M\",\n)\n\n\ndef get_prompt(context: Dict[str, Any]) -> str:\n    variables: Dict[str, Any] = context['vars']\n    text: str = variables['text']\n    # text = promptfoo_utils.unescape(text)\n    prompt = f\"\"\"Extract data in JSON format from the following TEXT.\n\nTEXT\n---\n{text}\n\"\"\"\n    logging.warning(f'prompt =\\n'\n                    f'{prompt}')\n    return prompt\n",
            "label": "/Users/marco/Downloads/promptfoo/promptfoo_bug/my_prompt.py:get_prompt"
          }
        ],
        "providers": [
          {
            "options": {
              "id": "file://my_provider.py:call_api",
              "config": {
                "basePath": "/Users/marco/Downloads/promptfoo/promptfoo_bug"
              },
              "env": {}
            },
            "isInitialized": false,
            "initializationPromise": null,
            "scriptPath": "my_provider.py",
            "functionName": "call_api",
            "label": "my_provider",
            "config": {
              "basePath": "/Users/marco/Downloads/promptfoo/promptfoo_bug"
            }
          }
        ],
        "providerPromptMap": {
          "file://my_provider.py:call_api": [
            "/Users/marco/Downloads/promptfoo/promptfoo_bug/my_prompt.py:get_prompt"
          ],
          "my_provider": [
            "/Users/marco/Downloads/promptfoo/promptfoo_bug/my_prompt.py:get_prompt"
          ]
        },
        "tests": [
          {
            "vars": {
              "unique_id": "32",
              "text": "{{'For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be\naround 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both\nmethods (XRD and RCVC) provide identical values.'}}"
            },
            "assert": [
              {
                "type": "python",
                "value": "file://my_metric.py:get_assert"
              }
            ]
          }
        ],
        "scenarios": [],
        "defaultTest": {
          "metadata": {},
          "options": {},
          "vars": {},
          "assert": []
        },
        "nunjucksFilters": {},
        "extensions": []
      },
      "basePath": "/Users/marco/Downloads/promptfoo/promptfoo_bug"
    }
  }
}
================================================================================================================================================================================
```

Hi @marcosbodio, thanks for filing this issue and apologies that it's taken some time for a response. I will look into this today and will respond here once I have an update for you!

Are you able to wrap each prompt in Nunjucks [raw tags](https://mozilla.github.io/nunjucks/templating.html#raw)? For example

```
{% raw %}
For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be
around 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both
methods (XRD and RCVC) provide identical values.
{% endraw %}
```

> Are you able to wrap each prompt in Nunjucks [raw tags](https://mozilla.github.io/nunjucks/templating.html#raw)? For example
> 
> ```
> {% raw %}
> For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be
> around 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both
> methods (XRD and RCVC) provide identical values.
> {% endraw %}
> ```

Hi @will-holley ,

I can do what you suggest above, but that is not good for me, because then my prompt becomes:

```
Extract data in JSON format from the following TEXT.

TEXT
---
{% raw %}For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be
around 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both
methods (XRD and RCVC) provide identical values.'{% endraw %}
```

And I want the prompt to be:

```
Extract data in JSON format from the following TEXT.

TEXT
---
For instance, for single vacancies (Fig. 9a), ^{{Ω}rel }is found to be
around 0.11 Ω 00 up to the maximum tested concentration of 1.4 %; both
methods (XRD and RCVC) provide identical values.'
```

In that case, do you require the use of Nunjucks templating?

It can be disabled by setting `PROMPTFOO_DISABLE_TEMPLATING=true` in your environment.

In the simple example that I posted above to re-create the issue I do not need Nunjucks templating, but in a more general case I might need.

Is there a way to avoid the problem without setting the environment variable `PROMPTFOO_DISABLE_TEMPLATING=true`?

Looking at the full stack track, the error originates from Nunjucks and appears to be caused by the inclusion of the unescaped double-curly opening brace.  If you need Nunjucks templating and cannot modify your prompt to include the raw tags, I recommend sanitizing your prompts prior to evaluation by escaping the offending curly-braces.

Hi @marcosbodio,

After reviewing this issue, the behavior you're experiencing is a fundamental limitation of how template engines like Nunjucks work - they process all text looking for `{{` before your prompt function runs.

**Available workarounds:**

1. **Disable templating globally** (if you don't need it):
   ```bash
   PROMPTFOO_DISABLE_TEMPLATING=true promptfoo eval
   ```

2. **Pre-process your data** to escape `{{` before passing to promptfoo:
   ```python
   text = text.replace('{{', '{ {')  # Add space
   ```
   Then in your prompt function, restore it:
   ```python
   text = text.replace('{ {', '{{')
   ```

3. **Use `{% raw %}` tags** (if you can post-process the prompt before sending to LLM)

Since there's no way to selectively disable templating for specific vars without fundamentally changing how promptfoo processes configs, and several workarounds are available, I'm closing this issue.

If you have a specific use case that none of these solve, feel free to comment or re-open with more details.

Thanks!