Repository: promptfoo/promptfoo
Issue: How to escape the `{{ placeholder }}`?

Hello team!
Thank you very much for the awesome software!

I have a question regarding placeholders.

In my use case I need to keep the string pattern `{{ something }}` as is, but unfortunately this collides with the Promptfoo placeholders. Is there a way to escape it?


Augmenting this with [a minimal example](https://stackoverflow.com/q/79599765/1866775):

I'd like to give this prompt to the LLM: `Output this exact string:{{some_string}}"`, i.e., `{{some_string}}` shall not be replaced by Promptfoo with some variable, but be verbatim instead.

My current attempt looks as follows:


```yaml
description: "test_2025-04-30_003"

prompts:
  - "Output this exact string:{{ '{{some_string}}' }}"

providers:
  - "openai:gpt-4o-mini"

tests:
    assert:
      - type: contains-all
        value: "{{ '{{some_string}}' }}"
```

But the output of

```bash
promptfoo eval --verbose
```

shows this:

```
[chat.js:107] Calling OpenAI API: {"model":"gpt-4o-mini","messages":[{"role":"user","content":"Output this exact string:"}],"max_tokens":1024}
```

So the `{{some_string}}` is missing in the message.

Promptfoo templating is nunjucks/jinja compatible, so you should be able to
do the following:

```
{{ '{{some_string}}' }}
```

On Wed, Apr 30, 2025, 5:30 AM Tobias Hermann ***@***.***>
wrote:

> *Dobiasd* left a comment (promptfoo/promptfoo#3855)
> <https://github.com/promptfoo/promptfoo/issues/3855#issuecomment-2841819001>
>
> Augmenting this with a minimal example:
>
> I'd like to give this prompt to the LLM: Output this exact
> string:{{some_string}}", i.e., {{some_string}} shall not be replaced by
> Promptfoo with some variable, but be verbatim instead.
>
> My current attempt looks as follows:
>
> description: "test_2025-04-30_003"
> prompts:
>   - "Output this exact string:{{ '{{some_string}}' }}"
> providers:
>   - "openai:gpt-4o-mini"
> tests:
>     assert:
>       - type: contains-all
>         value: "{{ '{{some_string}}' }}"
>
> But the output of
>
> promptfoo eval --verbose
>
> shows this:
>
> [chat.js:107] Calling OpenAI API: {"model":"gpt-4o-mini","messages":[{"role":"user","content":"Output this exact string:"}],"max_tokens":1024}
>
> So the {{some_string}} is missing in the message.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/3855#issuecomment-2841819001>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJQQRMFG26PWIFRZVD324C65DAVCNFSM6AAAAAB4FI7WFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBRHAYTSMBQGE>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Thanks for the response! But that's exactly what I already tried. Or am I missing something?

Hmm sorry about that and thank you for the repro.  This is something we'll have to look into.  

BTW, you can set the `PROMPTFOO_DISABLE_TEMPLATING` environment variable to completely disable templating - not sure if it helps you here.

Thanks! Setting `PROMPTFOO_DISABLE_TEMPLATING=true` works, but it's not a good solution for me, because I need the templating functionality in my actual test configuration. So I don't want to disable it globally. 

Do the ergonomics of

```yaml
prompts:
  - "Output this exact string:{% raw %}{{some_string}}{% endraw %}"

providers:
  - "openai:gpt-4o-mini"

tests:
  - assert:
      - type: contains
        value: "{% raw %}{{some_string}}{% endraw %}"
```

work for you, @bhky / @Dobiasd?  Using [`raw`](https://mozilla.github.io/nunjucks/templating.html#raw) solves your escaping problem.

Thanks for the suggestion, I just [tried it, but it did not work either](https://gist.github.com/Dobiasd/c294c2a258e2f7a7444015c507186d7e).

```
[chat.js:107] Calling OpenAI API: {"model":"gpt-4o-mini","messages":[{"role":"user","content":"Output this exact string:"}],"max_tokens":1024}
```

I'm using Promptfoo version `0.112.1`.

Check out 0.112.2, @Dobiasd, and let me know if you run into any more issues!

Awesome! Thanks a lot for the quick fix. ❤