Repository: promptfoo/promptfoo
Issue: Request: better traceback on Python `SyntaxError`
Root cause: Validation Gap

I am trying something like this:

```yaml
---
prompts: >-
  Extract the single letter answer from the following question and answer
  \n\nDoes protein encoded by TDRD1 have symmetrical or asymmetrical
  dimethylarginine modifications, or no dimethylarginine at all?
  \n\nOptions:
  A) Insufficient information to answer this question
  B) No dimethylarginine
  C) Symmetrical dimethylarginine modifications
  D) Asymmetrical dimethylarginine modifications
  \n\nAnswer:
providers:
  - openai:chat:gpt-4-0613
tests:
  - description: "Check answer with regex"
    assert:
      - type: python
        value: |
          import re

          def extract_answer(answer: str) -> str:
              # first capital letter, like A)
              result = re.search(r"([A-Z])\)", answer, re.DOTALL)
              if result is not None:
                  return result.group(1)
              raise NotImplementedError

          result = extract_answer(output).strip()
          pass, reason = False, "Incorrect"
          if unsure_mc_answer and result[0].lower() == unsure_mc_answer[0].lower():
              reason = "Unsure"
          if result[0].lower() == ideal_mc_answer[0].lower():
              pass, reason = True, "Unsure"
          return {"pass": pass, "reason": reason}
```

With `promptfoo==0.55.0`, running `npm run promptfoo eval`, I just get:

```
Error running Python script: Error: SyntaxError: invalid syntax
Error executing Python code: Error: SyntaxError: invalid syntax
```

And the `promptfoo` table says `[FAIL] Python code execution failed: SyntaxError: invalid syntax`.

Do you think we can have `promptfoo` print more context to help debug these `SyntaxError`?

---

Edit: for the record I know there's some latent `NameError`s here, so maybe this is not the best minimal repro, but the idea holds 😆 

I think in this case, the issue may be that you are using "pass" as a
variable name, but it is a reserved keyword in python.

On Sat, Apr 27, 2024 at 10:20 PM James Braza ***@***.***>
wrote:

> I am trying something like this:
>
> ---prompts: >-  Extract the single letter answer from the following question and answer  \n\nDoes protein encoded by TDRD1 have symmetrical or asymmetrical  dimethylarginine modifications, or no dimethylarginine at all?  \n\nOptions:  A) Insufficient information to answer this question  B) No dimethylarginine  C) Symmetrical dimethylarginine modifications  D) Asymmetrical dimethylarginine modifications  \n\nAnswer:providers:
>   - openai:chat:gpt-4-0613tests:
>   - description: "Check answer with regex"
>     assert:
>       - type: python
>         value: |          import re          def extract_answer(answer: str) -> str:              # first capital letter, like A)              result = re.search(r"([A-Z])\)", answer, re.DOTALL)              if result is not None:                  return result.group(1)              raise NotImplementedError          result = extract_answer(output).strip()          pass, reason = False, "Incorrect"          if unsure_mc_answer and result[0].lower() == unsure_mc_answer[0].lower():              reason = "Unsure"          if result[0].lower() == ideal_mc_answer[0].lower():              pass, reason = True, "Unsure"          return {"pass": pass, "reason": reason}
>
> With promptfoo==0.55.0, running npm run promptfoo eval, I just get:
>
> Error running Python script: Error: SyntaxError: invalid syntax
> Error executing Python code: Error: SyntaxError: invalid syntax
>
> And the promptfoo table says [FAIL] Python code execution failed:
> SyntaxError: invalid syntax.
>
> Do you think we can have promptfoo print more context to help debug these
> SyntaxError?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/722>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJTM7SS353DB2EB2TRTY7SBLJAVCNFSM6AAAAABG4V34MCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DOMZVGA2DAMA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


#724 adds python tracebacks to help you debug! Thank you for the suggestion

This change is included in release [0.56.0](https://github.com/promptfoo/promptfoo/releases/tag/0.56.0)

Oh wow, thank you for doing this! Appreciated very much for the quick turnaround too.

With the new traceback printing, debugging/iterating is much easier:

```none
Error running Python script: SyntaxError: invalid syntax
Stack Trace: Error: SyntaxError: invalid syntax
    at PythonShell.parseError (/Users/user/code/repo/node_modules/python-shell/index.js:303:21)
    ...
    File "/var/folders/a/b/T/temp-python-code-1714363289598-6384143182b03.py", line 14
      pass, reason = False, "Incorrect"
          ^
```