Repository: promptfoo/promptfoo
Issue: Response from Claude v2 (Bedrock) causes errors
Root cause: Interface Contract Mismatch

I spun up the demo with some slight modifications to use amazon bedrock. I ran the tests and several of the links result in the following error:

<img width="751" alt="image" src="https://github.com/promptfoo/promptfoo/assets/5089480/dfe54a8c-41d6-4f53-9449-72ff1ac55c6d">

<img width="474" alt="image" src="https://github.com/promptfoo/promptfoo/assets/5089480/1ddb299f-b9b9-459a-b90c-2eb2a3fbc82a">


Full config:
```yaml
# This configuration compares LLM output of 2 prompts x 2 GPT models across 3 test cases.
# Learn more: https://promptfoo.dev/docs/configuration/guide
description: 'My first eval'

prompts:
  - 'Write a tweet about {{topic}}'
  - 'Write a very concise, funny tweet about {{topic}}'

providers:
  - bedrock:completion:anthropic.claude-v2

tests:
  - vars:
      topic: bananas

  - vars:
      topic: avocado toast
    assert:
      # For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
      - type: icontains
        value: avocado
      - type: javascript
        value: 1 / (output.length + 1) # prefer shorter outputs

  - vars:
      topic: new york city
    assert:
      # For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
      - type: llm-rubric
        value: ensure that the output is funny
```

Are you able to share the contents of `~/.promptfoo/output/latest.json`?  That will help me troubleshoot, as I don't have access to the claude-v2 model on bedrock.

Sure! 
[latest.json](https://github.com/promptfoo/promptfoo/files/14284203/latest.json)


Thanks, very helpful!  It looks like Claude v2 may be returning a different format from expected.  The client-side error occurs because the `score` value is `null`.  

I'll handle the client side error so it doesn't crash the page, but this will also require an update to the Bedrock provider code.

One potentially temporary workaround is to add a [`transform` directive](https://promptfoo.dev/docs/configuration/guide/#transforming-outputs) to your test cases to unpack the nested `completion` value in the Claude response.  That would look something like this:

```yaml
defaultTest:
  options:
    transform: JSON.parse(output).completion
```

Thanks a lot @typpo! 

Unfortunately, the transform didn't work.

```
    │ [FAIL] SyntaxError: Unexpected token o in JSON                     │ [FAIL] SyntaxError: Unexpected token o in JSON                     │
│                                                                    │ at position 1                                                      │ at position 1                                                      │
│                                                                    │                                                                    │                                                                    │
│                                                                    │ SyntaxError: Unexpected token o in JSON at position 1              │ SyntaxError: Unexpected token o in JSON at position 1              │
│                                                                    │ m                                                                  │ m                                                                  │
│                                                                    │     at JSON.parse (<anonymous>)                                    │     at JSON.parse (<anonymous>)                                    │
│                                                                    │     at eval (eval at transformOutput (/Users/sean.hellebu          │     at eval (eval at transformOutput (/Users/sean.hellebu          │
│                                                                    │ sch/.npm/_npx/81bbc6515d992ace/node_modules/promp...               │ sch/.npm/_npx/81bbc6515d992ace/node_modules/promp...
```

Ok. I should be able to get you a confirmed solution (and a proper fix) as
soon as my model access request is approved :)

On Thu, Feb 15, 2024, 8:46 AM Sean Hellebusch ***@***.***>
wrote:

> Unfortunately, the transform didn't work.
>
>     │ [FAIL] SyntaxError: Unexpected token o in JSON                     │ [FAIL] SyntaxError: Unexpected token o in JSON                     │
> │                                                                    │ at position 1                                                      │ at position 1                                                      │
> │                                                                    │                                                                    │                                                                    │
> │                                                                    │ SyntaxError: Unexpected token o in JSON at position 1              │ SyntaxError: Unexpected token o in JSON at position 1              │
> │                                                                    │ m                                                                  │ m                                                                  │
> │                                                                    │     at JSON.parse (<anonymous>)                                    │     at JSON.parse (<anonymous>)                                    │
> │                                                                    │     at eval (eval at transformOutput (/Users/sean.hellebu          │     at eval (eval at transformOutput (/Users/sean.hellebu          │
> │                                                                    │ sch/.npm/_npx/81bbc6515d992ace/node_modules/promp...               │ sch/.npm/_npx/81bbc6515d992ace/node_modules/promp...
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/474#issuecomment-1946560141>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJXIHE3OB6VJGJYUMB3YTY3X3AVCNFSM6AAAAABDIQSS2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBWGU3DAMJUGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Sorry, it's been a while and I still don't have access to Claude 2.1 on Bedrock.  However, I think I noticed the issue in code (a030796476d52236b02e487f7c7f264e70b2e902).  The fix should be out in 0.43.1, and as a temporarily workaround I believe you can disable the cache (`promptfoo eval --no-cache`)