Repository: promptfoo/promptfoo
Issue: Unable to Prevent Circular Reference during Serialization.
Root cause: Validation Gap

Hi there,

I’m encountering an issue when using the GROQ API with the configuration in my config.yaml file. Specifically, I’m running into a circular reference error during serialization. The error message is shown in the screenshot below:

![Screenshot 2024-12-18 000234](https://github.com/user-attachments/assets/ffb6312a-3100-4a93-8669-b596ea1b5e25)

Here’s the configuration I’m using:
 
```
providers:
  - id: groq:llama3-8b-8192
    config:
      temperature: 0.7
      max_tokens: 100
      apiKey: ${GROQ_API_KEY}

prompts:
  - id: test-prompt
    raw: "This is a test prompt."

tests:
  - name: "Simple Test"
    prompts:
      - test-prompt
    provider: groq:llama3-8b-8192
```

The error occurs when trying to serialize the data, and I’m unable to prevent the circular reference.

Any assistance or help in resolving this issue would be greatly appreciated!

Thanks in advance.


Hi @ankit-agarwal-x, thanks for reporting this issue. It looks like your test setup not quite correct. Specifically, this part needs to be adjusted:

```yaml
tests:
  - name: "Simple Test"
    prompts:
      - test-prompt
    provider: groq:llama3-8b-8192
```

You can use something like this instead 

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - label: test-prompt
    id: "Write a tweet about: {{topic}}"
providers:
  - id: groq:llama3-8b-8192
    config:
      temperature: 0.7
      max_tokens: 100

tests:
  - vars:
      topic: "cats"
    assert:
      - type: "contains"
        value: "cats"
```

I recommend adding the following line to the top of your `config.yaml` file to validate your configuration (if your editor supports YAML schemas – VSCode does by default):

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
```

For details on setting up tests, take a look at our [Configuration Guide](https://www.promptfoo.dev/docs/configuration/guide/) and some [examples](https://github.com/promptfoo/promptfoo/tree/main/examples) to ensure everything is correct.

Finally, I highly recommend checking out this short video from OpenAI on [how to use Promptfoo](https://vimeo.com/1023317525).


Hi @mldangelo,

Thank you for your earlier guidance regarding my configuration. It helped clarify the setup, and I’ve made the suggested changes to align with the schema. However, I’m now encountering a circular reference error when using the GROQ provider with the configuration below:

```
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: groq:llama3-8b-8192
    config:
      temperature: 0.7
      max_tokens: 100
      apiKey: ${GROQ_API_KEY}
tests:
  - name: "RAG Security and Integrity Test Suite"
    prompts:
      - simple-injection-test
      - document-bypass
    provider: groq:llama3-8b-8192
    assertions:
      - type: model-graded-security
```

The error message includes: 

```
Error saving result: TypeError: Converting circular structure to JSON  
Starting at object with constructor 'Groq'  
    property 'completions' -> object with constructor 'Completions'  
    property '_client' closes the circle  
```
![Screenshot 2024-12-18 185723](https://github.com/user-attachments/assets/a90d1be5-a4c3-4548-b170-fbd970e502b1)

Additionally, the metrics table view isn’t rendering because results cannot be saved, likely due to this serialization issue.

If you have any insights or recommendations to resolve this, I’d greatly appreciate it. Thanks again for the previous help and for the excellent work on Promptfoo!

Best regards,
Ankit

Hi @ankit-agarwal-x,  thanks for the update. A few corrections:  

1. Test cases don’t support a `prompt` attribute. You should define your prompts as shown in my example above or in the [[Configuration Guide](https://promptfoo.dev/docs/configuration)](https://promptfoo.dev/docs/configuration).  
2. There’s no need to set a `provider` on your test case. This should be handled with the top-level `providers` key.  
3. Test cases don’t support a `name` attribute either. If you only have one test case, you don’t need to worry about this.  
4. `model-graded-security` is not a supported assertion type. You can check our supported assertions here: [[Supported Assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/)](https://www.promptfoo.dev/docs/configuration/expected-outputs/).  

I highly recommend starting with our examples. We even have a GROQ example available [[here](https://github.com/promptfoo/promptfoo/tree/main/examples/groq)](https://github.com/promptfoo/promptfoo/tree/main/examples/groq). You can get a copy with:  
```bash
npx promptfoo@latest init --example groq
```

Let me know if you have any other questions!  

Hi @mldangelo,

Thank you for your detailed guidance and the prompt resolution of the initial issue. After carefully reviewing the updated configuration examples and aligning them with the supported schema, I identified the root cause of the problem. The circular reference error stemmed from improperly retrieving metadata from the Groq provider and incorrectly serializing the data before sending it to Promptfoo.

By restructuring the metadata handling logic and ensuring proper serialization, the issue has been resolved. Specifically, I isolated `the _client` reference within the Groq completion object, preventing circular dependencies during JSON serialization. This adjustment also enabled smooth integration with the Promptfoo evaluator pipeline, and the metrics view now renders without any issues.

![Screenshot 2024-12-22 012810](https://github.com/user-attachments/assets/c5a1f2a6-906a-4285-805c-657d1d8a6d75)

I appreciate your help in pointing out the nuances of the YAML configuration and schema alignment—it significantly improved my understanding of the setup process. Thank you once again for your support and the excellent documentation!

Best regards,
Ankit