Repository: explodinggradients/ragas
Issue: Expensive agentic failure loop
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
In some cases / with some models, the instructions to generate json result in something like

```txt
Here's the generated abstract conceptual question in the requested JSON format::

{"text": ...}

Would you like me to explain in more detail?
```

The built-in validator will fail this because it cannot parse the json, and then make several repeated requests to try to conform the response to the specified schema.  If this is an error associated with an LLM's default response preferences, the agentic retry loop is basically a waste of tokens.

Ragas version: 0.2.1
Python version:

**Code to Reproduce**
Use Anthropic Claude 3.5 Sonnet with the new testset generator

**Error trace**
[error.txt](https://github.com/user-attachments/files/17445255/error.txt)


**Expected behavior**
Intelligent extraction of json object from text blob

**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I'd like to take this on; I have an idea for a simple heuristic-based json extractor that should have a harmless fail mode.

hey @ahgraber that would be really useful - do give it a shot. 

I'm guessing a simple json re is what you have in mind? would be helpful! One more thing I wanted to check was if the JSON decoding errors are being passed into the prompts

the other main improvement is throwing an error when generation stops due to max-tokens - this one I'm trying to fix 

Hopefully we'll be able to make the json parser much better 🙂 ❤️ 

I don't even think I'll need regex -- basically I'm planning to search for paired `[]` or `{}`.  

I've noticed that newer models tend to return complete JSON (provided there's no truncation for token count), but the models are frequently verbose and will try to be too helpful e.g., "Here's the json requested: {...} Please ask for further explanation".   Langchain's default json/markdown parser fails in this case when the json is embedded in text and not delimited with a '```json'.

I'll keep an eye on my LMStudio logs and see if I can find the rendered repair prompt.  

<!-- Answer -->
The issue you're encountering seems to be related to the JSON validation and extraction process in Ragas version 0.2.1. The current implementation uses the `RagasOutputParser` class, which attempts to parse the output string to ensure it adheres to the specified JSON schema. If the output string doesn't meet the JSON format requirements, it uses a retry mechanism to correct it, which can lead to repeated requests and token wastage if the initial response is not in the expected format <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/prompt/pydantic_prompt.py)</sup>.

Your idea of implementing a heuristic-based JSON extractor could potentially help in intelligently extracting JSON objects from text blobs, reducing unnecessary retries. This approach might involve identifying and isolating JSON-like structures within the text, which could then be parsed separately.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@jjmachan  See attached log for repair loop from a HeadlinesExtractor transformation.

[parse-errors.log](https://github.com/user-attachments/files/17446850/parse-errors.log)

1. It does not look like json decoding error information is being included in the repair prompt
2. The original text is included with increasing levels of escape characters, ultimately leading (in my case) to a crash due to exceeding the context length.