Repository: explodinggradients/ragas
Issue: adapt pydantic.v1.error_wrappers.ValidationError: 1 validation error for Prompt
Root cause: Validation Gap

I want to customize the adapt function,  change the language to Chinese

this is code：
```py
cache_dir = ".cache"

adapt(
    metrics=evaluate_stk,
    language="Chinese",
    cache_dir=cache_dir,
    llm=llm
)
```

and this is the error
```
Traceback (most recent call last):
  File "/Users/Desktop/side2side-eval/langchain_idealab/ragas_evaluate.py", line 87, in <module>
    adapt(
  File "/Users/opt/anaconda3/envs/self-rag/lib/python3.9/site-packages/ragas/adaptation.py", line 36, in adapt
    metric.adapt(language, cache_dir=cache_dir)
  File "/Users/opt/anaconda3/envs/self-rag/lib/python3.9/site-packages/ragas/metrics/_answer_relevance.py", line 173, in adapt
    self.question_generation = self.question_generation.adapt(
  File "/Users/opt/anaconda3/envs/self-rag/lib/python3.9/site-packages/ragas/llms/prompt.py", line 181, in adapt
    return self._load(language, self.name, cache_dir)
  File "/Users/opt/anaconda3/envs/self-rag/lib/python3.9/site-packages/ragas/llms/prompt.py", line 281, in _load
    return cls(**json.load(open(path)))
  File "/Users/opt/anaconda3/envs/self-rag/lib/python3.9/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Prompt
__root__
  output in example 1 is not in valid json format: Expecting value: line 1 column 1 (char 0) (type=value_error)
```

<!-- Greeting -->
Hey @runwean! 👋 I'm here to help you out with any bugs, questions, or contributions while you wait for a human maintainer. Let's tackle this issue together!

<!-- Answer -->
To resolve the `pydantic.v1.error_wrappers.ValidationError` related to the JSON format in the `Prompt` class when changing the language to Chinese in the `adapt` function, you need to ensure that the JSON structure is correctly handled and validated. The error likely arises from the JSON structure not matching the expected format after translation.

Here is a step-by-step approach to address this issue:

1. **Ensure JSON Structure Consistency**: Make sure that the translated JSON maintains the same structure as the original JSON. This includes having the same keys and nested structures.

2. **Validate JSON Structure**: After translation, validate the JSON structure to ensure it matches the expected format.

3. **Handle JSON Loading and Parsing**: Use safe methods to load and parse JSON to avoid issues with incorrect formats.

Here is the relevant part of the `adapt` function with added validation and error handling:

```python
def adapt(
    self, language: str, llm: BaseRagasLLM, cache_dir: t.Optional[str] = None
) -> Prompt:
    def get_all_keys(nested_json):
        keys = set()
        for key, value in nested_json.items():
            keys.add(key)
            if isinstance(value, dict):
                keys = keys.union(get_all_keys(value))
        return keys

    if self.language == language:
        return self

    # TODO: Add callbacks
    cache_dir = cache_dir if cache_dir else get_cache_dir()
    if os.path.exists(os.path.join(cache_dir, language, f"{self.name}.json")):
        return self._load(language, self.name, cache_dir)

    logger.info("Adapting %s to %s", self.name, language)
    prompts = []
    output_keys = []
    for example in self.examples:
        prompts.extend(
            [
                str_translation.format(
                    translate_to=language, input=example.get(key)
                )
                for key in self.input_keys
            ]
        )
        prompts.append(
            json_translatation.format(
                translate_to=language, input=example.get(self.output_key)
            )
            if self.output_type.lower() == "json"
            else str_translation.format(
                translate_to=language, input=example.get(self.output_key)
            )
        )
        if self.output_type.lower() == "json":
            output = example.get(self.output_key)
            if isinstance(output, str):
                output = json.loads(output)
            if isinstance(output, dict):
                output_keys.append(get_all_keys(output))
            elif isinstance(output, list) and all(
                isinstance(item, dict) for item in output
            ):
                output_keys.append([get_all_keys(item) for item in output])

    # NOTE: this is a slow loop, consider Executor to fasten this
    results = []
    for p in prompts:
        results.append(llm.generate_text(p).generations[0][0].text)
    per_example_items = len(self.input_keys) + 1
    grouped_results = [
        results[i : i + per_example_items]
        for i in range(0, len(results), per_example_items)
    ]
    assert len(grouped_results) == len(
        self.examples
    ), "examples and adapted examples must be of equal length"
    for i, example in enumerate(grouped_results):
        example_dict = {}
        example_dict.update(
            {k: v for k, v in zip(self.input_keys, example[: len(self.input_keys)])}
        )
        example_dict[self.output_key] = (
            json_loader._safe_load(example[-1], llm)
            if self.output_type.lower() == "json"
            else example[-1]
        )

        if self.output_type.lower() == "json":
            output = example_dict[self.output_key]
            if isinstance(output, dict):
                assert (
                    set(output.keys()) == output_keys[i]
                ), f"Adapted output keys {set(output.keys())=} do not match with the original output keys: {output_keys[i]=}"
            elif isinstance(output, list) and all(
                isinstance(item, dict) for item in output
            ):
                assert all(
                    set(item.keys()) in output_keys[i] for item in output
                ), "Adapted output keys do not match with the original output keys"

        self.examples[i] = example_dict

    self.language = language

    # TODO:Validate the prompt after adaptation

    return self
```

This code ensures that the JSON structure is validated after translation, which should help in resolving the `pydantic.v1.error_wrappers.ValidationError` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/prompt.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


same issue

@Mervyn-1 @runwean sorry about this but do keep an eye on #1101 and #890 . will update that issue when we have a fix

I've encountered the same issue. It would be greatly appreciated if the author could look into it. Thank you so much!

It might not be an issue with the LLM API. I tried GPT-4o, GPT-4o-mini, and DeepSeek, and their translated cache JSON is almost the same, and I can't find any errors. Also, the first time you execute the adaptation and test generation, it works fine, so I guess something is wrong when it's trying to read the promt JSON saved as cache.

> I guess something is wrong when it's trying to read the prompt JSON saved as cache.

~~Assuming this is correct, how would I clear / remove the JSON saved as cache?~~

Answer was provided [here](https://github.com/explodinggradients/ragas/issues/1101#issuecomment-2276105041)


> 我遇到了同样的问题。如果作者能研究一下，我们将不胜感激。非常感谢！

请问你解决了吗

This has been fixed with v0.2 - I know finally 😅 🎉

do checkout the docs here: https://docs.ragas.io/en/stable/howtos/customizations/metrics/_metrics_language_adaptation/
reference here: https://docs.ragas.io/en/stable/references/prompt/#ragas.prompt.PromptMixin

and if you're migrating from v0.1 check out the migration docs here: https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02