Repository: explodinggradients/ragas
Issue: Litmus Integration (https://github.com/google/litmus)
Root cause: Documentation Deficiency

Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and assessing the performance of your LLM-powered applications on Google Cloud.

## Features

- **Automated Test Execution:** Submit test runs using pre-defined templates to evaluate responses against golden answers using AI.
- **Flexible Test Templates:** Define and manage test templates specifying the structure and parameters of your tests. You can choose between two template types: "Test Run" for single-turn interactions and "Test Mission" for multi-turn interactions where the LLM generates its requests.
- **User-Friendly Web Interface:** Interact with the Litmus platform through a visually appealing and intuitive web interface.
- **Detailed Results:** View the status, progress, and detailed results of your test runs.
- **Advanced Filtering:** Filter responses from test runs based on specific JSON paths for in-depth analysis.
- **Performance Monitoring:** Track the performance of your responses and identify areas for improvement by using AI.
- **Multiple LLM Evaluation Methods:** Leverage a variety of LLM evaluation methods:
  - **Custom LLM Evaluation with Customizable Prompts:** Use an LLM to compare actual responses with expected (golden) responses, utilizing flexible prompts tailored to your evaluation needs.
  - **Ragas Evaluation:** Apply Ragas metrics, including answer relevancy, context recall, context precision, harmfulness, and answer similarity.
  - **DeepEval Evaluation:** Leverage DeepEval's LLM-based metrics, such as answer relevancy, faithfulness, contextual precision, contextual recall, hallucination, bias, and toxicity.
- **Proxy Service for Enhanced LLM Monitoring:** Analyze your LLM interactions in greater detail with the optional proxy service, capturing comprehensive logs of requests and responses.
- **Cloud Integration:** Leverage the power of Google Cloud Platform (Firestore, Cloud Run, BigQuery, Vertex AI) for efficient data storage, execution, and analysis.

## Enabling Ragas in Litmus

By default, Ragas evaluation is disabled in Litmus. To enable it, you need to modify your test templates:

1. **Edit your test template:**
   - In the Litmus UI, navigate to the "Templates" page and click the "Edit" button next to the template you want to modify.
2. **Enable Ragas in the "LLM Evaluation Prompt" tab:**
   - Check the checkbox for **Ragas**.
3. **Save your template:**
   - Click the "Update Template" button to save your changes.

## Using Ragas

Once Ragas is enabled, Litmus will automatically use it to evaluate LLM responses for test runs that utilize the modified template. The results are embedded within the `assessment` field of the test case:

```json
{
  "status": "Passed",
  "response": {
    "output": "This is the answer"
  },
  "assessment": {
    "ragas_evaluation": {
      "answer_relevancy": 1.0,
      "context_recall": 1.0,
      "context_precision": 1.0,
      "harmfulness": 0.0,
      "answer_similarity": 1.0
    }
  }
}
```

## Configuring and Extending Ragas

Currently, Litmus utilizes a predefined set of Ragas metrics, including answer relevancy, context recall, context precision, harmfulness, and answer similarity. **Extending this set or adjusting metric thresholds would require code modifications within the worker service**.

For instance:

- **Adding new metrics:** To include additional Ragas metrics like `Aspect Critique` or `Answer Correctness`, you would need to modify the `ragas_metrics` list in the `ragas_eval.py` file within the worker service code.
- **Adjusting thresholds:** To modify the default thresholds for determining pass/fail, you would need to adjust the metric objects within the `ragas_eval.py` file.

**Note:** These modifications involve code changes and require rebuilding and redeploying the worker Docker image.


## Learn more about Limus:
- Evaluating with Ragas: https://google.github.io/litmus/evaluate-ragas
- Docs: https://google.github.io/litmus/
- Github Library: https://github.com/google/litmus