Repository: explodinggradients/ragas
Issue: Initialize a API based wrapper to set the llm for ragas evaluation
Root cause: Documentation Deficiency

How to initialize a llm having api values such as - api_path, url,header and workspace name ?

Can you please share an implementation how can I use these values to set the llm for context recall, context precision , faithfullness and response relevancy metrics?

@dosubot Please provide a guide to proceed further.


<!-- Answer -->
To initialize a language model (LLM) with API values such as `api_path`, URL, header, and workspace name for evaluating context recall, context precision, faithfulness, and response relevancy metrics in the Ragas framework, follow these steps:

1. **Set Environment Variables**: Start by setting your API keys and other necessary environment variables.

    ```python
    import os
    os.environ["LANGFUSE_SECRET_KEY"] = "sk-..."
    os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-..."
    ```

2. **Install Required Packages**: Make sure you have the necessary packages installed.

    ```python
    %pip install datasets ragas llama_index python-dotenv --upgrade
    ```

3. **Initialize the LLM and Embeddings**: Use the Langchain wrappers to initialize the LLM and embeddings.

    ```python
    from langchain_openai.chat_models import ChatOpenAI
    from langchain_openai.embeddings import OpenAIEmbeddings
    from ragas.llms import LangchainLLMWrapper
    from ragas.embeddings import LangchainEmbeddingsWrapper

    llm = ChatOpenAI()
    emb = OpenAIEmbeddings()
    ```

4. **Initialize Metrics**: Use the `init_ragas_metrics` function to initialize the metrics with the LLM and embeddings.

    ```python
    from ragas.run_config import RunConfig
    from ragas.metrics.base import MetricWithLLM, MetricWithEmbeddings
    from ragas.metrics import faithfulness, answer_relevancy, context_precision
    from ragas.metrics.critique import harmfulness

    metrics = [faithfulness, answer_relevancy, context_precision, harmfulness]

    def init_ragas_metrics(metrics, llm, embedding):
        for metric in metrics:
            if isinstance(metric, MetricWithLLM):
                metric.llm = llm
            if isinstance(metric, MetricWithEmbeddings):
                metric.embeddings = embedding
            run_config = RunConfig()
            metric.init(run_config)

    init_ragas_metrics(
        metrics,
        llm=LangchainLLMWrapper(llm),
        embedding=LangchainEmbeddingsWrapper(emb),
    )
    ```

5. **Score the Trace**: Define a function to score your trace with the chosen metrics.

    ```python
    async def score_with_ragas(query, chunks, answer):
        scores = {}
        for m in metrics:
            print(f"calculating {m.name}")
            scores[m.name] = await m.ascore(
                row={"question": query, "contexts": chunks, "answer": answer}
            )
        return scores
    ```

6. **Run the Evaluation**: Use the `score_with_ragas` function to evaluate the metrics for a given question, contexts, and answer.

    ```python
    question, contexts, answer = row["question"], row["contexts"], row["answer"]
    await score_with_ragas(question, contexts, answer)
    ```

This setup will allow you to initialize the LLM with the necessary API values and use it to evaluate the specified metrics <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/_langfuse.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/langfuse.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/9ce19314-d6d7-449f-a7b0-199e3b84eab0?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/9ce19314-d6d7-449f-a7b0-199e3b84eab0?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/9ce19314-d6d7-449f-a7b0-199e3b84eab0?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/9ce19314-d6d7-449f-a7b0-199e3b84eab0?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/9ce19314-d6d7-449f-a7b0-199e3b84eab0?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/9ce19314-d6d7-449f-a7b0-199e3b84eab0?feedback_type=other)</sup>


@dosu 

api_path = 'path'
url_35 = f'{api_path}/v1.1/Chat/Completions'
workspace_name = 'space_name'
 
header = {"api-key": "api_key"
         ,"Content-Type": "application/json",'workspaceName': workspace_name
         }

import requests


configuration_llm = 'llm' 
def get_response(configuration_llm,prompt_template,query):
  myobj = {
                'messages': [{'role': 'system','content': prompt_template},
                            {'role': 'user','content': query}],
                'model': configuration_llm,
                  'top_p': 0.5,
                  'temperature': 1,
                  'frequency_penalty': 0,
                  'presence_penalty': 0,
                  'max_tokens': 400,
                  'stream': False,
                  'stop': None,
                  'logitBias': None
                }
  x = requests.post(url_35 ,json=myobj, headers = header, verify=False)
  final_data = x.json()['choices'][0]['message']['content']
  return final_data

I have this method for llm calling , can you suggest how can we set llm for ragas using it?

<!-- Answer -->
To set the LLM for Ragas using your existing method for LLM calling, you can integrate it with the `llm_factory` function provided by Ragas. Here's how you can adapt your code:

1. **Define the LLM Factory Function**: Use the `llm_factory` function to create an LLM instance with your API configuration.

```python
from ragas.run_config import RunConfig
from ragas.llms import BaseRagasLLM

# Define your parameters
model = 'llm'  # Your model name
run_config = RunConfig(timeout=30)  # Set your desired timeout value
default_headers = {
    "api-key": "api_key",
    "Content-Type": "application/json",
    "workspaceName": "space_name"
}
base_url = f'{api_path}/v1.1/Chat/Completions'

# Create the LLM instance
llm_instance = BaseRagasLLM(
    model=model,
    run_config=run_config,
    default_headers=default_headers,
    base_url=base_url
)
```

2. **Integrate with Ragas**: Use this `llm_instance` in your Ragas evaluation setup. You can wrap it in `LangchainLLMWrapper` if needed.

```python
from ragas.llms import LangchainLLMWrapper

# Wrap the LLM instance if necessary
wrapped_llm = LangchainLLMWrapper(llm_instance)

# Use wrapped_llm in your Ragas evaluation
```

This setup allows you to use your existing API-based LLM calling method within the Ragas framework by configuring it as a `BaseRagasLLM` instance. Adjust the timeout and other parameters as needed for your specific use case.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/c3de2bd2-c85a-4678-a502-29cbcc4e6bb1?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/c3de2bd2-c85a-4678-a502-29cbcc4e6bb1?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/c3de2bd2-c85a-4678-a502-29cbcc4e6bb1?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/c3de2bd2-c85a-4678-a502-29cbcc4e6bb1?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c3de2bd2-c85a-4678-a502-29cbcc4e6bb1?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/c3de2bd2-c85a-4678-a502-29cbcc4e6bb1?feedback_type=other)</sup>


@dosu 

from ragas.run_config import RunConfig
from ragas.llms import BaseRagasLLM

# Define your parameters
model = configuration_llm  # Your model name
run_config = RunConfig(timeout=30)  # Set your desired timeout value
default_headers = {
    "api-key": "api_key",
    "Content-Type": "application/json",
    "workspaceName": workspace_name
}
base_url = f'{api_path}/v1.1/Chat/Completions'

# Create the LLM instance
llm_instance = BaseRagasLLM(
    model=model,
    run_config=run_config,
    default_headers=default_headers,
    base_url=base_url
)
from ragas.llms import LangchainLLMWrapper

# Wrap the LLM instance if necessary
wrapped_llm = LangchainLLMWrapper(llm_instance)
print(wrapped_llm)


TypeError: Can't instantiate abstract class BaseRagasLLM with abstract methods agenerate_text, generate_text

Can you help me with this?

<!-- Answer -->
To resolve the `TypeError` you're encountering, you need to implement the abstract methods `agenerate_text` and `generate_text` in a subclass of `BaseRagasLLM`. You cannot directly instantiate `BaseRagasLLM` because it contains abstract methods that must be defined in a subclass.

Here's how you can create a subclass that implements these methods:

```python
from ragas.llms import BaseRagasLLM
from ragas.llms.prompt import PromptValue
from ragas.llms.base import LLMResult, Generation

class CustomLLM(BaseRagasLLM):
    def generate_text(self, prompt: PromptValue, n=1, temperature=1.0, stop=None, callbacks=None) -> LLMResult:
        # Implement your API call logic here
        # For example, using your existing get_response function
        response_text = get_response(self.model, prompt.prompt_str)
        generations = [[Generation(text=response_text)] * n]
        return LLMResult(generations=generations)

    async def agenerate_text(self, prompt: PromptValue, n=1, temperature=1.0, stop=None, callbacks=None) -> LLMResult:
        # You can use the same logic as generate_text for async
        return self.generate_text(prompt, n, temperature, stop, callbacks)

# Create an instance of your custom LLM
llm_instance = CustomLLM(
    model=model,
    run_config=run_config,
    default_headers=default_headers,
    base_url=base_url
)

# Wrap the LLM instance if necessary
wrapped_llm = LangchainLLMWrapper(llm_instance)
print(wrapped_llm)

# Use wrapped_llm in your Ragas evaluation
```

This code defines a `CustomLLM` class that implements the required methods, allowing you to instantiate it and use it within the Ragas framework <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/tests/unit/llms/test_llm.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/31fee7f4-2e61-469a-bf4c-e64826074b36?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/31fee7f4-2e61-469a-bf4c-e64826074b36?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/31fee7f4-2e61-469a-bf4c-e64826074b36?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/31fee7f4-2e61-469a-bf4c-e64826074b36?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/31fee7f4-2e61-469a-bf4c-e64826074b36?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/31fee7f4-2e61-469a-bf4c-e64826074b36?feedback_type=other)</sup>