Repository: explodinggradients/ragas
Issue: Error w/ evaluate function using LLamaIndex AzureOpenAI models -  maybe integrate LlamaIndexLLMWrapper into code
Root cause: Architectural Constraint

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Further request for LLamaIndex support regarding Azure OpenAI models (`from llama_index.llms.azure_openai import AzureOpenAI`). Getting error `AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'`

My guess at a solution is to use [LlamaIndexLLMWrapper](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py#L119). But unlike Langchain this wrapper is not integrated in `evaluation.py`. 


Ragas version: `0.1.16`
Python version: `3.10.14`

**Code to Reproduce**

```python
from datasets import Dataset 
from ragas.metrics import context_recall
from ragas import evaluate
from ragas.run_config import RunConfig
from ragas.llms.base import LlamaIndexLLMWrapper
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from llama_index.llms.azure_openai import AzureOpenAI

gpt_35 = AzureOpenAI(model="gpt-35-turbo",  # model name
                               engine="gpt-35-turbo",  # deployment name
                               temperature=0.1,
                               max_tokens=None,
                               api_version="2024-06-01",
                               # azure specific
                               azure_endpoint="https://XXXXXXX",
                               azure_ad_token_provider=get_bearer_token_provider(DefaultAzureCredential(),
                                                                                 "https://cognitiveservices.azure.com/.default"),
                               use_azure_ad=True)


data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[context_recall], llm=gpt_35) # LlamaIndexLLMWrapper(llm=critic_llm, run_config=RunConfig(timeout=30))
score.to_pandas()

```


**Error trace**


```
File ~//.venv_anaragsystem_poetry_v5/lib/python3.10/site-packages/ragas/_analytics.py:129, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    126 @wraps(func)
    127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    128     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129     result = func(*args, **kwargs)
    130     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    132     return result

File ~//.venv_anaragsystem_poetry_v5/lib/python3.10/site-packages/ragas/evaluation.py:206, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
    203             reproducable_metrics.append(i)
    205     # init all the models
--> 206     metric.init(run_config)
    208 executor = Executor(
    209     desc="Evaluating",
    210     keep_progress_bar=True,
    211     raise_exceptions=raise_exceptions,
    212     run_config=run_config,
    213 )
    215 # Ragas Callbacks
    216 # init the callbacks we need for various tasks

File ~//.venv_anaragsystem_poetry_v5/lib/python3.10/site-packages/ragas/metrics/base.py:153, in MetricWithLLM.init(self, run_config)
    149 if self.llm is None:
    150     raise ValueError(
    151         f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run."  # noqa
    152     )
--> 153 self.llm.set_run_config(run_config)
```

```
AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'
```

**Expected behavior**
Expecting the code to run like in issue #1277

**Additional context**
I see that in [evaluation.py](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py#L181) there is integrated support for Langchain. This kind of support would be nice for LLamaIndex as well. 

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Answer -->
I found a similar issue that was resolved by integrating the `LlamaIndexLLMWrapper` into the code. Here’s how you can resolve the `AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'` by using the `LlamaIndexLLMWrapper`:

1. **Wrap your AzureOpenAI instance with `LlamaIndexLLMWrapper`**:
   ```python
   from ragas.llms.base import LlamaIndexLLMWrapper, RunConfig
   from azure.identity import DefaultAzureCredential, get_bearer_token_provider
   from llama_index.llms.azure_openai import AzureOpenAI

   gpt_35 = AzureOpenAI(
       model="gpt-35-turbo",
       engine="gpt-35-turbo",
       temperature=0.1,
       max_tokens=None,
       api_version="2024-06-01",
       azure_endpoint="https://XXXXXXX",
       azure_ad_token_provider=get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"),
       use_azure_ad=True
   )

   # Create a run configuration
   run_config = RunConfig(timeout=30)

   # Wrap the AzureOpenAI instance with LlamaIndexLLMWrapper
   llama_index_llm = LlamaIndexLLMWrapper(llm=gpt_35, run_config=run_config)

   data_samples = {
       'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
       'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
       'contexts': [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
                    ['The Green Bay Packers...Green Bay, Wisconsin.', 'The Packers compete...Football Conference']],
       'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
   }

   dataset = Dataset.from_dict(data_samples)
   score = evaluate(dataset, metrics=[context_recall], llm=llama_index_llm)
   score.to_pandas()
   ```

By wrapping your `AzureOpenAI` instance with `LlamaIndexLLMWrapper`, you ensure that the `set_run_config` method is available, thus resolving the `AttributeError` <sup>[[1]](https://github.com/explodinggradients/ragas/issues/595)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I tried wrapping my `AzureOpenAI` from `LLamaIndex` in the Ragas wrapper (`LlamaIndexLLMWrapper`) but I still get the same error....

```python
from ragas.llms.base import LlamaIndexLLMWrapper, RunConfig

# Create a run configuration
run_config = RunConfig(timeout=30)

# Wrap the AzureOpenAI instance with LlamaIndexLLMWrapper
llama_index_llm = LlamaIndexLLMWrapper(llm=gpt_35, run_config=run_config)

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts': [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
                 ['The Green Bay Packers...Green Bay, Wisconsin.', 'The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[context_recall], llm=llama_index_llm)
score.to_pandas() 
```

**Error:** 
`AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'`


<!-- Answer -->
To resolve the `AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'` when wrapping your `AzureOpenAI` instance from `LLamaIndex` in the `LlamaIndexLLMWrapper`, you need to ensure that the `LlamaIndexLLMWrapper` class includes the `set_run_config` method. Here is the complete implementation of the `LlamaIndexLLMWrapper` class with the `set_run_config` method:

```python
class LlamaIndexLLMWrapper(BaseRagasLLM):
    """
    An Adaptor for LlamaIndex LLMs
    """

    def __init__(
        self,
        llm: BaseLLM,
        run_config: t.Optional[RunConfig] = None,
    ):
        self.llm = llm

        self._signature = ""
        if type(self.llm).__name__.lower() == "bedrock":
            self._signature = "bedrock"
        if run_config is None:
            run_config = RunConfig()
        self.set_run_config(run_config)

    def set_run_config(self, run_config: RunConfig):
        self.run_config = run_config

    def check_args(
        self,
        n: int,
        temperature: float,
        stop: t.Optional[t.List[str]],
        callbacks: Callbacks,
    ) -> dict[str, t.Any]:
        if n != 1:
            logger.warning("n values greater than 1 not support for LlamaIndex LLMs")
        if temperature != 1e-8:
            logger.info("temperature kwarg passed to LlamaIndex LLM")
        if stop is not None:
            logger.info("stop kwarg passed to LlamaIndex LLM")
        if callbacks is not None:
            logger.info(
                "callbacks not supported for LlamaIndex LLMs, ignoring callbacks"
            )
        if self._signature == "bedrock":
            return {"temperature": temperature}
        else:
            return {
                "n": n,
                "temperature": temperature,
                "stop": stop,
            }

    def generate_text(
        self,
        prompt: PromptValue,
        n: int = 1,
        temperature: float = 1e-8,
        stop: t.Optional[t.List[str]] = None,
        callbacks: Callbacks = None,
    ) -> LLMResult:
        kwargs = self.check_args(n, temperature, stop, callbacks)
        li_response = self.llm.complete(prompt.to_string(), **kwargs)

        return LLMResult(generations=[[Generation(text=li_response.text)]])

    async def agenerate_text(
        self,
        prompt: PromptValue,
        n: int = 1,
        temperature: t.Optional[float] = None,
        stop: t.Optional[t.List[str]] = None,
        callbacks: Callbacks = None,
    ) -> LLMResult:
        if temperature is None:
            temperature = 1e-8

        kwargs = self.check_args(n, temperature, stop, callbacks)
        li_response = await self.llm.acomplete(prompt.to_string(), **kwargs)

        return LLMResult(generations=[[Generation(text=li_response.text)]])
```

Make sure that your `LlamaIndexLLMWrapper` class includes the `set_run_config` method as shown above. This should resolve the `AttributeError` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I will try this @dosu but this code is dependent on a lot of package imports. 

In general, is there a reason the existing `class LlamaIndexLLMWrapper(BaseRagasLLM)` [link](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py#L217C1-L217C41) in the Ragas repo does NOT have the `set_run_config()` function implemented within the class

Contrasting to  `class LangchainLLMWrapper(BaseRagasLLM)` which has the `set_run_config()` function implemented 



<!-- Answer -->
The `LlamaIndexLLMWrapper` class in the Ragas repository does have the `set_run_config()` function implemented. Here is the relevant part of the code:

```python
class LlamaIndexLLMWrapper(BaseRagasLLM):
    """
    An Adaptor for LlamaIndex LLMs
    """

    def __init__(
        self,
        llm: BaseLLM,
        run_config: t.Optional[RunConfig] = None,
    ):
        self.llm = llm

        self._signature = ""
        if type(self.llm).__name__.lower() == "bedrock":
            self._signature = "bedrock"
        if run_config is None:
            run_config = RunConfig()
        self.set_run_config(run_config)

    def set_run_config(self, run_config: RunConfig):
        self.run_config = run_config
```

The `set_run_config()` function is called in the constructor of the `LlamaIndexLLMWrapper` class, similar to the `LangchainLLMWrapper` class. Therefore, the `LlamaIndexLLMWrapper` class does have the `set_run_config()` function implemented <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I see that now but either way I am still getting 😢 :

AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config' 

@Dosu i face the same problem. Is there a fix for this? 


<!-- Answer -->
This is a known integration issue: Ragas expects the LLM you pass in to have a set_run_config method, but the raw AzureOpenAI from LlamaIndex doesn't provide it. Even though LlamaIndexLLMWrapper is supposed to solve this, users still report the same error—usually because somewhere in the pipeline, the unwrapped LLM is still being used or passed to a metric, not the wrapper.

There’s no official fix yet, and support for LlamaIndex models in Ragas is incomplete. The best-known workaround is to use Langchain’s AzureOpenAI with LangchainLLMWrapper instead, which has been confirmed to work for similar cases ([example](https://github.com/explodinggradients/ragas/issues/2006)). Make sure you wrap the LLM before passing it to evaluate and when initializing metrics.

If you must use LlamaIndex, double-check that every reference to the LLM in your code uses the wrapper, not the raw AzureOpenAI instance. If you still see the error, it’s likely due to a spot where the wrapper isn’t being used.

Let me know if you want a code snippet for the Langchain workaround or have more details about your setup!


<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*

---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/89b944d7-5436-4e7b-8f4e-6c4efb9263a5?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/explodinggradients/ragas/issues/1278)