Repository: explodinggradients/ragas
Issue: llm_factory with litellm provider incompatible with evaluate() - missing agenerate_prompt
Root cause: Interface Contract Mismatch

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When using llm_factory with `provider="litellm"` (the recommended replacement for deprecated `LangchainLLMWrapper`), calling `evaluate()` fails with:

`AttributeError: 'InstructorLLM' object has no attribute 'agenerate_prompt'`

Ragas version: 0.3.8, 0.3.9
Python version: 3.12.12

**Code to Reproduce**

I am basically following this guide: [ragas/howtos/customizations/customize_models.md](https://github.com/vibrantlabsai/ragas/blob/d2e2315a/docs/howtos/customizations/customize_models.md)

```python
import litellm
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory

azure_configs = {  
    "api_base": os.getenv("AZURE_OPENAI_URL"),
    "api_key": os.getenv("AZURE_OPENAI_API_KEY"),
    "api_version": "2025-01-01-preview",
    "model_deployment": "gpt-4.1",
}

# Configure LiteLLM for Azure OpenAI (used by LLM calls)
litellm.api_base = azure_configs["api_base"]
litellm.api_key = azure_configs["api_key"]
litellm.api_version = azure_configs["api_version"]

# Create LLM using llm_factory with litellm provider
# Note: Use deployment name, not model name for Azure
# Important: Pass litellm.completion (the function), not the module
azure_llm = llm_factory(
    f"azure/{azure_configs['model_deployment']}",
    provider="litellm",
    client=litellm.completion,
)

metrics = [
    Faithfulness(llm=azure_llm),
    ContextRecall(llm=azure_llm),
    FactualCorrectness(llm=azure_llm),
]

samples = [
    SingleTurnSample(
        user_input="Who introduced the theory of relativity?",
        retrieved_contexts=[
            "Albert Einstein proposed the theory of relativity in the early 20th century."
        ],
        response="Albert Einstein introduced the theory of relativity.",
        reference="Albert Einstein proposed the theory of relativity."
    )
]

# Evaluate
results = evaluate(
    dataset=EvaluationDataset(samples=samples),
    metrics=metrics,
    raise_exceptions=True
)
```

**Error trace**
```
AttributeError                            Traceback (most recent call last)
Cell In[3], line 52
     32 samples = [
     33     SingleTurnSample(
     34         user_input="Who introduced the theory of relativity?",
   (...)     48     ),
     49 ]
     51 # Evaluate
---> 52 results = evaluate(
     53     dataset=EvaluationDataset(samples=samples),
     54     metrics=metrics,
     55     raise_exceptions=True
     56 )
     58 print(results)

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/_analytics.py:277, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    274 @wraps(func)
    275 def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
    276     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 277     result = func(*args, **kwargs)
    278     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    280     return result

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/evaluation.py:461, in evaluate(dataset, metrics, llm, embeddings, experiment_name, callbacks, run_config, token_usage_parser, raise_exceptions, column_map, show_progress, batch_size, _run_id, _pbar, return_executor, allow_nest_asyncio)
    457 else:
    458     # Default behavior: use nest_asyncio for backward compatibility (Jupyter notebooks)
    459     from ragas.async_utils import run
--> 461     return run(_async_wrapper())

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/async_utils.py:156, in run(async_func, allow_nest_asyncio)
    148     loop_type = type(loop).__name__
    149     raise RuntimeError(
    150         f"Cannot execute nested async code with {loop_type}. "
    151         f"uvloop does not support nested event loop execution. "
    152         f"Please use asyncio's standard event loop in Jupyter environments, "
    153         f"or refactor your code to avoid nested async calls."
    154     )
--> 156 return asyncio.run(coro)

File ~/dev/ragas/.venv/lib/python3.12/site-packages/nest_asyncio.py:30, in _patch_asyncio.<locals>.run(main, debug)
     28 task = asyncio.ensure_future(main)
     29 try:
---> 30     return loop.run_until_complete(task)
     31 finally:
     32     if not task.done():

File ~/dev/ragas/.venv/lib/python3.12/site-packages/nest_asyncio.py:98, in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File ~/.pyenv/versions/3.12.12/lib/python3.12/asyncio/futures.py:202, in Future.result(self)
    200 self.__log_traceback = False
    201 if self._exception is not None:
--> 202     raise self._exception.with_traceback(self._exception_tb)
    203 return self._result

File ~/.pyenv/versions/3.12.12/lib/python3.12/asyncio/tasks.py:314, in Task.__step_run_and_handle_result(***failed resolving arguments***)
    310 try:
    311     if exc is None:
    312         # We use the `send` method directly, because coroutines
    313         # don't have `__iter__` and `__next__` methods.
--> 314         result = coro.send(None)
    315     else:
    316         result = coro.throw(exc)

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/evaluation.py:434, in evaluate.<locals>._async_wrapper()
    433 async def _async_wrapper():
--> 434     return await aevaluate(
    435         dataset=dataset,
    436         metrics=metrics,
    437         llm=llm,
    438         embeddings=embeddings,
    439         experiment_name=experiment_name,
    440         callbacks=callbacks,
    441         run_config=run_config,
    442         token_usage_parser=token_usage_parser,
    443         raise_exceptions=raise_exceptions,
    444         column_map=column_map,
    445         show_progress=show_progress,
    446         batch_size=batch_size,
    447         _run_id=_run_id,
    448         _pbar=_pbar,
    449         return_executor=return_executor,
    450     )

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/evaluation.py:295, in aevaluate(dataset, metrics, llm, embeddings, experiment_name, callbacks, run_config, token_usage_parser, raise_exceptions, column_map, show_progress, batch_size, _run_id, _pbar, return_executor)
    292     if not evaluation_group_cm.ended:
    293         evaluation_rm.on_chain_error(e)
--> 295     raise e
    296 else:
    297     # evalution run was successful
    298     # now lets process the results
    299     cost_cb = ragas_callbacks["cost_cb"] if "cost_cb" in ragas_callbacks else None

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/evaluation.py:271, in aevaluate(dataset, metrics, llm, embeddings, experiment_name, callbacks, run_config, token_usage_parser, raise_exceptions, column_map, show_progress, batch_size, _run_id, _pbar, return_executor)
    268 scores: t.List[t.Dict[str, t.Any]] = []
    269 try:
    270     # get the results using async method
--> 271     results = await executor.aresults()
    272     if results == []:
    273         raise ExceptionInRunner()

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/executor.py:200, in Executor.aresults(self)
    193 async def aresults(self) -> t.List[t.Any]:
    194     """
    195     Execute all submitted jobs and return their results asynchronously.
    196     The results are returned in the order of job submission.
    197 
    198     This is the async entry point for executing async jobs when already in an async context.
    199     """
--> 200     results = await self._process_jobs()
    201     sorted_results = sorted(results, key=lambda x: x[0])
    202     return [r[1] for r in sorted_results]

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/executor.py:131, in Executor._process_jobs(self)
    129 if self.pbar is None:
    130     with pbm.create_single_bar(len(jobs_to_process)) as internal_pbar:
--> 131         await self._process_coroutines(
    132             jobs_to_process, internal_pbar, results, max_workers
    133         )
    134 else:
    135     await self._process_coroutines(
    136         jobs_to_process, self.pbar, results, max_workers
    137     )

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/executor.py:189, in Executor._process_coroutines(self, jobs, pbar, results, max_workers)
    184 async for result in process_futures(
    185     as_completed(coroutines, max_workers, cancel_check=self.is_cancelled)
    186 ):
    187     # If jobs are configured to raise exceptions, propagate immediately
    188     if isinstance(result, Exception) and self.raise_exceptions:
--> 189         raise result
    190     results.append(result)
    191     pbar.update(1)

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/async_utils.py:114, in process_futures(futures)
    112 for future in futures:
    113     try:
--> 114         result = await future
    115     except asyncio.CancelledError:
    116         raise  # Re-raise CancelledError to ensure proper cancellation

File ~/.pyenv/versions/3.12.12/lib/python3.12/asyncio/tasks.py:631, in as_completed.<locals>._wait_for_one()
    628 if f is None:
    629     # Dummy value from _on_timeout().
    630     raise exceptions.TimeoutError
--> 631 return f.result()

File ~/.pyenv/versions/3.12.12/lib/python3.12/asyncio/futures.py:202, in Future.result(self)
    200 self.__log_traceback = False
    201 if self._exception is not None:
--> 202     raise self._exception.with_traceback(self._exception_tb)
    203 return self._result

File ~/.pyenv/versions/3.12.12/lib/python3.12/asyncio/tasks.py:314, in Task.__step_run_and_handle_result(***failed resolving arguments***)
    310 try:
    311     if exc is None:
    312         # We use the `send` method directly, because coroutines
    313         # don't have `__iter__` and `__next__` methods.
--> 314         result = coro.send(None)
    315     else:
    316         result = coro.throw(exc)

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/async_utils.py:77, in as_completed.<locals>.sema_coro(coro)
     75 async def sema_coro(coro):
     76     async with semaphore:
---> 77         return await coro

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/executor.py:73, in Executor.wrap_callable_with_index.<locals>.wrapped_callable_async(*args, **kwargs)
     71 except Exception as e:
     72     if self.raise_exceptions:
---> 73         raise e
     74     else:
     75         exec_name = type(e).__name__

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/executor.py:69, in Executor.wrap_callable_with_index.<locals>.wrapped_callable_async(*args, **kwargs)
     67 async def wrapped_callable_async(*args, **kwargs) -> t.Tuple[int, t.Any]:
     68     try:
---> 69         result = await callable(*args, **kwargs)
     70         return counter, result
     71     except Exception as e:

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/metrics/base.py:561, in SingleTurnMetric.single_turn_ascore(self, sample, callbacks, timeout)
    559     if not group_cm.ended:
    560         rm.on_chain_error(e)
--> 561     raise e
    562 else:
    563     if not group_cm.ended:

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/metrics/base.py:554, in SingleTurnMetric.single_turn_ascore(self, sample, callbacks, timeout)
    547 rm, group_cm = new_group(
    548     self.name,
    549     inputs=sample.to_dict(),
    550     callbacks=callbacks,
    551     metadata={"type": ChainType.METRIC},
    552 )
    553 try:
--> 554     score = await asyncio.wait_for(
    555         self._single_turn_ascore(sample=sample, callbacks=group_cm),
    556         timeout=timeout,
    557     )
    558 except Exception as e:
    559     if not group_cm.ended:

File ~/.pyenv/versions/3.12.12/lib/python3.12/asyncio/tasks.py:520, in wait_for(fut, timeout)
    517         raise TimeoutError from exc
    519 async with timeouts.timeout(timeout):
--> 520     return await fut

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/metrics/_faithfulness.py:200, in Faithfulness._single_turn_ascore(self, sample, callbacks)
    196 async def _single_turn_ascore(
    197     self, sample: SingleTurnSample, callbacks: Callbacks
    198 ) -> float:
    199     row = sample.to_dict()
--> 200     return await self._ascore(row, callbacks)

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/metrics/_faithfulness.py:208, in Faithfulness._ascore(self, row, callbacks)
    203 """
    204 returns the NLI score for each (q, c, a) pair
    205 """
    206 assert self.llm is not None, "LLM is not set"
--> 208 statements = await self._create_statements(row, callbacks)
    209 statements = statements.statements
    210 if statements == []:

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/metrics/_faithfulness.py:174, in Faithfulness._create_statements(self, row, callbacks)
    171 text, question = row["response"], row["user_input"]
    173 prompt_input = StatementGeneratorInput(question=question, answer=text)
--> 174 statements = await self.statement_generator_prompt.generate(
    175     llm=self.llm,
    176     data=prompt_input,
    177     callbacks=callbacks,
    178 )
    180 return statements

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/prompt/pydantic_prompt.py:168, in PydanticPrompt.generate(self, llm, data, temperature, stop, callbacks, retries_left)
    165 callbacks = callbacks or []
    167 # this is just a special case of generate_multiple
--> 168 output_single = await self.generate_multiple(
    169     llm=llm,
    170     data=data,
    171     n=1,
    172     temperature=temperature,
    173     stop=stop,
    174     callbacks=callbacks,
    175     retries_left=retries_left,
    176 )
    177 return output_single[0]

File ~/dev/ragas/.venv/lib/python3.12/site-packages/ragas/prompt/pydantic_prompt.py:238, in PydanticPrompt.generate_multiple(self, llm, data, n, temperature, stop, callbacks, retries_left)
    236     # LangChain doesn't support n parameter directly, so we batch multiple prompts
    237     prompts = t.cast(t.List[t.Any], [prompt_value for _ in range(n)])
--> 238     resp = await langchain_llm.agenerate_prompt(
    239         prompts,
    240         stop=stop,
    241         callbacks=prompt_cb,
    242     )
    243 else:
    244     # This is a Ragas LLM - use generate()
    245     ragas_llm = t.cast(BaseRagasLLM, llm)

AttributeError: 'InstructorLLM' object has no attribute 'agenerate_prompt'
```

**Expected behavior**
It works :)

**Additional context**
If I wrap it in a `LangchainLLMWrapper`:
```
from langchain_openai import AzureChatOpenAI

azure_llm_client = AzureChatOpenAI(
    azure_endpoint="https://aoai-atlantix-tests.openai.azure.com/",
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_version="2025-01-01-preview",
    azure_deployment="gpt-4.1",
    temperature=0,
)

azure_llm = LangchainLLMWrapper(azure_llm_client)

```

it works normally, but it raises this deprecation warning:
```
DeprecationWarning: LangchainLLMWrapper is deprecated and will be removed in a future version. Use llm_factory instead: from openai import OpenAI; from ragas.llms import llm_factory; llm = llm_factory('gpt-4o-mini', client=OpenAI(api_key='...'))
  azure_llm = LangchainLLMWrapper(azure_llm_client)
```

Thanks for raising this @DavidOrti 

The evaluate() function is being deprecated so it's not supposed to work with llm_factory.

We recommend migrating to the `@experiment` API.

I see, that makes sense now. Thanks @anistark!