Repository: explodinggradients/ragas
Issue: LLM factory does not work with self-hosted LLM
Root cause: Unimplemented Feature Gap

[X ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I cannot run any metric with a self hosted llm

Ragas version: 0.4.0
Python version: 3.12.1

**Code to Reproduce**
```

openai_client = AsyncOpenAI(
        api_key=settings.llm_api_key,
        base_url=settings.llm_base_url
    )
MODEL_NAME = settings.llm_model # <-- your actual model name here  # <--- jouw modelnaam
llm = llm_factory(model=MODEL_NAME, client=openai_client, provider="openai")

async def main() -> None:
    metric = Faithfulness(llm=llm)
    result = await metric.ascore(
        user_input="What is AI?",
        response="AI is artificial intelligence...",
        retrieved_contexts=["Context 1", "Context 2"]
    )
    print(score)

asyncio.run(main())
```

**Error trace**
Traceback (most recent call last):
  File "instructor\core\retry.py", line 439, in retry_async
    raise e
  File "instructor\core\retry.py", line 355, in retry_async
    return await process_response_async(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "instructor\processing\response.py", line 240, in process_response_async
    model = response_model.from_response(  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "instructor\processing\function_calls.py", line 227, in from_response
    return cls.parse_tools(completion, validation_context, strict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File instructor\processing\function_calls.py", line 684, in parse_tools
    assert len(message.tool_calls or []) == 1, (
AssertionError: Instructor does not support multiple tool calls, use List[Model] instead

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "instructor\core\retry.py", line 344, in retry_async
    async for attempt in max_retries:
  File "tenacity\asyncio\__init__.py", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tenacity\asyncio\__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tenacity\_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "tenacity\__init__.py", line 421, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x2 state=finished raised AssertionError>]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "ragas_tryout.py", line 67, in <module>
    asyncio.run(main())
  File "asyncio\runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncio\base_events.py", line 684, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "ragas_tryout.py", line 60, in main
    result = await metric.ascore(
             ^^^^^^^^^^^^^^^^^^^^
  File "ragas\metrics\collections\faithfulness\metric.py", line 125, in ascore
    verdicts = await self._create_verdicts(statements, context_str)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ragas\metrics\collections\faithfulness\metric.py", line 145, in _create_verdicts
    result = await self.llm.agenerate(prompt_str, NLIStatementOutput)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ragas\llms\base.py", line 999, in agenerate
    result = await self.client.chat.completions.create(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "instructor\core\client.py", line 649, in create
    return await self.create_fn(
           ^^^^^^^^^^^^^^^^^^^^^
  File "instructor\core\patch.py", line 188, in new_create_async
    response = await retry_async(
               ^^^^^^^^^^^^^^^^^^
  File "instructor\core\retry.py", line 442, in retry_async
    raise InstructorRetryException(
instructor.core.exceptions.InstructorRetryException: <failed_attempts>

**Expected behavior**
A result to be outputted






Thanks for raising the issue @floorkouwenberg 🙌🏼 

The issue seems to be multiple tool calls, which is an issue with instructor.

Can you try using `litellm` adapter:

```py
llm = llm_factory(
    model=MODEL_NAME,
    client=openai_client,
    provider="openai",
    adapter="litellm"  # ← Update this: Use litellm adapter
)
```



I tried, but that also does not work: 

```
File "ragas\metrics\collections\faithfulness\metric.py", line 117, in ascore
    statements = await self._create_statements(user_input, response)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ragas\metrics\collections\faithfulness\metric.py", line 136, in _create_statements
    result = await self.llm.agenerate(prompt_str, StatementGeneratorOutput)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ragas\llms\litellm_llm.py", line 224, in agenerate
    raise TypeError(
TypeError: Cannot use agenerate() with a synchronous client. Use generate() instead.
```

My llm instance is hosted with VLLM and OWUI

@floorkouwenberg Thanks for testing it out. 

Need to add support for JSON mode for instructor for local llm support. Tools mode is failing cause of multiple tool calls in local llm. 

It works now! I did file another bug report, as ragas still tries to connect to openai for a tokenizer that i do not need.