Repository: explodinggradients/ragas
Issue: InstructorRetryException with custom OpenAI API
Root cause: Interface Contract Mismatch

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am trying to compute the answer correctness metric on a single record. I have setup `gemma3` and `nomic-embed-text` using the LLM and embeddings factories. Their are both hosted by an in-house OpenAI-like API. However, the calculation returns a `InstructorRetryException error`.

Ragas version: 0.1.9
Python version:  3.12

**Code to Reproduce**
```python
from openai import AsyncOpenAI
from ragas.metrics.collections import AnswerCorrectness
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory

openai_client = AsyncOpenAI(
    api_key="this_key",
    base_url="https://custom.openai.com/",
)

llm = llm_factory(
    model='gemma3',
    client=openai_client,
     adapter='instructor',
)

embedder = embedding_factory(
    model='nomic-embed-text',
    client=openai_client,
)

scorer = AnswerCorrectness(llm=self.llm, embeddings=self.embedder)

for q in queries:
  result = asyncio.run( self.scorer.ascore(
      user_input=q.user_input,
      response=q.response,
      reference=q.reference,
  ))

  print(result)

```

Noticeably, I have also tried the `litellm` instructor but it doesn't work.

**Error trace**
```python
func = <bound method AsyncCompletions.create of <openai.resources.chat.completions.completions.AsyncCompletions object at 0x0000018599415D30>>
response_model = <class 'ragas.metrics.collections.answer_correctness.util.StatementGeneratorOutput'>
args = ()
kwargs = {'max_tokens': 1024, 'messages': [{'content': 'Given a question and an answer, analyze the complexity of each sentence...swer": "the cake is a lie"\n}\nOutput: ', 'role': 'user'}], 'model': 'gemma3', 'temperature': 0.01, ...}
context = None
max_retries = <AsyncRetrying object at 0x18599292510 (stop=<tenacity.stop.stop_after_attempt object at 0x0000018599292660>, wait=<te...B15F90>, before=<function before_nothing at 0x0000018521BC4B80>, after=<function after_nothing at 0x0000018521BC4E00>)>
strict = True, mode = <Mode.TOOLS: 'tool_call'>
hooks = <instructor.core.hooks.Hooks object at 0x0000018599415FD0>

    async def retry_async(
        func: Callable[T_ParamSpec, T_Retval],
        response_model: type[T_Model] | None,
        args: Any,
        kwargs: Any,
        context: dict[str, Any] | None = None,
        max_retries: int | AsyncRetrying = 1,
        strict: bool | None = None,
        mode: Mode = Mode.TOOLS,
        hooks: Hooks | None = None,
    ) -> T_Model | None:
        """
        Retry an asynchronous function upon specified exceptions.
    
        Args:
            func (Callable[T_ParamSpec, T_Retval]): The asynchronous function to retry.
            response_model (Optional[type[T_Model]]): The model to validate the response against.
            context (Optional[Dict[str, Any]]): Additional context for validation.
            args (Any): Positional arguments for the function.
            kwargs (Any): Keyword arguments for the function.
            max_retries (int | AsyncRetrying, optional): Maximum number of retries or an async retrying object. Defaults to 1.
            strict (Optional[bool], optional): Strict mode flag. Defaults to None.
            mode (Mode, optional): The mode of operation. Defaults to Mode.TOOLS.
            hooks (Optional[Hooks], optional): Hooks for emitting events. Defaults to None.
    
        Returns:
            T_Model | None: The processed response model or None.
    
        Raises:
            InstructorRetryException: If all retry attempts fail.
        """
        hooks = hooks or Hooks()
        total_usage = initialize_usage(mode)
        # Extract timeout from kwargs if available (for global timeout across retries)
        timeout = kwargs.get("timeout")
        max_retries = initialize_retrying(max_retries, is_async=True, timeout=timeout)
    
        # Pre-extract stream flag to avoid repeated lookup
        stream = kwargs.get("stream", False)
    
        # Track all failed attempts
        failed_attempts: list[FailedAttempt] = []
    
        try:
            response = None
            async for attempt in max_retries:
                logger.debug(f"Retrying, attempt: {attempt.retry_state.attempt_number}")
                with attempt:
                    try:
                        hooks.emit_completion_arguments(*args, **kwargs)
                        response: ChatCompletion = await func(*args, **kwargs)
                        hooks.emit_completion_response(response)
                        response = update_total_usage(
                            response=response, total_usage=total_usage
                        )
    
                        return await process_response_async(
                            response=response,
                            response_model=response_model,
                            validation_context=context,
                            strict=strict,
                            mode=mode,
                            stream=stream,
                        )
                    except (
                        ValidationError,
                        JSONDecodeError,
                        AsyncValidationError,
                        InstructorValidationError,
                    ) as e:
                        logger.debug(f"Parse error: {e}")
                        hooks.emit_parse_error(e)
    
                        # Track this failed attempt
                        failed_attempts.append(
                            FailedAttempt(
                                attempt_number=attempt.retry_state.attempt_number,
                                exception=e,
                                completion=response,
                            )
                        )
    
                        # Check if this is the last attempt
                        if isinstance(max_retries, AsyncRetrying) and hasattr(
                            max_retries, "stop"
                        ):
                            # For tenacity AsyncRetrying objects, check if next attempt would exceed limit
                            will_retry = (
                                attempt.retry_state.outcome is None
                                or not attempt.retry_state.outcome.failed
                            )
                            is_last_attempt = (
                                not will_retry
                                or attempt.retry_state.attempt_number
                                >= getattr(
                                    max_retries.stop, "max_attempt_number", float("inf")
                                )
                            )
                            if is_last_attempt:
                                hooks.emit_completion_last_attempt(e)
    
                        kwargs = handle_reask_kwargs(
                            kwargs=kwargs,
                            mode=mode,
                            response=response,
                            exception=e,
                            failed_attempts=failed_attempts,
                        )
                        raise e
                    except Exception as e:
                        # Emit completion:error for non-validation errors (API errors, network errors, etc.)
                        logger.debug(f"Completion error: {e}")
                        hooks.emit_completion_error(e)
    
                        # Track this failed attempt
                        failed_attempts.append(
                            FailedAttempt(
                                attempt_number=attempt.retry_state.attempt_number,
                                exception=e,
                                completion=response,
                            )
                        )
    
                        # Check if this is the last attempt for completion errors
                        if isinstance(max_retries, AsyncRetrying) and hasattr(
                            max_retries, "stop"
                        ):
                            will_retry = (
                                attempt.retry_state.outcome is None
                                or not attempt.retry_state.outcome.failed
                            )
                            is_last_attempt = (
                                not will_retry
                                or attempt.retry_state.attempt_number
                                >= getattr(
                                    max_retries.stop, "max_attempt_number", float("inf")
                                )
                            )
                            if is_last_attempt:
                                hooks.emit_completion_last_attempt(e)
                        raise e
        except RetryError as e:
            logger.debug(f"Retry error: {e}")
>           raise InstructorRetryException(
                e.last_attempt._exception,
                last_completion=response,
                n_attempts=attempt.retry_state.attempt_number,
                #! deprecate messages soon
                messages=extract_messages(
                    kwargs
                ),  # Use the optimized function instead of nested lookups
                create_kwargs=kwargs,
                total_usage=total_usage,
                failed_attempts=failed_attempts,
            ) from e
E           instructor.core.exceptions.InstructorRetryException: <failed_attempts>
E           
E           <generation number="1">
E           <exception>
E               Instructor does not support multiple tool calls, use List[Model] instead
E           </exception>
E           <completion>
E               ChatCompletion(id='chatcmpl-112', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='```json\n{\n    "statements": [\n        "Lyon is the capital of the Gauls."\n    ]\n}\n```', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=1765561816, model='gemma3:12b', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=90, prompt_tokens=1137, total_tokens=1227, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=None), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))
E           </completion>
E           </generation>
E           
E           <generation number="2">
E           <exception>
E               Instructor does not support multiple tool calls, use List[Model] instead
E           </exception>
E           <completion>
E               ChatCompletion(id='chatcmpl-748', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='```json\n{\n    "statements": [\n        "Lyon is the capital of the Gauls."\n    ]\n}\n```', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=1765561817, model='gemma3:12b', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=90, prompt_tokens=1137, total_tokens=1227, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=None), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))
E           </completion>
E           </generation>
E           
E           <generation number="3">
E           <exception>
E               Instructor does not support multiple tool calls, use List[Model] instead
E           </exception>
E           <completion>
E               ChatCompletion(id='chatcmpl-698', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='```json\n{\n    "statements": [\n        "Lyon is the capital of the Gauls."\n    ]\n}\n```', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=1765561817, model='gemma3:12b', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=90, prompt_tokens=1137, total_tokens=1227, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=None), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))
E           </completion>
E           </generation>
E           
E           </failed_attempts>
E           
E           <last_exception>
E               Instructor does not support multiple tool calls, use List[Model] instead
E           </last_exception>

..\.venv\Lib\site-packages\instructor\core\retry.py:442: InstructorRetryException
```





@NickBear-star Are you sure you're using `0.1.9` 
We only have metrics collections only in latest versions.


My mistake, 

I am using v0.4.1 (copy/paste from the wrong pyproject.toml)



I believe PR #2505 takes care of this issue as well. Would be available in next release.

Would you like to check it out?

I checked out the version 0.4.2.dev18+gcd593e25, was able to send the request to the API, but get a 422 error as response:
```python
                log.debug("Re-raising status error")
>               raise self._make_status_error_from_response(err.response) from None
E               openai.UnprocessableEntityError: Error code: 422 - {'detail': [{'type': 'list_type', 'loc': ['body', 'input'], 'msg': 'Input should be a valid list', 'input': 'the cake, it's a lie!'}]}

..\.venv\Lib\site-packages\openai\_base_client.py:1594: UnprocessableEntityError
```
As far as I understand, the API was expecting a list as input, while it gets a string. Is it an issue on the API side or on RAGAS side? As I said, it is a in-house API So maybe the OpenAI interface is not fully implemented... 

> As far as I understand, the API was expecting a list as input, while it gets a string. Is it an issue on the API side or on RAGAS side? As I said, it is a in-house API So maybe the OpenAI interface is not fully implemented...

It's probably a array/text issue.

Try sending your texts like array: `embedder.embed_texts(["Hello world"])[0]`

indeed, the error comes from the API which only supports array of strings, but not string. We have to configure the API to accept both string and array of strings. If I understand correctly, an OpenAI API is supposed to accept both input formats.