Repository: explodinggradients/ragas
Issue: Running RAGAS behind a proxy
Root cause: Documentation Deficiency

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am using Ragas with in-house openAI API. When I am using it directly, everything is fine. However, when I am trying  to use it through our proxy, I encountered the infamous APIConnectionError. I have to specify that when I am using the API directly with the ChatOpenAI instance, everything is fine, with ou without the proxy (same thing for the proxy). So, the issue is really at the Ragas level

Ragas version: 0.3.8
Python version: 3.12

**Code to Reproduce**
```python
import httpx
import nest_asyncio
from langchain_openai import ChatOpenAI
from langchain_core.embeddings import Embeddings
from ragas import evaluate, EvaluationDataset
from ragas.metrics import answer_correctness

client = httpx.Client(proxy="http://user:passwd@proxy-server:8080")

agent = ChatOpenAI(
        base_url=config.agent.url,
        api_key=config.session.token,
        model=config.agent.model_name,
        http_client=client
)

# The agent run perfectly with the proxy
answer = agent.invoke("Answer 'pong' if you get a 'ping', answer 'crash' otherwise:\n'ping'")

# I cannot provide the exact embedder code, but it is not the culprit here
embedder = Embeddings()


nest_asyncio.apply()

evaluation = evaluate(
                dataset,
                metrics=[answer_correctness],
                llm=agent,
                embeddings=embedder,
                allow_nest_asyncio=True,
                raise_exceptions=True,
                show_progress=False,
)
```

**Error trace**
```
@contextlib.contextmanager
    def map_httpcore_exceptions() -> typing.Iterator[None]:
        global HTTPCORE_EXC_MAP
        if len(HTTPCORE_EXC_MAP) == 0:
            HTTPCORE_EXC_MAP = _load_httpcore_exceptions()
        try:
>           yield
...
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

map = {<class 'TimeoutError'>: <class 'httpcore.ConnectTimeout'>, <class 'OSError'>: <class 'httpcore.ConnectError'>, <class 'anyio.BrokenResourceError'>: <class 'httpcore.ConnectError'>}

    @contextlib.contextmanager
    def map_exceptions(map: ExceptionMapping) -> typing.Iterator[None]:
        try:
            yield
        except Exception as exc:  # noqa: PIE786
            for from_exc, to_exc in map.items():
                if isinstance(exc, from_exc):
>                   raise to_exc(exc) from exc
E                   httpcore.ConnectError: All connection attempts failed

..\.venv\Lib\site-packages\httpcore\_exceptions.py:14: ConnectError
The above exception was the direct cause of the following exception:

self = <openai.AsyncOpenAI object at 0x000001F79C995160>
cast_to = <class 'openai.types.chat.chat_completion.ChatCompletion'>
options = FinalRequestOptions(method='post', url='/chat/completions', params={}, headers={'X-Stainless-Raw-Response': 'true'}, m...s."\n}\nOutput: ', 'role': 'user'}], 'model': 'gemma3', 'n': 1, 'stream': False, 'temperature': 0.01}, extra_json=None)

    async def request(
        self,
        cast_to: Type[ResponseT],
        options: FinalRequestOptions,
        *,
        stream: bool = False,
        stream_cls: type[_AsyncStreamT] | None = None,
    ) -> ResponseT | _AsyncStreamT:
        if self._platform is None:
            # `get_platform` can make blocking IO calls so we
            # execute it earlier while we are in an async context
            self._platform = await asyncify(get_platform)()
    
        cast_to = self._maybe_override_cast_to(cast_to, options)
    
        # create a copy of the options we were given so that if the
        # options are mutated later & we then retry, the retries are
        # given the original options
        input_options = model_copy(options)
        if input_options.idempotency_key is None and input_options.method.lower() != "get":
            # ensure the idempotency key is reused between requests
            input_options.idempotency_key = self._idempotency_key()
    
        response: httpx.Response | None = None
        max_retries = input_options.get_max_retries(self.max_retries)
    
        retries_taken = 0
        for retries_taken in range(max_retries + 1):
            options = model_copy(input_options)
            options = await self._prepare_options(options)
    
            remaining_retries = max_retries - retries_taken
            request = self._build_request(options, retries_taken=retries_taken)
            await self._prepare_request(request)
    
            kwargs: HttpxSendArgs = {}
            if self.custom_auth is not None:
                kwargs["auth"] = self.custom_auth
    
            if options.follow_redirects is not None:
                kwargs["follow_redirects"] = options.follow_redirects
    
            log.debug("Sending HTTP Request: %s %s", request.method, request.url)
    
            response = None
            try:
>               response = await self._client.send(
                    request,
                    stream=stream or self._should_stream_response_body(request=request),
                    **kwargs,
                )

..\.venv\Lib\site-packages\openai\_base_client.py:1529: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
...
                log.debug("Raising connection error")
>               raise APIConnectionError(request=request) from err
E               openai.APIConnectionError: Connection error.

..\.venv\Lib\site-packages\openai\_base_client.py:1561: APIConnectionError
```

**Expected behavior**
The evaluate() function has to manage proxy connection. Would it change something if the proxy parameters are set through environment variables, such as:
```
os.environ["HTTP_PROXY"] = "http://proxy-server:8080"
os.environ["HTTPS_PROXY"] = "http://proxy-server:8080"
```



@NickBear-star first thing please use the new metrics collections. We'll be deprecating `evaluate` soon: 

```py
from ragas.metrics.collections import AnswerCorrectness
```

Rest should work for you as is.

Something like:

```py
import httpx
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import AnswerCorrectness
from ragas import aevaluate

async def evaluate_with_proxy():
    # Create proxied client
    http_client = httpx.AsyncClient(
        mounts={
            "https://":
httpx.AsyncHTTPTransport(proxy="http://proxy.company.com:8080"),
            "http://":
httpx.AsyncHTTPTransport(proxy="http://proxy.company.com:8080"),
        }
    )

    # Create OpenAI client with proxy
    client = AsyncOpenAI(api_key="key", http_client=http_client)

    # Both use the SAME proxied client
    llm = llm_factory("gpt-4o-mini", client=client)
    embeddings = embedding_factory("openai", model="text-embedding-3-small",
client=client)

    # Create and evaluate metric
    metric = AnswerCorrectness(llm=llm, embeddings=embeddings)
    result = await aevaluate(dataset, metrics=[metric], llm=llm)

    return result
```


It seems the issue was answered, closing this now.

Sorry for the late answer, 

I wasn't able to test the code until now. In brief, it doesn't work yet because the `AnswerCorrectness` instance is not recognize as a metric:
```python
[...]
        # Check metrics are correct type
        if not isinstance(metrics, (type(None), list)):
            raise TypeError(
                "Metrics should be provided in a list, e.g: metrics=[BleuScore()]"
            )
    
        if isinstance(metrics, list) and any(not isinstance(m, Metric) for m in metrics):
>           raise TypeError(
                "All metrics must be initialised metric objects, e.g: metrics=[BleuScore(), AspectCritic()]"
            )
E           TypeError: All metrics must be initialised metric objects, e.g: metrics=[BleuScore(), AspectCritic()]

..\.venv\Lib\site-packages\ragas\evaluation.py:123: TypeError
```

I have change the code to follow the [documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/?h=#answer-correctness):
```python
from httpx import AsyncClient
from ragas.metrics.collections import AnswerCorrectness
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory

from core.models.query import Query

class RagasAnswerCorrectnessScorer:
    """
     Management of the RAGAS answer correctness score calculation
    """

    def __init__(self, http_client=AsyncClient):
        self.name = 'answer_correctness'

        self.llm = llm_factory(
            model='gemma3',
            provider="litellm",
            base_url="https://api.lite.llm/v1/",
            client=http_client)

        self.embedder = embedding_factory(
            model='nomic-embed-text',
            provider="litellm",
            base_url="https://api.lite.llm/v1/embeddings",
            client=http_client)


        self.scorer = AnswerCorrectness(llm=self.llm, embeddings=self.embedder)

    async def get(self, queries: list[Query]) -> list[Query]:
        """
        Calculates the answer correctness score for the whole dataset.
        """
        for q in range(len(queries)):
            result = await self.scorer.ascore(
                user_input=queries[q].user_input,
                response=queries[q].response,
                reference=queries[q].reference,
            )

            print(f"score: {result}")
            queries[q].set_score(self.name, result.value)

        return queries
```
When I invoke the scoring function:
````python
scorer = RagasAnswerCorrectnessScorer(http_client=client_provider.get_async_client())
scored = asyncio.run(scorer.get(queries))
```

I am ending up with a TypeError:
```python
self = InstructorLLM(provider='litellm', model='gemma3', client=<Instructor:sync>, temperature=0.01, max_tokens=1024, top_p=0.1, +1 more)
prompt = 'Given a question and an answer, analyze the complexity of each sentence in the answer. Break down each sentence into ...'
response_model = <class 'ragas.metrics.collections._answer_correctness.StatementGeneratorOutput'>

    async def agenerate(
        self,
        prompt: str,
        response_model: t.Type[InstructorTypeVar],
    ) -> InstructorTypeVar:
        """Asynchronously generate a response using the configured LLM."""
        messages = [{"role": "user", "content": prompt}]
    
        # If client is not async, raise a helpful error
        if not self.is_async:
>           raise TypeError(
                "Cannot use agenerate() with a synchronous client. Use generate() instead."
            )
E           TypeError: Cannot use agenerate() with a synchronous client. Use generate() instead.

..\.venv\Lib\site-packages\ragas\llms\base.py:731: TypeError
```
I am not master of asynchronous requests... where the issue could come from?

@NickBear-star Thanks for getting back. `metric.ascore` should work with 
`metric = AnswerCorrectness(llm=llm, embeddings=embeddings)`

However, since you're using `litellm`, there might an issue. I'm working on PR #2446 which might fix it. 

Keeping this issue open till resolved.

This should now be resolved. You can use proxy config at client init using litellm adapter and the new metrics collections.

I have pulled the main branch and tested the code. Unfortunately, the errors are still showing up. The only difference is that the class for the response model is now:
```
response_model = <class 'ragas.metrics.collections.answer_correctness.util.StatementGeneratorOutput'>
```

What I don't understand is that the client of the InstructorLLM instance is of type sync:
```
InstructorLLM(provider='litellm', model='gemma3', client=<Instructor:sync>, temperature=0.01, max_tokens=1024, top_p=0.1, +1 more)
```

While the instance has been initialized with a client of class `'httpx.AsyncClient'`.

Yes @NickBear-star 
It seems that even though `litellm.Router` IS async, `instructor.from_litellm()` wraps it in a sync `Instructor` wrapper.

The fix would be to correctly detect async capability even though the wrapper looks sync. Will open a PR for it shortly.


What I meant for the usage with PR #2446 was that now you can use litellm adapter and not have to use instructor at all. something like:

```py
router = Router(config)

llm = llm_factory("model", provider="litellm", client=router) # this is auto-detect adapter as litellm

# or you can mention explicitly the adapter to use.
llm = llm_factory("model", provider="litellm", client=router, adapter="litellm")
```


I a using a simple http client factory:
```python
import httpx
from core.models.config.session import SessionConfig

class ClientProvider:
    """
    Provides a configured http client using HTTPX
    """
    def __init__(self, config: SessionConfig):
        self.token = config.token
        self.use_proxy = config.use_proxy
        self.proxy_url = config.proxy_url
        self.timeout = config.timeout
        self.retry = config.retry

    def get_client(self) -> httpx.Client:
        """
        Creates a sync client using HTTPX
        """

        client_args = {'timeout': self.timeout, 'transport': httpx.HTTPTransport(retries=self.retry)}

        if self.use_proxy and self.proxy_url:
            client = httpx.Client(proxy=self.proxy_url, **client_args)
        else:
            client = httpx.Client(**client_args)

        return client

    def get_async_client(self) -> httpx.AsyncClient:
        """
        Creates an asynchronous client using HTTPX
        """

        client_args = {'timeout': self.timeout, 'transport': httpx.AsyncHTTPTransport(retries=self.retry)}

        if self.use_proxy and self.proxy_url:
            mounts = {
                'http://': httpx.AsyncHTTPTransport(proxy=self.proxy_url, retries=self.retry),
                'https://': httpx.AsyncHTTPTransport(proxy=self.proxy_url, retries=self.retry),
            }
            client = httpx.AsyncClient(mounts=mounts, **client_args)
        else:
            client = httpx.AsyncClient(**client_args)

        return client
```

From that factory, I create an async http client with the `get_async_client()` function, then, I instantiate the LLM and the embeddings:
```python
http_client = client_factory.get_async_client()

llm = llm_factory(
            model='gemma3',
            provider="litellm",
            base_api="https://api.lite.llm/v1/",
            client=http_client,
            adapter='litellm',
        )

embedder = embedding_factory(
            model='nomic-embed-text',
            provider="litellm",
            base_url="https://api.lite.llm/v1/embeddings",
            adapter='litellm',
            client=http_client)
```
However, I keep falling on the same `ValueError`. Indeed, the LLM factory doesn't create an async instance:
```
print(type(http_client))
print(llm)
print(embedder)
>>>
<class 'httpx.AsyncClient'>
LiteLLMStructuredLLM(model='gemma3', provider='litellm', is_async=False)
LiteLLMEmbeddings(provider='litellm', model='nomic-embed-text', +2 litellm_params)
```
Where is my misunderstanding? Do I need to use another client? Is there a way to work in full sync instead?

@NickBear-star `httpx.AsyncClient` is a generic HTTP client, not an LLM client. It won't work. Also, embeddings doesn't take client.

Use **LiteLLM Router** as your client instead.


Sample code:


```py
from litellm import Router
from ragas.llms import llm_factory
from ragas.embeddings import embedding_factory
from ragas.metrics.collections import AnswerCorrectness

class RagasAnswerCorrectnessScorer:
    def __init__(self, proxy_url: str = None):
        # Create LiteLLM Router (llm client)
        router_config = {
            "model_name": "gemma3",
            "litellm_params": {
                "model": "gemini/gemma-3-9b",
                "base_url": "https://api.lite.llm/v1/",
                "api_key": "your-api-key"
            }
        }

        # Add proxy if needed
        if proxy_url:
            router_config["litellm_params"]["proxies"] = proxy_url

        self.router = Router(model_list=[router_config])

        # Pass the router (not httpx.AsyncClient) to llm_factory
        self.llm = llm_factory(
            model='gemma3',
            provider="litellm",
            client=self.router,
            adapter='litellm'
        )
        # Result: is_async=True

        # Embeddings
        self.embedder = embedding_factory(
            provider="litellm",
            model='nomic-embed-text',
            api_base="https://api.lite.llm/v1/embeddings"
        )

        self.scorer = AnswerCorrectness(llm=self.llm, embeddings=self.embedder)

    async def get(self, queries):
        for q in queries:
            result = await self.scorer.ascore(
                user_input=q.user_input,
                response=q.response,
                reference=q.reference,
            )
            q.set_score(self.name, result.value)
        return queries
```



Yes it works. Thanks a lot!