Repository: explodinggradients/ragas
Issue: When I use my own LLM model to evaluate answer_relevancy, the following error occurs.
Root cause: Configuration Error

I used the method provided on the official website to call my own LLM model, and I installed the latest version of ragas. When I evaluate answer_relevancy, the following error occurs.

Ragas version: latest version
Python version: Python 3.10.13

**Code to Reproduce**

from langchain.chat_models import ChatOpenAI
from ragas.llms.base import LangchainLLMWrapper

inference_server_url = "http://localhost:8000/v1"

#create vLLM Langchain instance
chat = ChatOpenAI(
    model= '/weights/huggingface/chatglm3-6b',
    openai_api_key="1111",
    openai_api_base=inference_server_url,
    max_tokens=1500,
    temperature=0,
)

vllm = LangchainLLMWrapper(chat)


from ragas.metrics import (
    answer_relevancy,
    answer_similarity,
)


answer_relevancy.llm = vllm
answer_similarity.llm = vllm

from datasets import Dataset

DATAS_JSON = '../data/data_short.json'
datasets = Dataset.from_json(DATAS_JSON)

#evaluate
from ragas import evaluate

result = evaluate(
    datasets, 
    metrics=[
            answer_relevancy, 
            answer_similarity],
)

print(result)


**Error trace**

Traceback (most recent call last):
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 887, in _request
    response = self._client.send(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 1016, in _send_single_request
    response = transport.handle_request(request)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 230, in handle_request
    with map_httpcore_exceptions():
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 887, in _request
    response = self._client.send(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 1016, in _send_single_request
    response = transport.handle_request(request)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 230, in handle_request
    with map_httpcore_exceptions():
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/cxzhou/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 887, in _request
    response = self._client.send(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_client.py", line 1016, in _send_single_request
    response = transport.handle_request(request)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 230, in handle_request
    with map_httpcore_exceptions():
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/evaluation.py", line 176, in evaluate
    raise e
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/evaluation.py", line 159, in evaluate
    results = executor.results()
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/executor.py", line 118, in results
    raise e
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/executor.py", line 114, in results
    r = future.result()
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/executor.py", line 36, in wrapped_callable
    return counter, callable(*args, **kwargs)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/metrics/base.py", line 75, in score
    raise e
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/metrics/base.py", line 71, in score
    score = self._score(row=row, callbacks=group_cm)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/metrics/_answer_similarity.py", line 70, in _score
    embeddings_1 = np.array(self.embeddings.embed_documents(ground_truths))
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/langchain_community/embeddings/openai.py", line 668, in embed_documents
    return self._get_len_safe_embeddings(texts, engine=engine)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/langchain_community/embeddings/openai.py", line 494, in _get_len_safe_embeddings
    response = embed_with_retry(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/langchain_community/embeddings/openai.py", line 116, in embed_with_retry
    return embeddings.client.create(**kwargs)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/resources/embeddings.py", line 103, in create
    return self._post(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 1112, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 859, in request
    return self._request(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 911, in _request
    return self._retry_request(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 982, in _retry_request
    return self._request(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 911, in _request
    return self._retry_request(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 982, in _retry_request
    return self._request(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/openai/_base_client.py", line 921, in _request
    raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cxzhou/code/wendy_now/ragas/scripts/latest_ownmodel.py", line 60, in <module>
    result = evaluate(
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/evaluation.py", line 178, in evaluate
    result = Result(
  File "<string>", line 6, in __init__
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/ragas/evaluation.py", line 207, in __post_init__
    for cn in self.scores[0].keys():
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2800, in __getitem__
    return self._getitem(key)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2784, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/home/cxzhou/.conda/envs/RAGAS_latest/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 0 is out of bounds for size 0

**Expected behavior**
I want answer_relevancy to be evaluated normally. 


**Additional context**
When I evaluate other metrics such as: faithfulness. There is no problem, but there are errors in answer_relevancy and answer_similarity.