Repository: explodinggradients/ragas
Issue: llama-cppp-python server not working in "Bring your own LLM"
Root cause: Validation Gap

**Your Question**
llama-cpp python server https://github.com/abetlen/llama-cpp-python/pkgs/container/llama-cpp-python offers what they call an OpenAI capable service. I set it up and tried to follow the instructions in the "bring your own llm" page https://docs.ragas.io/en/stable/howtos/customisations/llms.html but it not working

**Code Examples**

```
export OPENAI_API_KEY="llama_key"
```

```
# Your code here
from ragas import evaluate
from datasets import Dataset
import os
from openai import OpenAI
import warnings
import openai

openai.api_key='none'

warnings.filterwarnings("ignore")

from langchain_openai.chat_models import ChatOpenAI

inference_server_url = "http://x.x.x.x:8000/v1" # working service


chat = ChatOpenAI(
    model="/models/llama-2-7b.Q4_0.gguf",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=5,
    temperature=0,
)

data_samples = {
    'question': ['Explain EIRP'],
    'answer':['Sure! Here\'s my answer:\n\nEIRP stands for Effective Isotropic Radiated Power. It represents the total power output of a transmitting antenna in all directions, taking into account the antenna gain and any coupling losses between the transmitter and antenna.'],
    'contexts':[['effective isotropic radiated power (EIRP) :  The equivalent power of a transmitted signal in terms of an isotropic (omnidirectional) radiator. The EIRP equals the product of the transmitter power and the antenna gain (reduced by any coupling losses between the transmitter and antenna)','emergency alert system (EAS) :  A U.S. national public warning system. ','extended service area (ESA) :  The area within which members of an extended service set (ESS) can communicate. An ESA is larger than or equal to a basic service area (BSA) and might involve several basic service sets (BSSs) in overlapping, disjointed, or both configurations.','IEEE 802.1X authentication :  Extensible Authentication Protocol (EAP) authentication transported by the IEEE 802.1X protocol.','master session key (MSK) :  Keying material that is derived between the Extensible Authentication Protocol (EAP) peer and exported by the EAP method to the Authentication Server (AS).']],
    'ground_truths':[['effective isotropic radiated power (EIRP) :  equivalent power of a transmitted signal in terms of an isotropic (omnidirectional) radiator. The EIRP equals the product of the transmitter power and the antenna gain (reduced by any coupling losses between the transmitter and antenna']]
}

dataset = Dataset.from_dict(data_samples)

results = evaluate(dataset)

print (results)
```
Error received

```
Traceback (most recent call last):
  File "checkRagas.py", line 45, in <module>
    results = evaluate(dataset)
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/evaluation.py", line 93, in evaluate
    [m.init_model() for m in metrics]
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/evaluation.py", line 93, in <listcomp>
    [m.init_model() for m in metrics]
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/metrics/_answer_relevance.py", line 96, in init_model
    super().init_model()
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/metrics/base.py", line 122, in init_model
    self.llm.validate_api_key()
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/llms/openai.py", line 203, in validate_api_key
    raise OpenAIKeyNotFound
ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable
```

**Additional context**
Wanted to understand if llama-cpp python server can be integrated with ragas package. 


Following comment in https://github.com/explodinggradients/ragas/issues/398#issuecomment-1880477590 installed version 0.1.1 but getting following error now

```
passing column names as 'ground_truths' is deprecated and will be removed in the next version, please use 'ground_truth' instead. Note that `ground_truth` should be of type string and not Sequence[string] like `ground_truths`
Evaluating:   0%|                                                                                      | 0/4 [00:00<?, ?it/s]
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.8/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/opt/homebrew/Cellar/python@3.8/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/executor.py", line 63, in _aresults
    raise e
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/opt/homebrew/Cellar/python@3.8/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/metrics/base.py", line 91, in ascore
    raise e
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/metrics/base.py", line 87, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/metrics/_answer_relevance.py", line 136, in _ascore
    result = await self.llm.generate(
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/llms/base.py", line 110, in generate
    return await loop.run_in_executor(None, generate_text)
  File "/opt/homebrew/Cellar/python@3.8/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homebrew/lib/python3.8/site-packages/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/opt/homebrew/lib/python3.8/site-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/opt/homebrew/lib/python3.8/site-packages/tenacity/__init__.py", line 314, in iter
    return fut.result()
  File "/opt/homebrew/Cellar/python@3.8/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/opt/homebrew/Cellar/python@3.8/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/homebrew/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/llms/base.py", line 139, in generate_text
    return self.langchain_llm.generate_prompt(
  File "/opt/homebrew/lib/python3.8/site-packages/langchain_core/language_models/chat_models.py", line 544, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/opt/homebrew/lib/python3.8/site-packages/langchain_core/language_models/chat_models.py", line 408, in generate
    raise e
  File "/opt/homebrew/lib/python3.8/site-packages/langchain_core/language_models/chat_models.py", line 398, in generate
    self._generate_with_cache(
  File "/opt/homebrew/lib/python3.8/site-packages/langchain_core/language_models/chat_models.py", line 577, in _generate_with_cache
    return self._generate(
  File "/opt/homebrew/lib/python3.8/site-packages/langchain_openai/chat_models/base.py", line 438, in _generate
    response = self.client.create(messages=message_dicts, **params)
  File "/opt/homebrew/lib/python3.8/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
  File "/opt/homebrew/lib/python3.8/site-packages/openai/resources/chat/completions.py", line 663, in create
    return self._post(
  File "/opt/homebrew/lib/python3.8/site-packages/openai/_base_client.py", line 1200, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/opt/homebrew/lib/python3.8/site-packages/openai/_base_client.py", line 889, in request
    return self._request(
  File "/opt/homebrew/lib/python3.8/site-packages/openai/_base_client.py", line 980, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: no-key. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
Traceback (most recent call last):
  File "checkRagas.py", line 40, in <module>
    results = evaluate(dataset)
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/evaluation.py", line 231, in evaluate
    raise e
  File "/opt/homebrew/lib/python3.8/site-packages/ragas/evaluation.py", line 213, in evaluate
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exception=False` incase you want to show only a warning message instead.
```

quick update 
```
chat = ChatOpenAI(
    model="/models/llama-2-7b.Q4_0.gguf",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=5,
    temperature=0,
)

result = evaluate(
    amnesty_qa["eval"].select(range(1)),  # showing only 1 for demonstration
    metrics=[answer_correctness],
    llm=chat,embeddings=hf_embeddings
)
```

has 3 effects:-  1. The api key not found error is removed.   2.  The Runner Exception above intermittently happens and always if I give all four default metrics. 3. Now getting nan as result for all the metrics.


@sujoyrc I see you fixed the passing llm and embedding correctly issue

2. is it the same authentication error that is happening?
3. this is hard without having a tracing to see what the llm is doing - this usually happens when the LLM is not able to follow JSON format