Repository: explodinggradients/ragas
Issue: Issue with "Bring Your Own LLMs"
Root cause: Documentation Deficiency

**Describe the bug**
In your tutorial, you say to use the openai_api_key "no-key", but then in your code, you throw and error if "no-key" is used as the API key.

Ragas version: 0.0.22
Python version: 3.10

**Code to Reproduce**
Follow the "Bring Your Own LLMs" tutorial and try to run it.

**Error trace**
OPENAI_API_KEY="no-key" python eval_rag.py 

beggining evaluation
Traceback (most recent call last):
  File "ragas_test.py", line 63, in <module>
    result = evaluate(
  File "/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py", line 93, in evaluate
    [m.init_model() for m in metrics]
  File "/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py", line 93, in <listcomp>
    [m.init_model() for m in metrics]
  File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 100, in init_model
    raise OpenAIKeyNotFound
ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable
**Expected behavior**
A clear and concise description of what you expected to happen.
It works

**Additional context**
Your tutorial:
![image](https://github.com/explodinggradients/ragas/assets/61555377/be0347f9-df56-4677-9e4f-f38200ce002b)

You check for the api key in your code [here](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/langchain.py#L89)
```
            if self.langchain_llm.openai_api_key == NO_KEY:
                raise OpenAIKeyNotFound
```
Your definition of NO_KEY [here](https://github.com/explodinggradients/ragas/blob/9850f096fd289c68a7a0cc84063f43797ac0c79f/src/ragas/utils.py#L18):
```
NO_KEY = "no-key"
```

So if you follow the tutorial exactly you get a nice OpenAIKeyNotFound exception.

TO CLARIFY:
I can run locally no problem if I set OPENAI_API_KEY to anything but "no-key", so OPENAI_API_KEY="no-key!" works. This is honestly a nit but your tutorial should not give an example password that automatically results in an exception.

                
            


I can use with  liteLLM (an OpenAI compatible API Proxy)

![image](https://github.com/explodinggradients/ragas/assets/46180030/8cb2c8d8-63cf-41f0-8d1c-4b709f810e79)

And use it as an evaluator

![image](https://github.com/explodinggradients/ragas/assets/46180030/6a2a953d-0a53-46d8-ac36-d033a897585c)




@dtthanh1971 Can you try with answer_relevancy? I have been able to work with local LLM for faithfulness, context_precision. answer_relevancy leads to "OpenAI API key not found!". My understanding is answer_relevancy uses embedding model. Is there any document or similar to use open source model?

I got a similar issue with answer_relevancy and I think there is no solution yet. See this issue, maybe some of the steps work for you. Let me know if this is the case
https://github.com/explodinggradients/ragas/issues/345

Just updated the description since I wasn't clear enough. **I can run everything just fine** if I set OPENAI_API_KEY to anything but "no-key", I just think it's misleading that this is the example key they give in the tutorial which results in an exception. This should be updated so others don't waste 30 minutes figuring out why the key is being rejected.

@jjmachan please address this issue.  I can correct the docs if that's needed.

Hey everyone 
Firstly my sincere apologies for the delayed response, we have been hard at work trying to ship an improved v0.1 release for you all and during that time maintenance work sadly took a backstep 😞 - I apologize again for the delay.

coming to the issue - this has been addressed with #390. The main issue was we used to init the LLM model when the `ragas` package was imported. Handling of `no-key` in this manner was a workaround for that - so that API key will only be checked at invocation time.

with #390 we have removed this issue. Now the LLM will only be initialized at `evaluation()` time or if you create an LLM object yourself. Documentation will be updated to reflect it once we release v0.1

Cheers 
Jithin

Has the docs updated for this change?

Yes, the stable version of docs is updated for this change https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html

So what is the conclusion? can we use any litellm supported model instead of OpenAI?

Hi @ogencoglu @dtthanh1971 do you use LiteLLM Proxy ? Can we hop on a **quick** call to learn how we can make litellm better for you 

Link to my [calendar](https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat?month=2024-02) for your convenience 

Hi all, I have a related issue and would appreciate your help with it. I am working on databricks clusters with no internet connection. I am trying to use databricks foundation model "databricks-llama-2-70b-chat" as the judge llm. i thought the integration with langchain is the only requirement for this to work with Ragas. I tried the code below:

```
from langchain_community.chat_models import ChatDatabricks

chat_model = ChatDatabricks(endpoint="databricks-llama-2-70b-chat")
chat_model.invoke("How are you?")

result = evaluate(fiqa_eval["baseline"], metrics=metrics, embeddings=fast_embeddings, llm = chat_model)
```

However, I am getting this exception:

```
WARNING:urllib3.connectionpool:Retrying (JitteredRetry(total=3, connect=5, read=3, redirect=5, status=5)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='oregon.cloud.databricks.com', port=443): Read timed out. (read timeout=120)")': /serving-endpoints/databricks-llama-2-70b-chat/invocations
WARNING:urllib3.connectionpool:Retrying (JitteredRetry(total=3, connect=5, read=3, redirect=5, status=5)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='oregon.cloud.databricks.com', port=443): Read timed out. (read timeout=120)")': /serving-endpoints/databricks-llama-2-70b-chat/invocations
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 63, in _aresults
    raise e
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/metrics/base.py", line 91, in ascore
    raise e
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/metrics/base.py", line 87, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/metrics/_faithfulness.py", line 190, in _ascore
    p = self._create_nli_prompt(row, statements.get("statements", []))
AttributeError: 'list' object has no attribute 'get'
```
I am wondering if I should wrap the databricks model using ChatOpenAI as the vLLM example in the docs: https://docs.ragas.io/en/stable/howtos/customisations/llms.html#evaluating-with-open-source-llms 

Thanks in advance! 



I tried 
```
from ragas.llms import LangchainLLMWrapper
dbx_chat = LangchainLLMWrapper(langchain_llm=chat_model)
```
and passed the dbx_chat to the Evaluate and got the same error.