Repository: explodinggradients/ragas
Issue: ERROR: Open source LLM with TestsetGenerator
Root cause: Architectural Constraint

I am trying testset generator on locally hosted llm (https://huggingface.co/Qwen/Qwen-72B-Chat) and embeddings (https://huggingface.co/BAAI/bge-m3) model as my system does not has internet access.
However, calling TestsetGenerator tries to access openai api and gives the following error even though model/embeddings is locally hosted.

ConnectTimeout: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /gpt-2/encodings/main/vocab.bpe (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fc8a2b43c10>, 'Connection to openaipublic.blob.core.windows.net timed out. (connect timeout=None)'))


model = ChatOpenAI(
    model_name = 'Qwen-72B-Chat',
    base_url = "http://xxx.xx.xx.x:8383/v1/",
    api_key = "EMPTY",
    temperature=0
)

embeddings = HuggingFaceBgeEmbeddings(
    model_name="/workspace/mounted/bge-m3", 
    encode_kwargs={'normalize_embeddings': True}
)


langchain_llm = LangchainLLMWrapper(model)
langchain_embeddings = LangchainEmbeddingsWrapper(embeddings)

###################################################
generator = TestsetGenerator.from_langchain(
    model,
    model,
    embeddings
)
###################################################

generator = TestsetGenerator.from_langchain(
    langchain_llm,
    langchain_llm,
    langchain_embeddings
)



trace:
File /workspace/ragas/src/ragas/testset/generator.py:93, in TestsetGenerator.from_langchain(cls, generator_llm, critic_llm, embeddings, docstore, run_config, chunk_size)
     90 if docstore is None:
     91     from langchain.text_splitter import TokenTextSplitter
---> 93     splitter = TokenTextSplitter(chunk_size=chunk_size, chunk_overlap=0)
     94     docstore = InMemoryDocumentStore(
     95         splitter=splitter,
     96         embeddings=embeddings_model,
     97         extractor=keyphrase_extractor,
     98         run_config=run_config,
     99     )
    100     return cls(
    101         generator_llm=generator_llm_model,
    102         critic_llm=critic_llm_model,
    103         embeddings=embeddings_model,
    104         docstore=docstore,
    105     )

File /usr/local/lib/python3.10/dist-packages/langchain/text_splitter.py:728, in TokenTextSplitter.__init__(self, encoding_name, model_name, allowed_special, disallowed_special, **kwargs)
    726     enc = tiktoken.encoding_for_model(model_name)
    727 else:
--> 728     enc = tiktoken.get_encoding(encoding_name)
    729 self._tokenizer = enc
    730 self._allowed_special = allowed_special

File /usr/local/lib/python3.10/dist-packages/tiktoken/registry.py:73, in get_encoding(encoding_name)
     68     raise ValueError(
     69         f"Unknown encoding {encoding_name}. Plugins found: {_available_plugin_modules()}"
     70     )
     72 constructor = ENCODING_CONSTRUCTORS[encoding_name]
---> 73 enc = Encoding(**constructor())
     74 ENCODINGS[encoding_name] = enc
     75 return enc

File /usr/local/lib/python3.10/dist-packages/tiktoken_ext/openai_public.py:11, in gpt2()
     10 def gpt2():
---> 11     mergeable_ranks = data_gym_to_mergeable_bpe_ranks(
     12         vocab_bpe_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/vocab.bpe",
     13         encoder_json_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/encoder.json",
     14     )
     15     return {
     16         "name": "gpt2",
     17         "explicit_n_vocab": 50257,
   (...)
     20         "special_tokens": {ENDOFTEXT: 50256},
     21     }


@Haxeebraja Issue seems to be linked to `from_langchain` function 

https://github.com/explodinggradients/ragas/blob/708ddab307ce71585a4092fe67f74c6b19859cea/src/ragas/testset/generator.py#L90

When you call the function without passing an existing `docstore`, he try to use langchain's `TokenTextSplitter` function. This one rely on gpt-2 by default 

A solution, for our case, could be the following: 

- Create your own docstore as follow 

```
from ragas.testset.generator import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings.base import LangchainEmbeddingsWrapper
from ragas.testset.docstore import InMemoryDocumentStore
from ragas.testset.extractor import KeyphraseExtractor

from langchain_openai import ChatOpenAI
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings

generator_llm = ChatOpenAI()
critic_llm = ChatOpenAI()

splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=0)
embeddings = HuggingFaceEmbeddings()
keyphrase_extractor = KeyphraseExtractor(llm=LangchainLLMWrapper(generator_llm))

docstore = InMemoryDocumentStore(
splitter=splitter, 
embeddings=LangchainEmbeddingsWrapper(embeddings), 
extractor=keyphrase_extractor)
```

Where you can use any Langchain's text splitter

- Then pass your new docstore as an argument

```
generator = TestsetGenerator.from_langchain(generator_llm=generator_llm, 
critic_llm=critic_llm, 
embeddings=embeddings, 
docstore=docstore
```