Repository: Marker-Inc-Korea/AutoRAG
Issue: [HotFix] Can't embedding for vectordb more than 2048 corpus size.
Root cause: Validation Gap

It occurs openai error when there are more than 2048 corpus.
We have to fix this problem because many corpus have more than 2048 size.

```
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/autorag/evaluator.py:120 in __embed                                                │               
                             │                                                                                                                                                                      │               
                             │   117 │   │   │   │   else:                                                                                                                                          │               
                             │   118 │   │   │   │   │   logger.error(f"embedding_model_str {embedding_model_str} does not                                                                          │               
                             │       exist.")                                                                                                                                                       │               
                             │   119 │   │   │   │   │   raise KeyError(f"embedding_model_str {embedding_model_str} does not                                                                        │               
                             │       exist.")                                                                                                                                                       │               
                             │ ❱ 120 │   │   │   │   vectordb_ingest(collection, self.corpus_data, embedding_model)                                                                                 │               
                             │   121 │   │   │   │   logger.info(f'VectorDB corpus embedding complete with                                                                                          │               
                             │       {embedding_model_str}.')                                                                                                                                       │               
                             │   122 │   │   else:                                                                                                                                                  │               
                             │   123 │   │   │   logger.info('No ingestion needed.')                                                                                                                │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/autorag/nodes/retrieval/vectordb.py:86 in vectordb_ingest                          │               
                             │                                                                                                                                                                      │               
                             │   83 │   contents = corpus_data['contents'].tolist()                                                                                                                 │               
                             │   84 │                                                                                                                                                               │               
                             │   85 │   # embed corpus                                                                                                                                              │               
                             │ ❱ 86 │   embedded_contents = embedding_model._get_text_embeddings(contents)                                                                                          │               
                             │   87 │   collection.add(ids=ids, embeddings=embedded_contents)                                                                                                       │               
                             │   88                                                                                                                                                                 │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/llama_index/embeddings/openai.py:413 in _get_text_embeddings                       │               
                             │                                                                                                                                                                      │               
                             │   410 │   │                                                                                                                                                          │               
                             │   411 │   │   """                                                                                                                                                    │               
                             │   412 │   │   client = self._get_client()                                                                                                                            │               
                             │ ❱ 413 │   │   return get_embeddings(                                                                                                                                 │               
                             │   414 │   │   │   client,                                                                                                                                            │               
                             │   415 │   │   │   texts,                                                                                                                                             │               
                             │   416 │   │   │   engine=self._text_engine,                                                                                                                          │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/tenacity/__init__.py:289 in wrapped_f                                              │               
                             │                                                                                                                                                                      │               
                             │   286 │   │                                                                                                                                                          │               
                             │   287 │   │   @functools.wraps(f)                                                                                                                                    │               
                             │   288 │   │   def wrapped_f(*args: t.Any, **kw: t.Any) -> t.Any:                                                                                                     │               
                             │ ❱ 289 │   │   │   return self(f, *args, **kw)                                                                                                                        │               
                             │   290 │   │                                                                                                                                                          │               
                             │   291 │   │   def retry_with(*args: t.Any, **kwargs: t.Any) -> WrappedFn:                                                                                            │               
                             │   292 │   │   │   return self.copy(*args, **kwargs).wraps(f)                                                                                                         │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/tenacity/__init__.py:379 in __call__                                               │               
                             │                                                                                                                                                                      │               
                             │   376 │   │                                                                                                                                                          │               
                             │   377 │   │   retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)                                                                       │               
                             │   378 │   │   while True:                                                                                                                                            │               
                             │ ❱ 379 │   │   │   do = self.iter(retry_state=retry_state)                                                                                                            │               
                             │   380 │   │   │   if isinstance(do, DoAttempt):                                                                                                                      │               
                             │   381 │   │   │   │   try:                                                                                                                                           │               
                             │   382 │   │   │   │   │   result = fn(*args, **kwargs)                                                                                                               │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/tenacity/__init__.py:314 in iter                                                   │               
                             │                                                                                                                                                                      │               
                             │   311 │   │                                                                                                                                                          │               
                             │   312 │   │   is_explicit_retry = fut.failed and isinstance(fut.exception(), TryAgain)                                                                               │               
                             │   313 │   │   if not (is_explicit_retry or self.retry(retry_state)):                                                                                                 │               
                             │ ❱ 314 │   │   │   return fut.result()                                                                                                                                │               
                             │   315 │   │                                                                                                                                                          │               
                             │   316 │   │   if self.after is not None:                                                                                                                             │               
                             │   317 │   │   │   self.after(retry_state)                                                                                                                            │               
                             │                                                                                                                                                                      │               
                             │ /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py:438 in result                        │               
                             │                                                                                                                                                                      │               
                             │   435 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                                                                                         │               
                             │   436 │   │   │   │   │   raise CancelledError()                                                                                                                     │               
                             │   437 │   │   │   │   elif self._state == FINISHED:                                                                                                                  │               
                             │ ❱ 438 │   │   │   │   │   return self.__get_result()                                                                                                                 │               
                             │   439 │   │   │   │                                                                                                                                                  │               
                             │   440 │   │   │   │   self._condition.wait(timeout)                                                                                                                  │               
                             │   441                                                                                                                                                                │               
                             │                                                                                                                                                                      │               
                             │ /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py:390 in __get_result                  │               
                             │                                                                                                                                                                      │               
                             │   387 │   def __get_result(self):                                                                                                                                    │               
                             │   388 │   │   if self._exception:                                                                                                                                    │               
                             │   389 │   │   │   try:                                                                                                                                               │               
                             │ ❱ 390 │   │   │   │   raise self._exception                                                                                                                          │               
                             │   391 │   │   │   finally:                                                                                                                                           │               
                             │   392 │   │   │   │   # Break a reference cycle with the exception in self._exception                                                                                │               
                             │   393 │   │   │   │   self = None                                                                                                                                    │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/tenacity/__init__.py:382 in __call__                                               │               
                             │                                                                                                                                                                      │               
                             │   379 │   │   │   do = self.iter(retry_state=retry_state)                                                                                                            │               
                             │   380 │   │   │   if isinstance(do, DoAttempt):                                                                                                                      │               
                             │   381 │   │   │   │   try:                                                                                                                                           │               
                             │ ❱ 382 │   │   │   │   │   result = fn(*args, **kwargs)                                                                                                               │               
                             │   383 │   │   │   │   except BaseException:  # noqa: B902                                                                                                            │               
                             │   384 │   │   │   │   │   retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]                                                                        │               
                             │   385 │   │   │   │   else:                                                                                                                                          │               
                             │                                                                                                                                                                      │               
                             │ /Users/jeffrey/PycharmProjects/AutoRAG-Benchmark/venv/lib/python3.9/site-packages/llama_index/embeddings/openai.py:174 in get_embeddings                             │               
                             │                                                                                                                                                                      │               
                             │   171 │   like matplotlib, plotly, scipy, sklearn.                                                                                                                   │               
                             │   172 │                                                                                                                                                              │               
                             │   173 │   """                                                                                                                                                        │               
                             │ ❱ 174 │   assert len(list_of_text) <= 2048, "The batch size should not be larger than 2048."                                                                         │               
                             │   175 │                                                                                                                                                              │               
                             │   176 │   list_of_text = [text.replace("\n", " ") for text in list_of_text]                                                                                          │               
                             │   177                                                                                                                                                                │               
                             ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯               
                             AssertionError: The batch size should not be larger than 2048. 
```

I think you must set batch to little low, because of the rate limit.
How about 512..?

Plus, this is related to #113

In my experiment, Llama_index `get_text_embedding` method put corpus once at Huggingface Embedding.
So, it occurs CUDA OOM error when corpus size is big, but when you set batch like 128, the model is so fast and embedding is done really quick.

I think we sholud get option about batch for embedding. I wonder where. `Evaluator` init parameter? Or config yaml file?