Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] How to control a large number of requests
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
It seems that an error occurs when there are too many requests. Would it be possible to have a parameter that allows us to control this?

**logs**
```
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/llama_index/llms/openai/base.py:627 in achat                │               
                             │                                                                                                                                                           │               
                             │   624 │   │   │   achat_fn = self._achat                                                                                                                  │               
                             │   625 │   │   else:                                                                                                                                       │               
                             │   626 │   │   │   achat_fn = acompletion_to_chat_decorator(self._acomplete)                                                                               │               
                             │ ❱ 627 │   │   return await achat_fn(messages, **kwargs)                                                                                                   │               
                             │   628 │                                                                                                                                                   │               
                             │   629 │   @llm_chat_callback()                                                                                                                            │               
                             │   630 │   async def astream_chat(                                                                                                                         │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:189 in async_wrapped           │               
                             │                                                                                                                                                           │               
                             │   186 │   │   │   # calling the same wrapped functions multiple times in the same stack                                                                   │               
                             │   187 │   │   │   copy = self.copy()                                                                                                                      │               
                             │   188 │   │   │   async_wrapped.statistics = copy.statistics  # type: ignore[attr-defined]                                                                │               
                             │ ❱ 189 │   │   │   return await copy(fn, *args, **kwargs)                                                                                                  │               
                             │   190 │   │                                                                                                                                               │               
                             │   191 │   │   # Preserve attributes                                                                                                                       │               
                             │   192 │   │   async_wrapped.retry = self  # type: ignore[attr-defined]                                                                                    │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:111 in __call__                │               
                             │                                                                                                                                                           │               
                             │   108 │   │                                                                                                                                               │               
                             │   109 │   │   retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)                                                            │               
                             │   110 │   │   while True:                                                                                                                                 │               
                             │ ❱ 111 │   │   │   do = await self.iter(retry_state=retry_state)                                                                                           │               
                             │   112 │   │   │   if isinstance(do, DoAttempt):                                                                                                           │               
                             │   113 │   │   │   │   try:                                                                                                                                │               
                             │   114 │   │   │   │   │   result = await fn(*args, **kwargs)                                                                                              │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:153 in iter                    │               
                             │                                                                                                                                                           │               
                             │   150 │   │   self._begin_iter(retry_state)                                                                                                               │               
                             │   151 │   │   result = None                                                                                                                               │               
                             │   152 │   │   for action in self.iter_state.actions:                                                                                                      │               
                             │ ❱ 153 │   │   │   result = await action(retry_state)                                                                                                      │               
                             │   154 │   │   return result                                                                                                                               │               
                             │   155 │                                                                                                                                                   │               
                             │   156 │   def __iter__(self) -> t.Generator[AttemptManager, None, None]:                                                                                  │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/_utils.py:99 in inner                              │               
                             │                                                                                                                                                           │               
                             │    96 │   │   return call                                                                                                                                 │               
                             │    97 │                                                                                                                                                   │               
                             │    98 │   async def inner(*args: typing.Any, **kwargs: typing.Any) -> typing.Any:                                                                         │               
                             │ ❱  99 │   │   return call(*args, **kwargs)                                                                                                                │               
                             │   100 │                                                                                                                                                   │               
                             │   101 │   return inner                                                                                                                                    │               
                             │   102                                                                                                                                                     │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/__init__.py:418 in exc_check                       │               
                             │                                                                                                                                                           │               
                             │   415 │   │   │   │   fut = t.cast(Future, rs.outcome)                                                                                                    │               
                             │   416 │   │   │   │   retry_exc = self.retry_error_cls(fut)                                                                                               │               
                             │   417 │   │   │   │   if self.reraise:                                                                                                                    │               
                             │ ❱ 418 │   │   │   │   │   raise retry_exc.reraise()                                                                                                       │               
                             │   419 │   │   │   │   raise retry_exc from fut.exception()                                                                                                │               
                             │   420 │   │   │                                                                                                                                           │               
                             │   421 │   │   │   self._add_action_func(exc_check)                                                                                                        │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/__init__.py:185 in reraise                         │               
                             │                                                                                                                                                           │               
                             │   182 │                                                                                                                                                   │               
                             │   183 │   def reraise(self) -> t.NoReturn:                                                                                                                │               
                             │   184 │   │   if self.last_attempt.failed:                                                                                                                │               
                             │ ❱ 185 │   │   │   raise self.last_attempt.result()                                                                                                        │               
                             │   186 │   │   raise self                                                                                                                                  │               
                             │   187 │                                                                                                                                                   │               
                             │   188 │   def __str__(self) -> str:                                                                                                                       │               
                             │                                                                                                                                                           │               
                             │ /root/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py:451 in result                                                                    │               
                             │                                                                                                                                                           │               
                             │   448 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                                                                              │               
                             │   449 │   │   │   │   │   raise CancelledError()                                                                                                          │               
                             │   450 │   │   │   │   elif self._state == FINISHED:                                                                                                       │               
                             │ ❱ 451 │   │   │   │   │   return self.__get_result()                                                                                                      │               
                             │   452 │   │   │   │                                                                                                                                       │               
                             │   453 │   │   │   │   self._condition.wait(timeout)                                                                                                       │               
                             │   454                                                                                                                                                     │               
                             │                                                                                                                                                           │               
                             │ /root/.pyenv/versions/3.10.13/lib/python3.10/concurrent/futures/_base.py:403 in __get_result                                                              │               
                             │                                                                                                                                                           │               
                             │   400 │   def __get_result(self):                                                                                                                         │               
                             │   401 │   │   if self._exception:                                                                                                                         │               
                             │   402 │   │   │   try:                                                                                                                                    │               
                             │ ❱ 403 │   │   │   │   raise self._exception                                                                                                               │               
                             │   404 │   │   │   finally:                                                                                                                                │               
                             │   405 │   │   │   │   # Break a reference cycle with the exception in self._exception                                                                     │               
                             │   406 │   │   │   │   self = None                                                                                                                         │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:114 in __call__                │               
                             │                                                                                                                                                           │               
                             │   111 │   │   │   do = await self.iter(retry_state=retry_state)                                                                                           │               
                             │   112 │   │   │   if isinstance(do, DoAttempt):                                                                                                           │               
                             │   113 │   │   │   │   try:                                                                                                                                │               
                             │ ❱ 114 │   │   │   │   │   result = await fn(*args, **kwargs)                                                                                              │               
                             │   115 │   │   │   │   except BaseException:  # noqa: B902                                                                                                 │               
                             │   116 │   │   │   │   │   retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]                                                             │               
                             │   117 │   │   │   │   else:                                                                                                                               │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/llama_index/llms/openai/base.py:674 in _achat               │               
                             │                                                                                                                                                           │               
                             │   671 │   │   message_dicts = to_openai_message_dicts(messages, model=self.model)                                                                         │               
                             │   672 │   │                                                                                                                                               │               
                             │   673 │   │   if self.reuse_client:                                                                                                                       │               
                             │ ❱ 674 │   │   │   response = await aclient.chat.completions.create(                                                                                       │               
                             │   675 │   │   │   │   messages=message_dicts, stream=False, **self._get_model_kwargs(**kwargs)                                                            │               
                             │   676 │   │   │   )                                                                                                                                       │               
                             │   677 │   │   else:                                                                                                                                       │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/resources/chat/completions.py:1661 in create         │               
                             │                                                                                                                                                           │               
                             │   1658 │   │   timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,                                                                              │               
                             │   1659 │   ) -> ChatCompletion | AsyncStream[ChatCompletionChunk]:                                                                                        │               
                             │   1660 │   │   validate_response_format(response_format)                                                                                                  │               
                             │ ❱ 1661 │   │   return await self._post(                                                                                                                   │               
                             │   1662 │   │   │   "/chat/completions",                                                                                                                   │               
                             │   1663 │   │   │   body=await async_maybe_transform(                                                                                                      │               
                             │   1664 │   │   │   │   {                                                                                                                                  │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1840 in post                         │               
                             │                                                                                                                                                           │               
                             │   1837 │   │   opts = FinalRequestOptions.construct(                                                                                                      │               
                             │   1838 │   │   │   method="post", url=path, json_data=body, files=await                                                                                   │               
                             │        async_to_httpx_files(files), **options                                                                                                             │               
                             │   1839 │   │   )                                                                                                                                          │               
                             │ ❱ 1840 │   │   return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)                                                             │               
                             │   1841 │                                                                                                                                                  │               
                             │   1842 │   async def patch(                                                                                                                               │               
                             │   1843 │   │   self,                                                                                                                                      │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1534 in request                      │               
                             │                                                                                                                                                           │               
                             │   1531 │   │   else:                                                                                                                                      │               
                             │   1532 │   │   │   retries_taken = 0                                                                                                                      │               
                             │   1533 │   │                                                                                                                                              │               
                             │ ❱ 1534 │   │   return await self._request(                                                                                                                │               
                             │   1535 │   │   │   cast_to=cast_to,                                                                                                                       │               
                             │   1536 │   │   │   options=options,                                                                                                                       │               
                             │   1537 │   │   │   stream=stream,                                                                                                                         │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1620 in _request                     │               
                             │                                                                                                                                                           │               
                             │   1617 │   │   │                                                                                                                                          │               
                             │   1618 │   │   │   if remaining_retries > 0 and self._should_retry(err.response):                                                                         │               
                             │   1619 │   │   │   │   await err.response.aclose()                                                                                                        │               
                             │ ❱ 1620 │   │   │   │   return await self._retry_request(                                                                                                  │               
                             │   1621 │   │   │   │   │   input_options,                                                                                                                 │               
                             │   1622 │   │   │   │   │   cast_to,                                                                                                                       │               
                             │   1623 │   │   │   │   │   retries_taken=retries_taken,                                                                                                   │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1667 in _retry_request               │               
                             │                                                                                                                                                           │               
                             │   1664 │   │                                                                                                                                              │               
                             │   1665 │   │   await anyio.sleep(timeout)                                                                                                                 │               
                             │   1666 │   │                                                                                                                                              │               
                             │ ❱ 1667 │   │   return await self._request(                                                                                                                │               
                             │   1668 │   │   │   options=options,                                                                                                                       │               
                             │   1669 │   │   │   cast_to=cast_to,                                                                                                                       │               
                             │   1670 │   │   │   retries_taken=retries_taken + 1,                                                                                                       │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1620 in _request                     │               
                             │                                                                                                                                                           │               
                             │   1617 │   │   │                                                                                                                                          │               
                             │   1618 │   │   │   if remaining_retries > 0 and self._should_retry(err.response):                                                                         │               
                             │   1619 │   │   │   │   await err.response.aclose()                                                                                                        │               
                             │ ❱ 1620 │   │   │   │   return await self._retry_request(                                                                                                  │               
                             │   1621 │   │   │   │   │   input_options,                                                                                                                 │               
                             │   1622 │   │   │   │   │   cast_to,                                                                                                                       │               
                             │   1623 │   │   │   │   │   retries_taken=retries_taken,                                                                                                   │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1667 in _retry_request               │               
                             │                                                                                                                                                           │               
                             │   1664 │   │                                                                                                                                              │               
                             │   1665 │   │   await anyio.sleep(timeout)                                                                                                                 │               
                             │   1666 │   │                                                                                                                                              │               
                             │ ❱ 1667 │   │   return await self._request(                                                                                                                │               
                             │   1668 │   │   │   options=options,                                                                                                                       │               
                             │   1669 │   │   │   cast_to=cast_to,                                                                                                                       │               
                             │   1670 │   │   │   retries_taken=retries_taken + 1,                                                                                                       │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1620 in _request                     │               
                             │                                                                                                                                                           │               
                             │   1617 │   │   │                                                                                                                                          │               
                             │   1618 │   │   │   if remaining_retries > 0 and self._should_retry(err.response):                                                                         │               
                             │   1619 │   │   │   │   await err.response.aclose()                                                                                                        │               
                             │ ❱ 1620 │   │   │   │   return await self._retry_request(                                                                                                  │               
                             │   1621 │   │   │   │   │   input_options,                                                                                                                 │               
                             │   1622 │   │   │   │   │   cast_to,                                                                                                                       │               
                             │   1623 │   │   │   │   │   retries_taken=retries_taken,                                                                                                   │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1667 in _retry_request               │               
                             │                                                                                                                                                           │               
                             │   1664 │   │                                                                                                                                              │               
                             │   1665 │   │   await anyio.sleep(timeout)                                                                                                                 │               
                             │   1666 │   │                                                                                                                                              │               
                             │ ❱ 1667 │   │   return await self._request(                                                                                                                │               
                             │   1668 │   │   │   options=options,                                                                                                                       │               
                             │   1669 │   │   │   cast_to=cast_to,                                                                                                                       │               
                             │   1670 │   │   │   retries_taken=retries_taken + 1,                                                                                                       │               
                             │                                                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py:1635 in _request                     │               
                             │                                                                                                                                                           │               
                             │   1632 │   │   │   │   await err.response.aread()                                                                                                         │               
                             │   1633 │   │   │                                                                                                                                          │               
                             │   1634 │   │   │   log.debug("Re-raising status error")                                                                                                   │               
                             │ ❱ 1635 │   │   │   raise self._make_status_error_from_response(err.response) from None                                                                    │               
                             │   1636 │   │                                                                                                                                              │               
                             │   1637 │   │   return await self._process_response(                                                                                                       │               
                             │   1638 │   │   │   cast_to=cast_to,                                                                                                                       │               
                             ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯               
                             RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o-mini in organization org-Y9VgTSCB4k6OcvPF42h5yAYO on tokens per min                  
                             (TPM): Limit 200000, Used 199980, Requested 2213. Please try again in 657ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type':                 
                             'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}  
```

@e7217 

You can decrease the batch_size at the YAML file! You can find 'batch_size' setting at the almost every modules that are using API calls. 
It will be slower, but it will prevent rate limit error.

> [@e7217](https://github.com/e7217)
> 
> You can decrease the batch_size at the YAML file! You can find 'batch_size' setting at the almost every modules that are using API calls. It will be slower, but it will prevent rate limit error.

@vkehfdl1 Oh! I completely forgot about that! Thank you!