Repository: truera/trulens
Issue: Any implementation to stay under OpenAI rate limit for feedback functions?
Root cause: Documentation Deficiency

I'm running a RAG pipeline with `LlamaIndex` and evaluate it using 4 evaluations from `trulens` as below. The query engine from LlamaIndex always work independently, and I also tried to implement some exponential backoff to delay sending the request. However, it always fails after sending one or two metrics to OpenAI

```python
import time
from tenacity import retry, stop_after_attempt, wait_random_exponential
from trulens_eval import FeedbackMode


tru_recorder = TruLlama(
    query_engine,
    app_id="Test",
    feedbacks=[
        f_groundtruth,
        f_qa_relevance,
        f_qs_relevance,
        f_groundedness,
    ],
)


# Avoid rate limit issues through exponential backoff
@retry(
    stop=stop_after_attempt(3),
    wait=wait_random_exponential(multiplier=2, min=30, max=120),
)
def ask(question: str):
    with tru_recorder as recording:
        query_engine.query(question)


for q in honest_questions:
    ask(q)
```

The error is something like:
```
Groundendess per statement in source:   0%|          | 0/14 [00:00<?, ?it/s]openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-pH2v2pPIHO0Gx1qW6X6RSpoO on requests per min (RPM): Limit 3, Used 3, Requested 1. Please try again in 20s. Visit https://platform.openai.com/account/rate-limits to learn more. You can increase your rate limit by adding a payment method to your account at https://platform.openai.com/account/billing.', 'type': 'requests', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.

[EXCLUDED BUT RETRIES]
Error calling wrapped function get_response.

[EXCLUDED BUT MORE RETRIES]

RateLimitError                            Traceback (most recent call last)

with tru_recorder as recording:
     query_engine.query(question)
...
     raise retry_exc from fut.exception()
     if self.wait:
     sleep = self.wait(retry_state)
     
RetryError: RetryError[<Future at 0x33159b690 state=finished raised RateLimitError>]
```

Hi; you can reuse the pacing scheme we use for the feedback functions. Here is an example:
```python
from trulens_eval import OpenAI
o = OpenAI()
o.endpoint.rpm = 3
while True:
    print(o.endpoint.pace_me())
    # Whatever follows will run at rpm of 3 max.
```

Thanks for quick response. I implemented your suggestion but only 2/15 questions in my test set went through the RAG triad of metrics in the best attempt.

I don't know if it's because I used the API when many other people were using it too.

Hey @chiffonng !

Can you try adding feedback_mode="with_app" to your app?

For example:

```python
tru_recorder = TruLlama(
    query_engine,
    app_id="Test",
    feedbacks=[
        f_groundtruth,
        f_qa_relevance,
        f_qs_relevance,
        f_groundedness,
    ],
   feedback_mode="with_app"
)
```


@chiffonng please re-open if you are still experiencing this issue. Closing for now.

Is there a way to implement the token limit instead of request limit? As the question and answer can be long or short, the token based could be better.


@zhongshuai-cao 
I am facing the same token limit issues. Did you come up with any solution?