Repository: truera/trulens
Issue: [BUG] Pace is not thread safe
Root cause: Resource Mishandling

**Bug Description**
When I query the "anthropic.claude-3-5-sonnet-20240620-v1:0" model on Bedrock (it also happens with ("anthropic.claude-3-haiku-20240307-v1:0", but here it's a smaller issue), a `botocore.errorfactory.ThrottlingException` is raised multiple times before the program finally exits due to the maximum number of retries being exceeded.

This happens for the `groundedness_measure_with_cot_reasons_consider_answerability`  metric, but it might also concern other metrics that run requests concurrently (such as `groundedness_measure_with_cot_reasons`) to a certain extent.

The issue occurs after the LLM generated a long list of bullet points with non-trivial statements, which leads to a high number of concurrent requests in following part of the code:

```python
results = []
with ThreadPoolExecutor() as executor:
    futures = [
        executor.submit(evaluate_hypothesis, i, hypothesis)
        for i, hypothesis in enumerate(hypotheses)
    ]
```

It seems these threads do not share the Pace (or Pace is not thread safe), as I am getting a throttling exception no matter how low I set `pace.marks_per_second` or `pace.seconds_per_period`.

For now, I implemented a local work-around, which solved the issue for me:

```python
results = [
    evaluate_hypothesis(i, hypothesis) for i, hypothesis in enumerate(hypotheses)
]
```

Limiting the number of workers in the `ThreadPoolExecutor()` might also help to a certain extent.

However, a proper solution should target `pace_utils.Pace` or related methods.

**To Reproduce**
```python
import boto3
from trulens.providers.bedrock import Bedrock

client = boto3.client(service_name="bedrock-runtime")
bedrock = Bedrock(
    model_id="anthropic.claude-3-5-sonnet-20240620-v1:0",
    client=client,
)
bedrock.endpoint.pace.marks_per_second = 0.01

question = "How do advancements in artificial intelligence influence the future of autonomous vehicles?"
source = """
* The quick brown fox
Jumps over the lazy dog
She sells seashells
By the seashore
How much wood
Would a woodchuck chuck
* Peter Piper picked
A peck of pickled peppers
A journey of a thousand miles
Begins with a single step
* To be or not to be
That is the question
All that glitters
Is not gold
"""
statement = """
In a galaxy far, far away, the following items were discovered:

1. The quick brown fox jumps over the lazy dog.
2. She sells seashells by the seashore.
3. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
4. Peter Piper picked a peck of pickled peppers.
5. A journey of a thousand miles begins with a single step.
6. To be or not to be, that is the question.
7. All that glitters is not gold.
8. The early bird catches the worm.
9. A picture is worth a thousand words.
10. Actions speak louder than words.
11. Beauty is in the eye of the beholder.
12. Better late than never.
13. Birds of a feather flock together.
14. Cleanliness is next to godliness.
15. Don't count your chickens before they hatch.
16. Every cloud has a silver lining.
17. Fortune favors the bold.
18. Honesty is the best policy.
19. If it ain't broke, don't fix it.
20. Laughter is the best medicine.
21. Necessity is the mother of invention.
22. Practice makes perfect.
23. The pen is mightier than the sword.
24. There's no place like home.
25. Time is money.
26. When in Rome, do as the Romans do.
27. You can't judge a book by its cover.
28. A watched pot never boils.
29. Absence makes the heart grow fonder.
30. Actions speak louder than words.
"""

bedrock.groundedness_measure_with_cot_reasons_consider_answerability(
    source=source, statement=statement, question=question
)

```

**Expected behavior**
The `Pace` object and related methods should abide by the settings (such as `marks_per_second`) in a multi-thread scenario.

**Relevant Logs/Tracebacks**
```
BedrockEndpoint request failed <class 'botocore.errorfactory.ThrottlingException'>=An error occurred (ThrottlingException) when calling the InvokeModel operation (reached max retries: x): Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.. Retries remaining=x.
```

followed by, at some point:

```
Traceback (most recent call last):
  File "/home/user/projects/my-project/scripts/my-script.py", line 78, in <module>
    source=source, statement=statement, question=question
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 1904, in groundedness_measure_with_cot_reasons_consider_answerability
    results.append(future.result())
                   ^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/core/utils/python.py", line 608, in _future_target_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 1860, in evaluate_hypothesis
    score, reason = self.generate_score_and_reasons(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/providers/bedrock/provider.py", line 257, in generate_score_and_reasons
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/core/feedback/endpoint.py", line 305, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint BedrockEndpoint request failed 4 time(s): 
        An error occurred (ThrottlingException) when calling the InvokeModel operation (reached max retries: 4): Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
        An error occurred (ThrottlingException) when calling the InvokeModel operation (reached max retries: 4): Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
        An error occurred (ThrottlingException) when calling the InvokeModel operation (reached max retries: 4): Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
        An error occurred (ThrottlingException) when calling the InvokeModel operation (reached max retries: 4): Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
```

**Environment:**

- OS: Ubuntu 22.04
- Python Version: 3.12.5
- TruLens version: 1.2.6


Hi @drooms-sandrus , I cannot replicate the issue. I'm adding an example notebook that uses Pace in multiple threads and am seeing expected results (see https://github.com/truera/trulens/blob/39b154b0a3df336750c44947424963db936ffc9d/examples/experimental/pace_example.ipynb ) . Two things might be happening:
- Pace allows for burst of requests and only maintains the target rate over a specified period of time. The rate is not guaranteed for time periods smaller than it. The period for the rate calculation can be reduced to reduce burstiness. Take a look at the notebook for more information.
- It may be possible that multiple Pace objects are being created in your case. In TruLens one is created for each Endpoint instance. Is that happening in your case?

Thanks @sfc-gh-pmardziel, I tried the examples in the notebook you provided and it seems the issue lies elsewhere.

The lowest possible "Marks per second" (printed in the notebook) averages to around 0.5:

```python
p = Pace(marks_per_second=0.5, seconds_per_period=2)
p.marks_per_second = 0.001
p.seconds_per_period = 0.002
```

leads to

```
Elapsed time: 26.354328870773315
Marks count: 14
Marks per second: 0.5312220268878051
```

This also explains why the `Pace` implementation raises for lower `marks_per_second` and `seconds_per_period`:

```python
max_marks = int(seconds_per_period * marks_per_second)
if max_marks == 0:
    raise ValueError(
    "Period is too short for the give rate. "
     "Increase `seconds_per_period` or `returns_per_second` (or both)."
)
```

The workaround above allows to achieve a lower theoretical `max_marks` by editing `Pace` after instantiation (I had to use this workaround in my code base due a related issue I reported [here](https://github.com/truera/trulens/issues/1646), and due to this workaround the error above was not being raised).

Is there a reason for why `max_marks` < 0.5 are not supported? I'm asking, because 0.5 is still too high for the example I provided at the beginning of this thread and I'm getting `botocore.errorfactory.ThrottlingException`.

 

Hi, looking over the `Pace` code, I think the idea is that `seconds_per_period` and `marks_per_second` should be used in a way that there's an integer number of marks per period. Otherwise, it's a bit strange. So for instance, if you wanted to have like 0.1 marks per second, you can set:
`marks_per_second=0.1` and `second_per_period=10`.
That way, you'll get what's desired. Does that make sense?

Hey @drooms-sandrus, I'm going to close this out for now but if you're still having issues please reopen.