Repository: openai/evals
Issue: Does not throttle API requests appropriately.
Root cause: Architectural Constraint

### Describe the bug

When running an eval on a Windows 11 machine, it makes too many requests to the api too quickly. I get a message about once per second saying it is going to back off for 20 seconds.

I've fixed this kind of problem in my own code base before by using something like time.sleep(20). However, I'm not familiar with the entire code base of this repository, so I don't know exactly which line to insert that without doing more research.

### To Reproduce
I added "set OMP_NUM_THREADS=1 &&" to the beginning in order to try to only run 1 thread.
The rest of the command is "oaieval gpt-3.5-turbo finance", to run the finance eval since that is one of the shorter examples I saw. I've copied and pasted the session below, then I killed it before it completed running:

C:\Python310>set OMP_NUM_THREADS=1 && oaieval gpt-3.5-turbo finance
[2023-04-23 00:51:13,249] [registry.py:249] Loading registry from C:\Python310\Lib\site-packages\evals\registry\evals
[2023-04-23 00:51:13,293] [registry.py:249] Loading registry from C:\Users\james\.evals\evals
[2023-04-23 00:51:13,303] [oaieval.py:110] Run started: 230423075113AQXOZKK4
[2023-04-23 00:51:13,304] [data.py:75] Fetching finance/credit.jsonl
[2023-04-23 00:51:13,309] [eval.py:34] Evaluating 5 samples
[2023-04-23 00:51:13,309] [eval.py:153] Running in threaded mode with 10 threads!
  0%|                                                                                            | 0/5 [00:00<?, ?it/s][2023-04-23 00:51:13,483] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 0.8s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:13,628] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 0.1s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:13,837] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 2.2s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
 20%|████████████████▊                                                                   | 1/5 [00:00<00:02,  1.46it/s][2023-04-23 00:51:14,346] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 1.3s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:15,678] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 3.9s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:16,128] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 4.1s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:19,638] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 4.9s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:20,303] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 2.1s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)
[2023-04-23 00:51:22,493] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 5.1s (openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-joVgZTYl2HzxPqx2nr8leC1d on requests per min. Limit: 3 / min. Please try again in 20s. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.)


### Code snippets

_No response_

### OS

Windows 11

### Python version

Python 3.10.11

### Library version

evals in c:\python310\lib\site-packages (1.0.3.post1)

I am running on Windows 11 but within WSL.  tried your command and got a fast and clean response. 
It is not a solution but walk around.

> set OMP_NUM_THREADS=1 && oaieval gpt-3.5-turbo finance
[2023-04-23 17:53:45,983] [registry.py:249] Loading registry from /home/ido/openai/evals/evals/registry/evals
[2023-04-23 17:53:46,048] [registry.py:249] Loading registry from /home/ido/.evals/evals
[2023-04-23 17:53:46,051] [oaieval.py:110] Run started: 2304231453465WNZI27B
[2023-04-23 17:53:46,052] [data.py:75] Fetching finance/credit.jsonl
[2023-04-23 17:53:46,054] [eval.py:34] Evaluating 5 samples
[2023-04-23 17:53:46,059] [eval.py:153] Running in threaded mode with 10 threads!
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:02<00:00,  1.86it/s]
[2023-04-23 17:53:48,747] [record.py:320] Final report: {'accuracy': 0.2, 'f1_score': 0.0}. Logged to /tmp/evallogs/2304231453465WNZI27B_gpt-3.5-turbo_finance.jsonl
[2023-04-23 17:53:48,747] [oaieval.py:147] Final report:
[2023-04-23 17:53:48,747] [oaieval.py:149] accuracy: 0.2
[2023-04-23 17:53:48,748] [oaieval.py:149] f1_score: 0.0
[2023-04-23 17:53:48,752] [record.py:309] Logged 15 rows of events to /tmp/evallogs/2304231453465WNZI27B_gpt-3.5-turbo_finance.jsonl: insert_time=3.222ms


I don't think that is a walk around at all. The logs/output that you pasted show you ran 5 samples in 3 seconds. I was trying to run this within the free tier of the API, which limits API calls to three per minute. I'm looking at some other walk arounds, like telling my computer to only use 1 processor for everything it is running, or using a slower computer. But I would prefer if the evals repository would automatically throttle to whatever speed openai wants us to use rather than failing to throttle down and then forcing a disconnection alter when we are half way through running an eval.

We have a environment variable you can set called `EVALS_THREADS`, which determines the number of parallel calls being made. Could you try setting it to 1 to see if that helps? We describe it here: https://github.com/openai/evals/blob/main/docs/run-evals.md#running-an-eval-set

OK, that didn't quite work, but it pointed me to a work around that does work. In evals\eval.py I added three lines of code. In the imports I added:

> from time import sleep

Then I added print and sleep statements in this method function like so:
```
        def worker_thread(args):
            """
            Worker thread for evaluating a single sample.
            """
            while True:
                executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
                future = executor.submit(eval_sample, args=args)
                try:
                    result = future.result(timeout=timeout)
                    print("timeout")
                    sleep(timeout)
                    return result
                except concurrent.futures.TimeoutError as e:
                    executor.shutdown(wait=False)
```

Finally I use this command in Windows 11 command prompt to run my own eval (pull request #812):

> set EVALS_THREADS=1 && set EVALS_THREAD_TIMEOUT=20 && oaieval gpt-3.5-turbo beam-analysis

I also defined EVALS_THREADS and EVALS_THREAD_TIMEOUT as system environmental variables, so they've been set twice. That changes the actual timing of API calls to about 21.1 seconds apart, allowing evals to be run on the free tier of the API from a fast computer in well behaved manner.

Great, I would be happy to review a PR if you're interested in contributing this Env Var.

Closing the issue for now as it seems resolved.