Repository: openai/evals
Issue: Can't run evals?
Root cause: Documentation Deficiency

When I got on the API waitlist, the email I received said that if I contribute evals that get merged, I can gain priority access. And it happens that I do have prompts and responses that I think would make for useful evals.

It seems like, in order create and test my evals, I should be able to run them. Since I'm a paid ChatGPT Plus subscriber, I would assume that wouldn't be any problem.

But when I try to run 

`oaieval gpt-3.5-turbo test-match`

I get never-ending errors, even with a new API Key:

```
oaieval gpt-3.5-turbo test-match
[2023-04-03 13:00:46,754] [registry.py:153] Loading registry from /Users/garyrob/Source/openai/evals/evals/registry/evals
[2023-04-03 13:00:46,817] [registry.py:153] Loading registry from /Users/garyrob/.evals/evals
[2023-04-03 13:00:47,464] [oaieval.py:187] Run started: 2304032000473QO4OU2X
[2023-04-03 13:00:47,469] [data.py:75] Fetching test_match/samples.jsonl
[2023-04-03 13:00:47,469] [eval.py:32] Evaluating 3 samples
[2023-04-03 13:00:47,520] [eval.py:152] Running in threaded mode with 10 threads!
  0%|                                                                                                              | 0/3 [00:00<?, ?it/s][2023-04-03 13:00:47,777] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 0.4s (openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.)
[2023-04-03 13:00:47,778] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 0.1s (openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.)
[2023-04-03 13:00:47,872] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 1.0s (openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.)
[2023-04-03 13:00:47,970] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 2.8s (openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.)
[2023-04-03 13:00:48,278] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 2.1s (openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.)
[2023-04-03 13:00:48,927] [_common.py:105] Backing off openai_chat_completion_create_retrying(...) for 0.5s (openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.)
```

What can I do about this? Or should I just submit evals without being able to run them??

You have to add your payment details to your OpenAI account on [platform.openai.com](https://platform.openai.com)

OK, thanks. I assumed that since I wouldn't get real API access until I got off the waitlist, I wouldn't have to pay until then, and/or that the fact that I'm paying for ChatGPT would be enough.

It may be useful to make it clearer in the introduction that you have to sign up for a paid account even while on the waitlist, or maybe I'm usual. :)  Whatever. Just a thought.

We have some documentation on this in the Setup section of the README: https://github.com/openai/evals#setup, hope that is clear enough