Repository: openai/evals
Issue: Accuracy is misreported as 100% although all answers are incorrect.
Root cause: Validation Gap

### Describe the bug

I was having a hard time believing that the GPT-3.5-turbo is getting everything correct although ChatGPT gets my questions wrong. So I decided to use some rather boring questions that ChatGPT almost always gets wrong. The reported accuracy was 1.0, which I take to mean 100%. So I added some print statements to the functions that make api calls so I could personally inspect the responses and see if they are accurate. The way I figure, I either get a work around to get better accuracy than ChatGPT, or I reveal some inconsistency in the reported accuracy. It appears that API calls are returning incorrect answers, but somehow the accuracy score is still reported as 100%.

### To Reproduce

Step 1. Add "print(results)" to each function that makes an api call in api.py and api_utils.py, so in particular, this code in the file 'api.py' :
```
   result["expected"] = expected
   result["match"] = match
   record_match(match, expected=expected, picked=picked, sampled=sampled, options=options)
   return picked
```

changes to:
```
   result["expected"] = expected
   result["match"] = match
   print(result)
   record_match(match, expected=expected, picked=picked, sampled=sampled, options=options)
   return picked
```

and two instances of the following code in 'api_utils.py' :
```
       raise openai.error.APIError(result["error"])
   return result
```

changes to:
```
       raise openai.error.APIError(result["error"])
   print(results)
   return result
```
    
Step 2. I then have a file at "C:\Python310\Lib\site-packages\evals\registry\data\AISC_areas\AISC-areas.jsonl" which has these three json lines in it:

```
{"input": [{"role": "system", "content": "You will be asked a question about a section property of a structural steel shape based on an AISC shapes database. Respond only with the correct answer as a number with three significant figures."}, {"role": "user", "content": "What is the cross-sectional area of a WT8X50 beam in square inches?\n\n"}], "ideal": "14.7"}
{"input": [{"role": "system", "content": "You will be asked a question about a section property of a structural steel shape based on an AISC shapes database. Respond only with the correct answer as a number with three significant figures."}, {"role": "user", "content": "What is the cross-sectional area of a WT8X44.5 beam in square inches?\n\n"}], "ideal": "13.1"}
{"input": [{"role": "system", "content": "You will be asked a question about a section property of a structural steel shape based on an AISC shapes database. Respond only with the correct answer as a number with three significant figures."}, {"role": "user", "content": "What is the cross-sectional area of a WT8X38.5 beam in square inches?\n\n"}], "ideal": "11.3"}
```

Step 3. Then I have another file at "C:\Python310\Lib\site-packages\evals\registry\evals\AISC-areas.yaml" that contains the following:
```
AISC-areas:
  id: AISC.areas.dev.v0
  description: Test the model's ability to solve beam analysis questions
  metrics: [accuracy]
AISC.areas.dev.v0:
  class: evals.elsuite.basic.fuzzy_match:FuzzyMatch
  args:
    samples_jsonl: AISC_areas/AISC-areas.jsonl
```

Step 4. Then I run the eval, as so:
C:\Python310\evals>oaieval gpt-3.5-turbo AISC-areas
[2023-04-23 20:04:04,514] [registry.py:249] Loading registry from C:\Python310\Lib\site-packages\evals\registry\evals
[2023-04-23 20:04:04,556] [registry.py:249] Loading registry from C:\Users\james\.evals\evals
[2023-04-23 20:04:04,556] [oaieval.py:110] Run started: 230424030404ZWWLMZUT
[2023-04-23 20:04:04,568] [data.py:75] Fetching AISC_areas/AISC-areas.jsonl
[2023-04-23 20:04:04,568] [eval.py:34] Evaluating 3 samples
[2023-04-23 20:04:04,568] [eval.py:153] Running in threaded mode with 10 threads!
  0%|                                                                                            | 0/3 [00:00<?, ?it/s]{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "11.2",
        "role": "assistant"
      }
    }
  ],
  "created": 1682305444,
  "id": "chatcmpl-78gmipkIMuMUcyUYxLq9Tm87gQId5",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 3,
    "prompt_tokens": 69,
    "total_tokens": 72
  }
}
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "11.8",
        "role": "assistant"
      }
    }
  ],
  "created": 1682305444,
  "id": "chatcmpl-78gmiCZES6xHg1FZF6e4MyoSnJr3E",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 3,
    "prompt_tokens": 67,
    "total_tokens": 70
  }
 33%|████████████████████████████                                                        | 1/3 [00:00<00:01,  1.57it/s]
{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "The cross-sectional area of a WT8X44.5 beam is 11",
        "role": "assistant"
      }
    }
  ],
  "created": 1682305444,
  "id": "chatcmpl-78gmivQG5ooLTuPsvN7PEouj8jjVO",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 16,
    "prompt_tokens": 69,
    "total_tokens": 85
  }
}
100%|████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.09it/s]
[2023-04-23 20:04:05,538] [record.py:320] Final report: {'accuracy': 1.0, 'f1_score': 0.0}. Logged to /tmp/evallogs/230424030404ZWWLMZUT_gpt-3.5-turbo_AISC-areas.jsonl
[2023-04-23 20:04:05,538] [oaieval.py:147] Final report:
[2023-04-23 20:04:05,538] [oaieval.py:149] accuracy: 1.0
[2023-04-23 20:04:05,538] [oaieval.py:149] f1_score: 0.0
[2023-04-23 20:04:05,538] [record.py:309] Logged 9 rows of events to /tmp/evallogs/230424030404ZWWLMZUT_gpt-3.5-turbo_AISC-areas.jsonl: insert_time=0.000ms

C:\Python310\evals>

So all the answers in the API response are wrong in that they don't contain the "ideal" answer from the JSON lines, yet accuracy is reported as 1.0.

### Code snippets

_No response_

### OS

Windows 11

### Python version

Python 3.10.11

### Library version

evals in c:\python310\lib\site-packages (1.0.3.post1)

You know that you're using fuzzy match, meaning that if any of the keywords given in the `ideal` field appear in the answer the eval will succeed? 

If you look at the JSONs in the .JSONL file, the value associated with the "ideal" keyword is just a string representation of a number, and it doesn't match anything being returned by the API's message content. For instance, for the first one "ideal" is "14.7".
The "content" returned by the API is "11.2".

For the second we have "ideal": "13.1", and from the API call "content": "11.8".

For the third we have "idea": "11.3", and from the API call "content": "The cross-sectional area of a WT8X44.5 beam is 11".

None of them actually match, but they are being counted as matches somehow.

I realize that each of them in this example shares the character "1". But is that how FuzzyMatch works? I thought with FuzzyMatch, one string has to contain the other - not just have a character overlap. I read the source code for it some days ago. I was even joking about it not being fuzzy because it seems to return a boolean not a float, nothing like the FuzzyWuzzy module functions. But it didn't look this poorly behaved. So I'm not sure whether or not that is where the problem is.

Also, even if they were processed out of order, I only had these three in this run. None of these "ideal" values - "14.7", "13.1", or "11.3" - should match any of the message contents from the API from this run - "11.2", "11.8", or "The cross-sectional area of a WT8X44.5 beam is 11".

Have you tried not using fuzzy match? What are the results? 

Thanks for the detailed bug report, I was quickly able to reproduce the issue.

FuzzyMatch currently was expecting the "ideal" answer to be a list of strings, e.g. `... "ideal": ["14.7"]}` instead of `... "ideal": "14.7"}`. It was iterating over the string chars as a result which is why you were seeing the incorrect accuracies.

This should be remedied with this fix: https://github.com/openai/evals/pull/800, I get 0% accuracy on your 3 line sample from `AISC_areas\AISC-areas.jsonl`