Repository: openai/evals
Issue:  assert ( AssertionError: Eval match_mmlu_anatomy not found. Available: ['coqa-closedqa',
Root cause: Validation Gap

hi, 
The following message appears when running " !oaieval gpt-3.5-turbo match_mmlu_anatomy", and "/tmp/evallogs" directory cannot generated. 

[2023-03-21 10:09:35,502] [registry.py:145] Loading registry from /home/coder/.local/lib/python3.9/site-packages/evals/registry/evals
[2023-03-21 10:09:35,533] [registry.py:145] Loading registry from /home/coder/.evals/evals
Traceback (most recent call last):
  File "/home/coder/.local/bin/oaieval", line 8, in <module>
    sys.exit(main())
  File "/home/coder/.local/lib/python3.9/site-packages/evals/cli/oaieval.py", line 225, in main
    run(args)
  File "/home/coder/.local/lib/python3.9/site-packages/evals/cli/oaieval.py", line 124, in run
    assert (
AssertionError: Eval match_mmlu_anatomy not found. Available: ['coqa-closedqa', 'coqa-closedqa.dev.v0', 'coqa-fact', 'coqa-fact-expl', 'coqa-fact-expl.dev.v0', 'coqa-fact.dev.v0', 'coqa-match', 'coqa.match.dev.v0', 'diversity', 'diversity.dev.v0', 'joke-animals', 'joke-animals-likert', 'joke-animals-likert.dev.v0', 'joke-animals-vs-fruits', 'joke-animals-vs-fruits.dev.v0', 'joke-animals.dev.v0', 'joke-fruits', 'joke-fruits-ans-meta', 'joke-fruits-ans-meta.dev.v0', 'joke-fruits-expl-meta', 'joke-fruits-expl-meta.dev.v0', 'joke-fruits-meta', 'joke-fruits-meta.dev.v0', 'joke-fruits.dev.v0', 'logic-fact', 'logic-fact.dev.v0', 'rap-animals-vs-fruits', 'rap-animals-vs-fruits.dev.v0', 'rap-people-vs-fruits', 'rap-people-vs-fruits.dev.v0', 'rap-people-vs-people', 'rap-people-vs-people.dev.v0', 'test-fuzzy-match', 'test-fuzzy-match.s1.simple-v0', 'test-includes', 'test-includes.s1.simple-v0', 'test-match', 'test-match.s1.simple-v0']


How should I fix it?  thank you !


Ja się ma tym kompletnie nie znam

wt., 21 mar 2023, 11:35 użytkownik 123456xxn ***@***.***>
napisał:

> hi,
> The following message appears when running " !oaieval gpt-3.5-turbo
> match_mmlu_anatomy", and "/tmp/evallogs" directory cannot generated.
>
> [2023-03-21 10:09:35,502] [registry.py:145] Loading registry from
> /home/coder/.local/lib/python3.9/site-packages/evals/registry/evals
> [2023-03-21 10:09:35,533] [registry.py:145] Loading registry from
> /home/coder/.evals/evals
> Traceback (most recent call last):
> File "/home/coder/.local/bin/oaieval", line 8, in
> sys.exit(main())
> File
> "/home/coder/.local/lib/python3.9/site-packages/evals/cli/oaieval.py", line
> 225, in main
> run(args)
> File
> "/home/coder/.local/lib/python3.9/site-packages/evals/cli/oaieval.py", line
> 124, in run
> assert (
> AssertionError: Eval match_mmlu_anatomy not found. Available:
> ['coqa-closedqa', 'coqa-closedqa.dev.v0', 'coqa-fact', 'coqa-fact-expl',
> 'coqa-fact-expl.dev.v0', 'coqa-fact.dev.v0', 'coqa-match',
> 'coqa.match.dev.v0', 'diversity', 'diversity.dev.v0', 'joke-animals',
> 'joke-animals-likert', 'joke-animals-likert.dev.v0',
> 'joke-animals-vs-fruits', 'joke-animals-vs-fruits.dev.v0',
> 'joke-animals.dev.v0', 'joke-fruits', 'joke-fruits-ans-meta',
> 'joke-fruits-ans-meta.dev.v0', 'joke-fruits-expl-meta',
> 'joke-fruits-expl-meta.dev.v0', 'joke-fruits-meta',
> 'joke-fruits-meta.dev.v0', 'joke-fruits.dev.v0', 'logic-fact',
> 'logic-fact.dev.v0', 'rap-animals-vs-fruits',
> 'rap-animals-vs-fruits.dev.v0', 'rap-people-vs-fruits',
> 'rap-people-vs-fruits.dev.v0', 'rap-people-vs-people',
> 'rap-people-vs-people.dev.v0', 'test-fuzzy-match',
> 'test-fuzzy-match.s1.simple-v0', 'test-includes',
> 'test-includes.s1.simple-v0', 'test-match', 'test-match.s1.simple-v0']
>
> How should I fix it? thank you !
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/378>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A6PJHHTVY3VB66KPF5U5PYLW5GAAJANCNFSM6AAAAAAWCHU27M>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


@123456xxn  You have to run `pip install .`. See https://github.com/pisrcio/evals-mahjong/blob/main/docs/custom-eval.md

As mentioned by @zxhmike, pip install should fix it. You can also use `pip install -e .` to install in editable so that you don't need to rerun pip install after modifying your local repo.