Repository: openai/evals
Issue: Meaning of "elsuite" folder name

Hello,

I came across the "elsuite" folder in this repository and I'm curious about its meaning or purpose. Could you please explain the significance behind the name "elsuite" and what it represents in the context of this project?

Thank you for your time and assistance.

Best regards

Hello @siftxxx,

Thank you for your question. For future reference, you might find the discussion tab more appropriate for queries like these, as it's primarily designed for general questions and conversations. The issues tab tends to be reserved for bug reports or feature requests, which helps maintain an organized project environment.

Nonetheless, I appreciate your good faith in asking, and I'm happy to share what I understand about the subject. From my perspective as a frequent contributor, I believe "elsuite" is an abbreviation for Evaluation Logic Suite, or something similar. This directory/folder houses the implementations of the evaluation code used to measure model performance, driven by configurations in the specific evals [yaml](https://github.com/openai/evals/tree/main/evals/registry/evals) and [jsonl](https://github.com/openai/evals/tree/main/evals/registry/data) files when you execute either [oaieval or oaievalset](https://github.com/openai/evals/tree/main/evals/cli), which is typically done via [command-line](https://github.com/openai/evals/blob/main/docs/run-evals.md#how-to-run-evals).

If you delve into any of the existing evals in the [registry](https://github.com/openai/evals/tree/main/evals/registry) directory, you're likely to find a config snippet similar to this [yaml file](https://github.com/openai/evals/blob/1df05834aa52ee4265c226bf7fe85850b25df15c/evals/registry/evals/word-association.yaml#L6):
`class:evals.elsuite.basic.includes:Includes`
This essentially instructs the script to apply the evaluation logic located in [evals/elsuite/basic/includes.py](https://github.com/openai/evals/blob/main/evals/elsuite/basic/includes.py), specifically the [Includes](https://github.com/openai/evals/blob/1df05834aa52ee4265c226bf7fe85850b25df15c/evals/elsuite/basic/includes.py#L9) class. Most of the other classes found in the elsuite directory follow a similar pattern, though the details may vary.

All of the evaluation classes in elsuite inherit from the abstract base class [Eval](https://github.com/openai/evals/blob/1df05834aa52ee4265c226bf7fe85850b25df15c/evals/eval.py#L43). Any class that inherits from this base class must implement a [run()](https://github.com/openai/evals/blob/1df05834aa52ee4265c226bf7fe85850b25df15c/evals/eval.py#L81) and an [eval_sample()](https://github.com/openai/evals/blob/1df05834aa52ee4265c226bf7fe85850b25df15c/evals/eval.py#L72) method to operate within the existing framework. This abstract base class also provides auxiliary methods to facilitate multi-threaded evaluations among other things such as helper methods.

However, an [important note](https://github.com/openai/evals#writing-evals) to keep in mind when contributing to this project: 
> "Important: Please note that we are currently not accepting Evals with custom code! While we ask you to not submit such evals at the moment, you can still submit modelgraded evals with custom modelgraded YAML files."

Based on this, you should avoid submitting a Pull Request with your own elsuite Eval class. I'd recommend not modifying this directory beyond personal experimentation unless you have a very specific reason to do so.

I trust this gives you a decent understanding of the "elsuite" directory, though it is by no means exhaustive—there are numerous other classes and functions within that facilitate the process. I hope this information has been helpful for your needs. Should you require any additional clarification or have further questions, feel free to raise your queries in the discussion tab. Feel free to tag me for any follow-up discussions.

It's crucial to note that, **although I am an active contributor, I'm not officially associated with OpenAI**, nor am I a collaborator or code owner of this project. Therefore, if you prefer to wait for a response from a more authoritative source, it's completely understandable. I'm here to assist as much as I can, but for definitive answers, please refer to the official project leads or maintainers.

_As a final note, for more details about this directory, you may also want to read this section of the documentation: 
[Existing templates for evals](https://github.com/openai/evals/blob/main/docs/eval-templates.md#existing-templates-for-evals>)_

Edit: This post resolved #1282 

Thank you for your detailed response! I really appreciate the time and effort you took to address my issue. Your explanation has been very helpful, and I now have a better understanding of the project. Thanks again for your assistance and for your great work on this project!

Best regards!