Repository: openai/evals
Issue: Evaluating LLMs on QA Tasks

Here's an idea on how to evaluate an LLM on various question-answering tasks, such as open-domain question answering, conversational question answering, answer selection, community question answering, and knowledge base question answering:

```
initialize model
initialize datasets
initialize evaluation_metrics

load_task_data:
    for each task in tasks:
        load data for task
        preprocess data if necessary (e.g., combine review summary and text)
        store data in datasets

embed_task_data:
    for each task in tasks:
        for each example in datasets[task]:
            obtain prompt from example
            obtain prompt_embedding using an embedding function
            store prompt_embedding in example

evaluate_model_on_task:
    for each task in tasks:
        for each example in datasets[task]:
            obtain prompt_embedding from example
            generate_answer_embedding = model.generate(prompt_embedding)

            calculate_metric = evaluation_metrics(example, generate_answer_embedding)
            store_metric_results_for_task(task)

aggregate_and_report_metrics:
    for each task in tasks:
        for each metric in evaluation_metrics:
            calculate average, median, or other aggregate metric values
            report metric value for task

main:
    load_task_data
    embed_task_data
    evaluate_model_on_task
    aggregate_and_report_metrics
```

I'd like to add a caveat about the pseudocode I provided:

- The provided pseudocode is only a starting point for exploring the evaluation of QA tasks using embeddings
- This pseudocode is not complete
- I invite the community to provide input

Hey there! Thanks for sharing your idea on how to evaluate an LLM on various question-answering tasks. I really appreciate your contribution and I think your pseudocode provides a great starting point for exploring and understanding the evaluation process. And you're right, there's always room for improvement, so I encourage you and others to share your thoughts and experiences to help enhance the understanding and implementation of this process. Keep up the good work! 


quite impressive

@slavakurilyak ok I know this is a weird question, but...did you generate this with ChatGPT? 👀 

It has a very similar tone. The pseudocode, the disclaimers, the step-by-step thing. It's very similar to when I ask ChatGPT for coding help.

Maybe

On Wed, 15 Mar, 2023, 11:07 am ricky-sb, ***@***.***> wrote:

> @slavakurilyak <https://github.com/slavakurilyak> ok I know this is a
> weird question, but...did you generate this with ChatGPT? 👀
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/openai/evals/issues/65#issuecomment-1469371857>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AY2ADJTJH6UB7EAO3CU2WRTW4FIQLANCNFSM6AAAAAAV3CYJAM>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


Id like to contribute.

🤓 Me ✌️ from 🇷🇼

The tasks described:
```
question-answering tasks, such as open-domain question answering, conversational question answering, answer selection, community question answering, and knowledge base question answering:
```

should already be supported by Evals, as you can make the input a Chat conversation object up until the next turn (which is when the model would respond)