Paper: GAIA: a benchmark for General AI Assistants (ICLR-2024)

Reviewer comment (Evaluation):
The paper mentions that "level 3 are questions for a near-perfect general assistant." How do you define a perfect general assistant, and how do you ensure that good performance on your dataset (at level 3) implies a perfect general assistant? These questions could help determine if there are other abilities not covered by your dataset.

Author response:
We thank the reviewer for their insightful comments.

## On perfect general assistants

We think about Level 3 as questions requiring arbitrarily many tools and steps (albeit staying conceptually simple for humans). We dub an assistant solving this level as “perfect” since there is theoretically no upper bound for the number of tools and steps. In addition, while the number of steps and tools required grows linearly between levels, the associated difficulty grows exponentially since one more step means many new possible mistakes.

Reviewer comment (Experiments):
What is the distribution of questions across the three levels?

Author response:
We thank the reviewer for their insightful comments.

## Distribution of questions and detailed performance table

There are 146 Level 1 questions, 245 Level 2 questions, and 75 Level 3 questions. The aggregated scores of each model is therefore as follows:

|               | Level 1 | Level 2 | Level 3 | Aggregated |
|---------------|---------|---------|---------|------------|
|Number of questions|146|245|75|466|
|Model and scores |||||
| Search Engine |    7.4     |     0    |     0    |     2.3       |
| GPT4          |     6.4    |     2.5    |    0     |   3.3         |
| AutoGPT4      |   14.4      |  6.5       |     0    |    7.9        |
| GPT4 + plugins|  30.3       |    9.7     |     0    |     14.6       |
| Human |  93.9       |    91.8     |     87.3    |   91.7        |

We will update Table 4 in the revised manuscript to include this information.

Reviewer comment (Presentation):
It would be helpful to have a detailed table showing the performance of each model, including the total score for all three levels.

Author response:
We thank the reviewer for their insightful comments.

## Distribution of questions and detailed performance table

There are 146 Level 1 questions, 245 Level 2 questions, and 75 Level 3 questions. The aggregated scores of each model is therefore as follows:

|               | Level 1 | Level 2 | Level 3 | Aggregated |
|---------------|---------|---------|---------|------------|
|Number of questions|146|245|75|466|
|Model and scores |||||
| Search Engine |    7.4     |     0    |     0    |     2.3       |
| GPT4          |     6.4    |     2.5    |    0     |   3.3         |
| AutoGPT4      |   14.4      |  6.5       |     0    |    7.9        |
| GPT4 + plugins|  30.3       |    9.7     |     0    |     14.6       |
| Human |  93.9       |    91.8     |     87.3    |   91.7        |

We will update Table 4 in the revised manuscript to include this information.

Reviewer comment (Evaluation):
How are humans scored? The paper writes "Human score corresponds to the portion of correct answers by validation annotators for valid questions." Are these the same validators involved for validating the data or a new fresh set of annotators? This is important as there are many datasets that claim suspiciously high human performance because they didn't run validation with a new set of annotators.

Author response:
The set of annotators used to measure human performance is different from the set that created the questions and there is no second try, hence we believe this score is reliable. To clarify, our process is as follows: 
- First, an annotator A creates a question and reports its answer.
- Then, the question is given to 2 new annotators B and C to answer. B does not see A and C’s answer, and similarly for C. There are two cases:
    - If the answers from annotators A, B, and C are similar the question is correct and the humans scored it well. We count B and C as correct answers for this question. 
    - If the answers are different, the annotators assess whether this was due to an ambiguity in the question or a human error, and there are again two cases:
         - The question was ambiguous, e.g., “How many layers does BERT have?” and one Reviewer answered 6 referring to BERT-small while the others said 12 referring to BERT-base. We do not use it to compute a human score.
         - The question was clear and an annotator, B or C, made a clear mistake when answering it. We count it as a human error.

We hope this clarification solves the reviewer's legit concerns.

Reviewer comment (Evaluation):
How do you grade open-ended answers? For example, "St. Petersburg" vs "Saint Petersburg". It was unclear from the paper how the correctness is actually determined.

Author response:
Thanks for raising this important point, we will add the following paragraph on evaluation in the manuscript:

“In practice, each question is formulated so that the correct answer is unique and is formatted to be either a string (one or a few words), a number, or a comma separated list of strings or floats, unless specified otherwise. There is therefore only one correct answer. Hence, evaluation can be done via exact match between a model's answer and the ground truth (up to some simple normalization that is tied to the “type” of the ground truth). A system (or prefix) prompt is used to inform the model about the required format. In practice, GPT4 level models easily follow our format.”

Prefix prompt: *You are a general AI assistant. I will ask you a question. Report your thoughts, and finish your answer with the following template: FINAL ANSWER: [YOUR FINAL ANSWER].
YOUR FINAL ANSWER should be a number OR as few words as possible OR a comma separated list of numbers and/or strings.
If you are asked for a number, don't use comma to write your number neither use units such as $ or percent sign unles specified otherwise.
If you are asked for a string, don't use articles, neither abbreviations (e.g. for cities), and write the digits in plain text unless specified otherwise.
If you are asked for a comma separated list, apply the above rules depending of whether the element to be put in the list is a number or a string.*

When needed, the question also details the format. Importantly, some examples in our submission including the one mentioned by the reviewer mistakenly come from an older version of the dataset. In the current GAIA version, the prefix prompt and, when needed, the question asks to not use abbreviations. Hence, the question in the manuscript is actually:

*Where were the Vietnamese specimens described by Kuznetzov in Nedoshivina's 2010 paper eventually deposited? Just give me the city name without abbreviations.*

St. Petersburg would actually be incorrect in the current version. We fixed this in the manuscript.

Finally, we provide our scoring function in the supplementary material.

Reviewer comment (Experiments):
Curiosity as to whether GPT-4 with vision can answer those questions that currently require such tools.

Author response:
GPT4-vision was not available at the time of the writing, and the API is currently limited to 100 requests per day and organization. We will add GPT4-vision results by the end of the rebuttal if possible (it will require at least a few days to run because of OpenAI’s limitation), and commit to add it in the final version of the paper.

Reviewer comment (Experiments):
The biggest limitation not fully discussed is the dataset's modest size, which, unfortunately, restricts it solely to the role of a performance benchmark rather than a potential source of training data. This limits the paper's potential impact.

Author response:
We thank the reviewer for their insightful comments. 


## Size of the dataset 

We discuss this concern, that is shared with other reviewers, in our general response.

Reviewer comment (Evaluation):
Another issue, discussed, but not fully acknowledged as a limitation, is the lack of "partial success" indicators. This is mitigated by the presence of questions with different difficulty levels, but might still be problematic. When solving a question requires a complex sequence of actions, it is highly desirable to have some measure of where the process breaks down. While the authors speculate that, potentially, in the future "the paradigm of LLMs calling tools for every task other than text understanding might not stay", the reality of today is such that having more nuanced/partial feedback would be helpful.

Author response:
We thank the reviewer for their insightful comments. 


## Adding indicators of partial success

This is also a great point. We already release the annotator's trace for the validation set, which can be used for this purpose since it contains which tools were used, how, and very often, intermediate information required to compose the final answer. Moving forward, it would be easy to modify the protocol to systematically include such intermediate information in a way that is easy to process. However, we believe it is not desirable to release such data for the test set, as this could lead to overfitting. We will add this discussion in the manuscript.

Reviewer comment (Experiments):
The authors insist that a component of human labor is crucial to create high-quality questions ("we believe human annotators are currently essential to have diverse and grounded questions, as opposed to programmatically generated ones"). However, this might be underselling one key idea behind their own work. It seems that the questions in GAIA are challenging because they require reverse-engineering a fairly random process of question generation, which includes randomly choosing which tool to use, what to google, and so on. The problems are challenging for AIs because they are under-defined inverse problems. At the same time, problem generation is a straightforward process, and the instructions given in appendix C could quite possibly be adapted to tool-assisted LLM prompts. Such LLM-driven automation might offer a good compromise of data price and quality, and such approaches seem to be getting traction (e.g. https://arxiv.org/abs/2303.15056, https://arxiv.org/abs/2305.13877). The reviewer encourages the authors not to dismiss this direction, as adding such automation might dramatically increase the impact of their work, and at the very least, it might be worth acknowledging this possibility in the discussion.

Author response:
We thank the reviewer for their insightful comments. 


## Further considering automation

This is a great point that we will discuss in the revised manuscript. 
- Current best models are not good enough for diverse generations given many strict constraints such as in our protocol (the most important and difficult being unambiguity in the question), which we believe is a different setup from https://arxiv.org/abs/2303.15056 and https://arxiv.org/abs/2305.13877. This is why human labor was instrumental to create valid questions. In fact, even human annotators had difficulties enforcing all the constraints in our protocol. Hence, we are pessimistic in the short term on automating generation of both varied and unambiguous questions.
- However, the problem becomes much more tractable by removing one of the constraints. Removing the diversity constraint would allow to automatically generate unambiguous questions for a number of sources of truth limited by the human workforce needed to write associated code for each source. For example, given a source of truth such as wikipedia, one could come with an implementable set of rules leading to unambiguous questions and automatically fetch the answer. Removing the unambiguity constraint seems more scalable. In that case however, the answer, that would not be unique anymore, should probably be evaluated differently e.g. via a reward model (for the reasoning path), a judge model, or a human judge, with associated downsides in each setup.

To conclude, removing the ambiguity constraint is a promising path towards automated generation of GAIA-type data for training, at the expense of making factual evaluation of the final answer challenging again.

Reviewer comment (Evaluation):
As a potential way to mitigate the absence of "partial success indicators", it might be useful to release a version of the dataset where one possible ground truth solution trace is explored and, potentially, in which associated files are pre-processed (e.g. images are replaced by their detailed verbal descriptions). This simplified dataset version would allow better diagnosis of existing models, and help disentangle a) conceptual difficulties of aggregating information from different sources, and b) procedural challenges (choosing what tool to use, what to google, and, in general, what information to collect).

Author response:
We thank the reviewer for their insightful comments. 


## Adding indicators of partial success

This is also a great point. We already release the annotator's trace for the validation set, which can be used for this purpose since it contains which tools were used, how, and very often, intermediate information required to compose the final answer. Moving forward, it would be easy to modify the protocol to systematically include such intermediate information in a way that is easy to process. However, we believe it is not desirable to release such data for the test set, as this could lead to overfitting. We will add this discussion in the manuscript.

Reviewer comment (Novelty):
The venue is not a great fit. The majority of the contribution here is annotated dat, and the design decisions made in doing so. This would make the work a better fit for conferences focussing on these aspects, including CHI and UIST. There are no learned representations, or models, putting it out of the domain of the ICLR community.

Author response:
The ICLR 2024 website lists benchmarks as a topic relevant to the conference. For example, the influential MMLU benchmark was published at ICLR 2021. Therefore, we cannot agree with the reviewer on the lack of fit for the venue. We would also like to emphasize that progress towards “learning better representation” would not be possible without clear ways to measure said progress, such as our proposed GAIA.

Reviewer comment (Experiments):
Experiments very thin: The utility of designing a benchmark could be justified if it taught reasonable insights about the behaviour of existing models. In its current form, the investigations presented in the paper do not offer any new insights about the behavior of these models.

Author response:
To the best of our knowledge, our experiments do provide new insights into current models:  in spite of being the best available assistants equipped with relevant tools such as web browsing or image understanding (in the form of plugins), the models we evaluate do badly at our realistic yet simple tasks. In particular, Figure 4 displays the performance of LLMs on GAIA per capability. This suggests that, beyond giving LLMs access to tools, a lot more work is needed for LLMs to become viable assistants, which goes against a popular discourse in the AI community.

We discuss the possibility for more detailed analysis in the general comment. The lack of more detailed insights is due to the scarcity in (i) capable models to evaluate and (ii) detailed logs of OpenAI model’s answers, including detailed tool calls. We are confident that going forward, such analysis will be possible thanks to submissions including detailed log and / or more open models. Does the reviewer have a particular experiment in mind?