Paper: The Generative AI Paradox: “What It Can Create, It May Not Understand” (ICLR-2024)

Reviewer comment (Experiments):
The results for the vision domain are fundamentally flawed because the Paradox hypothesis in Hyp 1 refers to the **same model** being equivalent in generative performance but worse in discriminative performance relative to humans. Therefore, it is not experimentally sound to have **different** vision models for understanding vs. generation. An adequate investigation into the hypothesis requires having the same model (as in the language task, e.g., GPT tested on generative and discriminative tasks) rather than one model for generative and (more than one, but different) for discriminative. This discrepancy invalidates any of the vision domain results.

Author response:
### **“I think it is not experimentally sound to have different vision models for understanding vs. generation”**
We agree that a single vision model that both generates and understands would be an ideal system to test here. However, we make some clarifications to explain our decision:

1. Hypothesis 1 is referring to generative models as a technology rather than a single model in particular. That is, surprisingly good generation is achievable with generative models, compared to sometimes weak understanding that such models have achieved. We have worked to make this clearer in the paper text. (Section 2.1)

2. Vision models are capable of both image generation and understanding. Image generators can be used for understanding [1] but with significantly weaker performance than image understanding specialist models (which we use). Thus, our current results likely underestimate the extent of the generation-understanding discrepancy when considering a single model, as the generation performance will remain the same but understanding performance will be lower than what we have reported. 

Thank you for bringing up this ambiguity. We have now clarified this in the paper text.

Reviewer comment (Experiments):
The authors should focus solely on the language domain unless they can obtain a vision (or language+vision) model for which generative and discriminative performance can be studied jointly in the same system. If focusing on just language, Fig 10 should be moved to the main text. However, making this change would be a major revision, leading to the reviewer recommending rejection at this time despite thinking the paradox idea and language results warrant further study.

Author response:
Regarding your concern about Hypothesis 1 and its discourse, we acknowledge the potential for confusion. To address this, we explicitly specified model_u and model_g in Equation 1 (Sec 2.1), making it clear that these can be different models for both generation and understanding (Sec 2.2.2 ). We further highlighted why we considered the same model for language versus two models for vision (Sec 2.2.2 ).

Reviewer comment (Theory):
The authors are perhaps too flippant about the use of the words “understanding” and “intelligence” from a human cognition perspective. They are encouraged to look into Gardner’s Theory of Multiple Intelligences. This is a smaller weakness compared to the first point.

Author response:
### **“the authors are perhaps a bit too flippant about the use of the word ‘understanding’ and ‘intelligence’ from a human cognition perspective.”**
We use these terms following the conventions in the field, specifically considering commonly used terms like “Artificial Intelligence” and “Natural Language Understanding”. We have further clarified that these words are intended in a directed way, as when used in the given terms. (Section 1)

Reviewer comment (Experiments):
What data was used for human generations in the language domain? This was not clearly spelled out. Was the language data from the benchmarks discussed? If so, provide more details.

Author response:
### **“What data was used for human generations in the language domain? This was not clearly spelled out from my reading?”**
We use the human-written reference responses from the existing datasets that we test on. As these tend to be extensively validated, the only risk here is of overestimating human performance which would underestimate the prevalence of the paradox. This has been clarified in the paper text  (Section 3 and Appendix D).

Reviewer comment (Evaluation):
Please provide further details on how participant agreement was calculated. In what sense was it pairwise? The Appendix footnote references “kappa” — is this Cohen’s Kappa? Provide more details on the skew noted.

Author response:
### **“Can you please provide further details on how participant agreement was calculated?”**
There is a natural label bias in the annotations where many of the responses fell into a single label. In this case, standard inter-annotator agreement statistics are not reliable (the well-known paradoxes of kappa [2]). We thus reported the pairwise agreement, that is, percentage of the time two or more annotators agreed on a label. .This has been clarified in the appendix footnote 8. 
### **“Can you provide more details on the skew noted?”**
This is explained by the response above, with respect to label imbalance.

Reviewer comment (Presentation):
Additional diagrams showing the steps in experiments would aid understanding. This would help highlight the difference between the experiments in Sections 3 and 4, where (from my understanding) the former uses existing candidates to evaluate discriminative understanding, and the latter uses generated outputs. Showing exactly where in the experimental process generative outputs vs existing data are used would be informative. Figure 1 doesn't quite capture the difference between the two cases in my opinion.

Author response:
We have added a diagram (Figure 8) to provide a more detailed explanation of the full process, thank you for bringing up this potential point of confusion! We are happy to elaborate on this.

Reviewer comment (Presentation):
For the discriminative vs generative subplot in Figure 2, it is difficult to see all of the blue points for GPT4. Adjusting the plot to use alpha values would allow these points to be seen and strengthen the claims supported by the plot. Similar for Figure 8 in the appendix.

Author response:
We have adjusted the alpha values in the plot in Figure 2, and have also slightly reduced marker size so that the full set of data points is more visible. We are happy to make further changes as the reviewer requests.

Reviewer comment (Evaluation):
A handful of concrete outputs would provide further contextualisation for the kind of errors the models are making, as well as give further clarity to the structure of experiments. Figure 1 provides some examples, but additional outputs would be beneficial. Figures 12 to 15 help with this, but the questions/outputs are the same as used in Figure 1.

Author response:
We have added a set of qualitative examples to the appendix (see Appendix C.3) demonstrating the inability of language models to understand their own generations (interrogative), and general errors of vision models (both interrogative and selective understanding).

Reviewer comment (Evaluation):
While the discussion touches on possible explanations for the paradox, it does not mention ways it could be mitigated.

Author response:
We have added ideas for possible mitigations to our discussion section. In short, we suggest alternative optimization objectives, reduction of model memorization, and broad incentivization of understanding capabilities as 3 paths forward, inspired by the causes that we propose. Understanding the cause of this phenomenon will be necessary to find a mitigation strategy.

Reviewer comment (Novelty):
The insights presented in this paper may not be particularly surprising to the research community.

Author response:
### **“insights presented in this paper may not be particularly surprising”**
While we agree that our work matches the intuition of many researchers, much work recently relies on the understanding ability of generative models (e.g. model-based NLG evaluation [1]), or even makes this idea explicit. For instance, recent highly-cited work on GPT-4 states that “one of the key aspects of GPT-4’s intelligence is its generality, the ability to seemingly understand and connect any topic” [2]. We believe that work like ours is needed to provide more concrete, experimental proof of the ways in which models can lack understanding, and break from human intuitions.

Reviewer comment (Evaluation):
The paper employs two metrics, selective and interrogative, to assess the understanding capability of generative models. However, it doesn't delve into which metric more accurately reflects the models' comprehension, nor does it discuss how each metric contributes to specific aspects of understanding. Additionally, a comparative analysis between the selective and interrogative abilities of the generative models is missing.

Author response:
### **“employs two metrics, selective and interrogative … doesn't delve into which metric more accurately reflects the models' comprehension”**
We agree that a single measure of understanding would simplify analysis. However, understanding is complex and has multiple distinct pieces, as demonstrated by the diverse ways that humans are tested for understanding (e.g. multiple choice questioning, long-answer, oral examination). Selective and interrogative evaluation in our experiments attempt to capture two core aspects of understanding but future work will need to explore more aspects, rather than collapsing onto one. We agree that a better description of what each evaluation is testing will be useful, and we have elaborated on this in the paper text (Section 2.1, definitions). In short, selective understanding concerns the ability to recognize correct answers, while interrogative concerns the ability to explain correct answers. 

### **“... comparative analysis of the generative models' selective and interrogative capabilities, and provide explanations for the observed results?”**
We study a different set of tasks for the two forms of understanding, especially in the language domain. We largely evaluate selective understanding on short-answer questions, where it is feasible to judge the correctness of an entire response (for human annotators), while we judge interrogative understanding on longer texts, in which it is more meaningful to understand the possibly complex information contained in the generation of the model. However, there are 3 tasks for which these evaluations overlap (XSUM, Mutual+, and HellaSwag) and so we have **added a comparison of these tasks to the Appendix** (Figure 14). The two forms of understanding seem to correlate, likely related to the relative difficulty of understanding the underlying task and stylistic aspects of the data. Understanding this relation warrants further study.

Reviewer comment (Experiments):
Did you collect any data which would reveal whether the participants can themselves figure out if the models don't have understanding? That is, if you let participants interact (perhaps on a task where they need to collaborate) with the model (without telling them that it is a model), what percentage will complain that the model does not understand what it is outputting?

Author response:
While we don’t collect a large dataset of such examples, we have added our own interactions with GPT-4 to the appendix (C.3). Particularly, we include examples of simple questions which ask the model about its own generated stories, and the (often incorrect) answers that it outputs, to provide explicit examples of what the authors judge to be qualitative examples of misunderstanding.