Ellipsis-Dependent Reasoning: a New Challenge for Large Language Models (2023)

Abstract: We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.

## introduction ellipsis is a fundamental feature of human language, occurring in all registers, where parts of sentences are omitted, although the missing parts are essential for understanding the meaning. the following is an example of verb phrase ellipsis (vpe) @xcite : with such examples, we can test understanding of ellipsis by targeting the ellipsis phrase with a simple yes/no question: if a system answers the question incorrectly for (1), the ellipsis example, but answers correctly for (2), the non-elliptical counterpart of (1), we can ascribe the result specifically to the challenge of ellipsis, since the examples are otherwise identical. as with pronominal anaphora and other discourse processes, there is great flexibility in the way ellipsis can occur in discourse. example (1) involves two simple adjacent sentences. it is also possible for ellipsis or the antecedent to occur in embedded clauses. furthermore, ellipsis can occur either before or after the antecedent. finally, an arbitrary amount of material can intervene between the antecedent and the ellipsis occurrence. in this paper, we propose the challenge of ellipsis-dependent reasoning. this challenge consists of examples involving an ellipsis clause, the target. each ellipsis example is paired with its nonelliptical counterpart, where the target clause is overt rather than elliptical. we then pose a question whose answer is dependent on the target clause. a key aspect of the challenge is that ellipsis occurrences are possible in a variety of diverse structural configurations. we test a series of gpt-3 models (gpt) on several such ellipsis structures. ## related work there is a large literature concerning the probing of language models from a variety of perspectives. furthermore, there has been substantial work specifically addressing ellipsis in nlp. in this paper, we are proposing the challenge of ellipsisdependent reasoning. this proposal builds on various strands of prior research; below we consider some particularly relevant aspects of this literature. ## data there is a great deal of variety in the structural configurations in which ellipsis can occur. in tables 1 and 2 we define structures for ellipsis and antecedent occurrences. in all structures, there is an ellipsis occurrence, and the question targets the ellipsis occurrence. furthermore, each ellipsis example is paired with a non-ellipsis version. here the vpe is in a subordinate clause. finally, we have a two actions structure; that is, two ellipsis occurrences each with their respective antecedent vps. we have two versions: one in which the target question has a correct answer of "yes", shown in ## test for each instantiation of a given structure, we produce paired ellipsis and non-ellipsis examples, with an associated yes/no question. we randomly select 1000 examples for each structure, including 500 ellipsis examples and 500 examples which are their non-elliptical counterparts. each example is presented to the system, preceded by the text, "please give a yes or no answer:". we test five gpt-3 models on these structures: davinci-003, davinci-002, figure 1 gives the accuracy for ellipsis and nonellipsis, for each of the five models. we have set up the test examples so that an ellipsis example is paired with a non-ellipsis example that is otherwise identical. because of this, we claim that the difference in accuracy of the non-ellipsis case vs. the ellipsis case provides a measurement of the difficulty specifically posed by ellipsis. for all but the least powerful model, ada, the non-ellipsis accuracy is substantially higher than ellipsis accuracy, supporting the hypothesis that ellipsis-dependent reasoning presents a difficult challenge for these models. while the ada model actually performs somewhat better for ellipsis than non-ellipsis, this is not because the ada model does well with ellipsis cases; rather, the model has great difficulty with both the ellipsis and non-ellipsis cases, and is close to a random guessing baseline of .50. in figures 2 through 6, we present results for each model. we show the accuracy for each structure, for both the ellipsis version and the non-ellipsis version. consider the most powerful models, davinci-003 and davinci-002. in figures 2 and 3, we can see that ellipsis is not difficult in the first two structures: 2sent (separate sentence) and 1sent (conjoined sentence). here the accuracy is nearly perfect for figure 2: ellipsis accuracy -davinci-003 (1000 examples per structure.) the ellipsis as well as the non-ellipsis condition. however, in all the other structures, there is a large divergence in accuracy between ellipsis and nonellipsis, for both the davinci-003 and davinci-002 models. subordination for either antecedent or ellipsis is quite challenging, with accuracies ranging from 48.8 to 85.8. the backwards and two actions structures are even more difficult for ellipsis. ## analysis for the two most powerful models, it is clear that ellipsis poses a difficult challenge, except in the two simplest ellipsis structures. for the less powerful models, the picture is mixed. for these models, the non-ellipsis examples are themselves a difficult challenge, so we are not able to observe the specific difficulties posed by ellipsis. as we can see in figure 1 , the davinci-002 model performs somewhat better overall than davincifigure 4: ellipsis accuracy -curie-001 (1000 examples per structure.) there are striking differences in the distribution of errors. for both the davinci-003 and davinci-002 models, errors are nearly always false negatives -that is, incorrect "no" answers. there are virtually no false positives, either for the ellipsis case or nonellipsis case. for the other three models, there are many errors of each type, with a much higher ratio of false positives. ## conclusion most of the current rapid progress in nlp is due to pre-trained large language models. gpt-3 is an impressive publicly available collection of such figure 6: ellipsis accuracy -ada-001 (1000 examples per structure.) in many ways, gpt-3 appears to understand the texts that it processes, often being able to answer questions that appear to rely on sophisticated reasoning. however, the challenge of ellipsisdependent reasoning provides evidence that gpt-3 is not able to understand in anything like the way humans do. figure 8: non-ellipsis case -errors by model. false positives (incorrect "yes") vs. false negatives (incorrect "no")