What does the Failure to Reason with “Respectively” in Zero/Few-Shot Settings Tell Us about Language Models? (2023)

Abstract: Humans can effortlessly understand the coordinate structure of sentences such as “Niels Bohr and Kurt Cobain were born in Copenhagen and Seattle, *respectively*”. In the context of natural language inference (NLI), we examine how language models (LMs) reason with respective readings (Gawron and Kehler, 2004) from two perspectives: syntactic-semantic and commonsense-world knowledge. We propose a controlled synthetic dataset WikiResNLI and a naturally occurring dataset NatResNLI to encompass various explicit and implicit realizations of “respectively”. We show that fine-tuned NLI models struggle with understanding such readings without explicit supervision. While few-shot learning is easy in the presence of explicit cues, longer training is required when the reading is evoked implicitly, leaving models to rely on common sense inferences. Furthermore, our fine-grained analysis indicates models fail to generalize across different constructions. To conclude, we demonstrate that LMs still lag behind humans in generalizing to the long tail of linguistic constructions.

## introduction transformer-based language models (lms) @xcite @xcite induce useful representations for a wide range of natural language understanding (nlu) tasks, including natural language inference (nli; @xcite , especially in in zeroshot or few-shot settings. to what extent this usefulness results from memorization, generalization or the ability of lms to draw common sense inferences remains an open question. to approach it, the linguistic phenomenon of respective readings @xcite serves as an excellent probe. this phenomenon has so far been underexplored in nlp, even though it has been studied extensively in linguistic semantics @xcite @xcite bohr and cobain were born in copenhagen and seattle respectively . bohr and cobain were born in copenhagen and seattle . bohr was born in copenhagen . cobain was born in seattle . figure 1 : an example of explicit (top, evoked by "respectively") and implicit (middle, with no overt marker) respecitve readings. humans can infer that both sentences have the same "cross-serial" meaning (bottom) by relying on commonsense knowledge (that a person is only born in one location) and world knowledge (that copenhagen and seattle are mutually exclusive). @xcite . in english, "respectively" is a rare word foot_0 used to establish a one-to-one mapping between two sets of participants and to distribute predicates over sets @xcite . for example, in figure 1 , the first conjunct in the subject corresponds to the first conjunct in the object and the second conjunct in the subject corresponds to the second conjunct in the object. the respective relation is bijective and respects the relative order of the elements of two different coordinate expressions; it is, in other words, cross-serial. "respectively" can have different syntactic or semantic properties depending on the context, e.g., as a conjunction or adverb. in this paper, we investigate how lms reason with respective readings. we propose two datasets, wikiresnli (a controlled synthetic dataset) and natresnli (a naturally occurring dataset) to cover various explicit and implicit realizations of "respectively". our research questions are: we experiment with state-of-the-art lms and analyze the results to gain insights into the limitations of current models and potential directions for future research. we show that lms are able to generalize effectively in a few-shot learning scenario when the word "respectively" is present. however, when the reading is evoked implicitly, a greater number of training instances are necessary. lms require significantly more instances to generalize to naturally occurring datasets than humans. in conclusion, our study demonstrates that lms continue to exhibit a deficit in generalizability to infrequent linguistic constructions with limited coverage in their training data. ## respective readings respective readings are closely related to several types of readings instantiated by plurals and mass terms: distributive readings, collective readings and cumulative readings @xcite . distributive readings. these usually refer to the application of a predicate to the subsets of a set or group. as for sentence 1(a), it is equivalent to "john smiled and mary smiled". the reading is available because of the nature of the predicate is atomic @xcite , similar instances including "sing" and "sleep". distributive reading can be enforced with overt distributive markers, i.e., "every" and "each" @xcite . in example 1(b), we enforce the reading by adding "each" at the end of the sentence so as to rule out the reading "john and mary earn 200 dollars together". collective readings. these are the opposite of distributive readings in that the predicates apply to the whole plural entity instead of individuals. the quantifiers "all" and "most" instead of "every" and "each" are usually compatible with collective readings as in example 2(b) @xcite . ## experiments we begin our experiments with the datasets by addressing our first research question: question 1 can nli models reason with the coordinate structure in "respectively" construction in zero-shot settings? given the popularity of nli as a classification task to test lms' ability of language understanding, many works have proposed new models achieving state-of-the-art results on datasets such as snli @xcite , multinli (mnli; @xcite and anli @xcite . on the glue leaderboard, foot_5 the state-of-the-art models have surpassed 90% and 95% accuracy on mnli and qnli which are deemed as solved challenges. anli has been one of the most challenging tasks in recent years, and the latest models such as deberta-v3-large @xcite and palm 540b @xcite can achieve 64% and 67.9%, respectively. while many works use anli as a medium to exhibit the models' growing reasoning ability, few of them analyze in depth in which case it fails and at which stage it gets to learn certain linguistic abilities. ## related work logical relations between two sentences are a core aspect of language understanding @xcite @xcite . to facilitate large-scale model evaluation, nlp researchers have developed manually labelled nli corpora, typically for 2/3-way classification @xcite @xcite . in recent years, researchers start to analyze the characteristics of these datasets, such as annotation artefacts @xcite , syntactic heuristics @xcite and adversarial collection process @xcite . in computational linguistics, distributive predication has been analyzed through means of dis-tributivity operators @xcite @xcite . and linguists have been working on extending first-order logical forms to include distributive and collective readings @xcite . @xcite present a recursive translation rule scheme to account for multi-level plurals. @xcite proposed a reasoner consisting of domain-dependent constraints and domainindependent axioms for collective and distributive ambiguity. @xcite described a simplified quantifier system to minimize distributive and collective ambiguities. respective readings have not yet been studied in modern nlp. relevant works include plural understanding, which has been studied as a coreference resolution task @xcite @xcite . @xcite proposed quantifier scope annotation in which plurals are annotated with distributive and collective readings at the constraint level. @xcite show that lms are better at reflexive anaphora tasks with distributive than collective constructions. ## conclusions the "respectively" construction is simple yet entails multiple levels of reasoning skills, including syntactic-semantic and commonsense-world knowledge. it is crucial that when an out-of-thebox model cannot reason over it, it should be able to learn with as few examples as possible. we proposed two datasets, wikiresnli (a controlled synthetic dataset) and natresnli (a naturally occurring dataset) to probe their ability to do so in zero-shot and few-shot settings. we find that explicit reasoning is easier to learn than implicit reasoning, and lms fail to generalize when common sense inference is needed. we confirm that diverse and complex training data are necessary to achieve human-level performance.