Inferring symmetry in natural language (2020)

Abstract: We present a methodological framework for inferring symmetry of verb predicates in natural language. Empirical work on predicate symmetry has taken two main approaches. The feature-based approach focuses on linguistic features pertaining to symmetry. The context-based approach denies the existence of absolute symmetry but instead argues that such inference is context dependent. We develop methods that formalize these approaches and evaluate them against a novel symmetry inference sentence (SIS) dataset comprised of 400 naturalistic usages of literature-informed verbs spanning the spectrum of symmetry-asymmetry. Our results show that a hybrid transfer learning model that integrates linguistic features with contextualized language models most faithfully predicts the empirical data. Our work integrates existing approaches to symmetry in natural language and suggests how symmetry inference can improve systematicity in state-of-the-art language models.

## introduction symmetry helps one make systematic inference about relations in the world and is a fundamental property of natural language @xcite . a symmetrical predicate describes a reciprocal relation and collective participation between entities. in logical terms, given a symmetrical relation r, for all entities x, y: r(x, y) ⇐⇒ r(y, x). for instance, knowing john met mark one can systematically infer that mark met john, and vice versa. here meet is perceived as symmetrical, because a meeting is implicitly reciprocal and occurring collectively with both participants. conversely, gab kissed anna does not imply that anna kissed gab. here kiss is perceived as asymmetrical. however, * equal contribution. empirical studies from psycholinguistics have taken two main approaches to sentence-level symmetry: 1) a feature-based approach @xcite ; and 2) a contextbased approach @xcite . gleitman and colleagues, after obtaining predicate-level symmetry ratings, had participants assess the degree of discrepancy in meaning between a sentence and its reversed counterpart (where the positions of the entities are switched). the logic behind this approach to symmetry inference can be demonstrated in the pair of sentences, gab kissed anna and anna kissed gab, which do not have the same meaning. the difference score for the pair would be high, rendering kiss asymmetrical. the feature-based approach. @xcite found that sentence interpretation heavily depends on its syntactic structure and the lexical-semantic properties of the predicate and entities involved. for example, any predicate can appear symmetrical in a non-directional sentence format (where the entities are placed on one side of the verb, e.g., anna and gab kissed). gleitman and colleagues' work suggests that symmetric inference is grounded in linguistic features. however, their findings were based purely on empirical investigation, and no formal approach has been developed to model symmetric inference in language and evaluated comprehensively against data. the feature-based approach is insufficient to capture all possible real-world relations between entities. as @xcite noted, context becomes relevant to determine degree of predicate symmetry such as in the following pair of sen-tences: my sister met meryl streep (judged asymmetric) and john met mark (judged symmetric), which indicates that sentences similar in lexical and syntactic features do not always yield the same symmetry judgment. the context-based approach. focusing on the symmetric predicate similar instead of verb predicates in their generality, @xcite elaborated further on the role of context. first they examined the nature of entities. they deliberately chose entities that are conceptually close in prominence (e.g. austria, west germany) or much different (e.g. england, jordan), and found that symmetric inference can depend on one's world knowledge. in a related experiment, they showed that inference involving the predicate similar can be manipulated with contextual information. for example, hungary was judged to be more similar to austria than sweden or norway, but sweden was judged to be more similar to austria than sovietaligned hungary or soviet-aligned poland. this approach highlights the need to formalize a contextual approach to symmetry and evaluate how it interacts and fairs with the feature approach. our view is that both linguistic features and contextual knowledge matter in symmetry judgment, and integrating the two approaches described should facilitate systematic inference @xcite in models of natural language processing (nlp). we develop a naturalistic sentence dataset for symmetry inference of literature-informed verbs spanning symmetry-asymmetry that is underrepresented in existing natural language inference datasets such as snli @xcite . we show that whereas a contextualized language model helps operationalize a context-based approach to symmetry inference, it is critically lacking in learning linguistic features pertaining to symmetry. we propose a hybrid transfer learning model that integrates linguistic features with context and demonstrate its efficacy in improving systematic inference of contextual language models. ## methodology we formulate symmetry inference as a regressionbased representational learning problem. we explore a set of representational schemes that capture the existing approaches to symmetry based on features and contextualization, as well as a hybrid model that integrates these representations. as illustrated in figure 1 , an encoder takes in an input sentence x (i) and provides hidden representation h (i) = f (x (i) ) with information pertaining to symmetry inference. an additional regression layer then takes this hidden representation and computes a continuous score that quantifies the degree of symmetry for sentence x i . to adopt a parsimonious approach, we use a simple linear layer during the learning process, the model is presented with a ground-truth symmetry score y (i) (from human annotation explained later) for each sentence. the objective function of the model follows a standard regularized mean-squared loss: we keep the regularization hyperparameter α = 1.0 over all experiments. the parameterized modules of the model are trained to minimize the loss l mse via back-propagation. to examine the importance of feature-based and contextual information, we consider four types of encoders with varying model architectures. feature model. for each input sentence, the feature-based encoder first performs dependency parsing, and then extracts a sequence of syntactically-induced, categorical feature variables h (i) j indicating the existence of certain linguistic patterns. we choose features that were 1) shown empirically to be associated with sentencelevel symmetry according to psycholinguistic literature; and 2) obtainable via an automatic featureextraction pipeline. following classic empirical studies of symmetry @xcite , our model will infer symmetry from pre-defined linguistic features (described in section 4) and a small amount of contextual information from these features (e.g., animacy). static word embedding model. as a baseline to the contextualized language models, we consider two static embedding encoders, word2vec and glove @xcite , based on pre-trained distributed word embeddings h (i) j for each token in x (i) , and we then compute the mean vector as hidden representation: as static word embeddings have been shown good at encoding rich lexical knowledge @xcite , we expect the simple average representation should capture useful semantic cues beyond the scope of the feature model. deep contextualized model. to better model variation in naturalistic context, we use the bert transformer encoder @xcite to compute a contextualized representation for each input sentence. in particular, we follow the standard approach of applying bert by considering a special classification token [cls] at the beginning, and a separation token [sep] at the end. an encoded vector h (i) j is then computed for each token, and the last hidden representation of [sep] is taken as the sentence embedding h (i) . as each h (i) j is a nonlinear function of all input tokens, we expect the contextualized encoders to be superior than static embedding models in extracting more context-sensitive information from input. the resulting representation is fed into the regression layer to compute a symmetry score. during training, the parameters of the bert encoder can be either fixed or updated, and we tested in both settings for a comprehensive evaluation. hybrid transfer learning model. the richness of knowledge encoded in contextualized models is helpful for many inference tasks, but it might not contain information directly pertaining to predicate symmetry (as made explicit in the feature model). we therefore propose a two-stage transfer learning model to coerce the contextualized model to attend to the symmetry-relevant features, as illustrated in figure 1 . in stage 1, the bert encoder is connected with a classification layer with weights w clf , and trained to predict the linguistic features by minimizing the negative log-likelihood loss: here k denotes the total number of features for prediction, and σ(•) the sigmoid function. after convergence or in stage 2, the feature-informed encoder is then topped by a regression layer to produce symmetry scores. this approach incorporates featural knowledge into the existing contextualized model from stage 1 and transferably applies that knowledge to inferring symmetry in stage 2. ## symmetry inference sentence dataset (sis) we collect data in two steps: (1) select seed verbs that are traditionally defined as (a)symmetrical and sentences that contain these verbs, and (2) obtain human symmetry ratings for each sentence based on its perceived similarity with its reversed counterpart (e.g., switched entities) in an online survey. seed verbs. we focused on verbs because they are the most extensively studied word class in symmetry and have many established features. we worked with 40 common verbs from the literature, divided equally into symmetric and asymmetric categories. ## model evaluation and results we report findings in four steps. first, as a replication we assess the correlation between sis dataset sentence ratings and verb-level symmetry scores reported by @xcite . second, we evaluate model predictions for sentence-level similarity ratings. third, we perform an error analysis and interpret the findings. fourth, we offer a focused analysis of model systematicity. a sentence is symmetrical if all participants are simultaneously on the giving and receiving end of the action described. if you switch the position of the participants, the overall meaning of the sentence won't change. in this task, you will be given a pair of sentences. the first describes at least two participants and describes their relationship. the second sentence conveys the same information as the first, except the positions of the participants in the sentence are switched. your task is to rate how alike in meaning the given two sentences are from a scale of 1-5, where 1 means the sentences do mean the same and 5 means do not mean the same. given the following pair of sentences: rate how alike in meaning the given two sentences are from a scale of 1-5, where 1 means the sentences do mean the same, and 5 means the sentences do not mean the same. ## conclusion we present to our knowledge the first formal framework for modelling sentence-level predicate symmetry and demonstrate that automated inference of verb symmetry is possible in natural context. contributing the symmetry inference sentence dataset, we show how existing approaches to symmetry, based on linguistic features and contextualization, are by themselves insufficient to explain sentencelevel symmetry judgment, but a hybrid approach improves systematic symmetry inference in stateof-the-art language models. future work may explore symmetry in other word classes (e.g., nouns and adjectives) and languages other than english.