How do Words Contribute to Sentence Semantics? Revisiting Sentence Embeddings with a Perturbation Method (2023)

Abstract: Understanding sentence semantics requires an interpretation of the main information from a concrete context. To investigate how individual word contributes to sentence semantics, we propose a perturbation method for unsupervised semantic analysis. We next re-examine SOTA sentence embedding models’ ability to capture the main semantics of a sentence by developing a new evaluation metric to adapt sentence compression datasets for automatic evaluation. Results on three datasets show that unsupervised discourse relation recognition can serve as a general inference task that can more effectively aggregate information to essential contents than several SOTA unsupervised sentence embedding models.

## introduction humans are usually able to understand sentence meaning based on a complex cognitive processcomposition of words @xcite . as main carriers of information, words generally have different levels of contribution to the final sentence semantics. for example, underlined words in "the city of austin is considering extending downtown parking meter hours to the weekends and later during the week" convey the most important information of this sentence. therefore, determining the primary semantics (or main meaning) of a sentence and estimating how sentence semantics distributes to individual words play critical roles in understanding sentence compositionality. recently, with the help of large pretrained transformers, sentence representation learning has achieved great success in downstream nlp tasks @xcite . however, most work either heavily relies on human annotation such as natural language inference (nli) data (williams et al., 1 code and data are available at https://github.com/ wenlinyao/eacl23-primarysemantics 2018; @xcite to do fine-tuning @xcite or adopt unsupervised contrastive learning to learn sentence embeddings @xcite . they neglect one critical property of an effective sentence embedding model that essential contents should contribute to sentence semantics more than non-essential contents when encoding a sentence. we observe that primary semantics can be acquired by learning to predict discourse relations because sentence primary semantics usually needs to support the logical relations at the discourse level. it is motivated by the distinction between asserted and projected content in semantic theory @xcite @xcite . asserted content is intended to be presented for discussion and information exchange, whereas projected content represents background information that is not under discussion. thus, asserted content should carry more weight in sentence representations than projected content, because it is at issue in the context by the speaker's intention. based on this observation, we apply 36 explicit discourse connectives to four big corpora and extract 9.8m sentence pairs. acquired sentence pairs are then used to train a universal sentence encoder. next, by perturbing the trained model, we directly estimate the contributions (importance) from individual words to the final sentence semantics. our assumption is that the importance of a word in a sentence is proportional to how much the new sentence representation drifts from the original sentence representation if we mask that word. figure 1 shows the overview of our approach. to overcome the lack of evaluation data and quantitatively evaluate a sentence encoder's sensitivity to key information of a sentence, we design a new evaluation metric -important information gain, which measures model's ability to concentrate on the important words instead of randomly se-s1 s2 ## discourse relations the city of austin is considering extending downtown parking meter hours to the weekends and later during the week. lected words. experiments on three datasets show that our model outperforms previous unsupervised and supervised sentence embedding models. our analysis also demonstrates our model is less biased than the model trained with human-annotated data. ## related work dissent @xcite also uses discourse prediction to train sentence embeddings, but their goal is to achieve sota fine-tuning results on senteval tasks @xcite and the pdtb task instead of investigating primary semantics. our work is closely related to sentence compression @xcite and extractive text summarization @xcite that mainly focus on extracting the salient text spans in an endto-end manner. in contrast, we aim to estimate the semantic saliency distribution to better understand sentence compositionality. perturbation methods have been also used in data augmentation (das and sarkar, 2019), robustness analysis @xcite , and textual adversarial attack @xcite . we are the first to use the perturbation method to estimate sentence semantics distribution. ## estimating sentence semantics distribution via perturbation to estimate the semantic distribution of the input sentence @xmath0, we mask each word w i individually to construct a new sentence and apply the sentence embedding model (section 3.2) to calculate the new representation. next, we calculate the cosine distance between the new and the original representation. the distance can quantitatively tell us how much the new sentence (after masking one word) semantically drift from the original sentence, which indicates the contribution of that word to the sentence semantics. 4 finally, we normalize the distances by their summation so that the importance scores of all words sum to 1. ## conclusion in this paper, we have introduced a perturbation method for estimating sentence semantic distribution and designed a new metric to achieve automatic evaluation. we find discourse relation recognition can serve as a general inference task to train an unsupervised sentence embedding model that estimates such distribution meaningfully than several sota sentence embedding models. to benefit from large-scale training data, we train the discourse relation recognizer on data that are automatically generated by matching explicit connectives. even we only select explicit connectives that have one dominant discourse relation, some of them may still reflect a different relation in some contexts. for example, connective in other words indicates a "equivalence" relation most of the time, but sometimes it can also indicate a "generalization" relation. in this regard, our method shares the same limitations as the broad class of weakly supervised methods where training data are automatically generated. considering discourse parser is not the main focus of this paper, we leave how to generate cleaner discourse relation data and train a better discourse relation recognizer for future work.