N orm N et: Normalize Noun Phrases for More Robust NLP (2023)

Abstract: A critical limitation of deep NLP models is their over-fitting over spurious features. Previous work has proposed several approaches to debunk such features and reduce their impact on the learned models. In this work, a normalization strategy is proposed to eliminate the false features caused by the textual surfaces of noun phrases. The motivation for this strategy is that noun phrases often play the role of slots in textual expressions and their exact forms are often not that important for performing the final task. As an intuitive example, consider the expression ” x \text{ like eating } y ". There are a huge number of suitable instantiations for x and y in the locale. However, humans can already infer the sentiment polarity of x toward y without knowing their exact forms.Based on this intuition, we introduce NormNet, a pretrained language model based network, to implement the normalization strategy. NormNet learns to replace as many noun phrases in the input sentence as possible with pre-defined base forms. The output of NormNet is then fed as input to a prompt-based learning model to perform label prediction. To evaluate the effectiveness of our strategy, we conducted experimental studies on several tasks, including aspect sentiment classification (ASC), semantic text similarity (STS), and natural language inference (NLI). The experimental results confirm the effectiveness of our strategy.

## introduction deep learning has proven quite effective in many nlp tasks @xcite @xcite . however, despite their great success and various nlp applications they power, critical limitations persist. a typical issue is their tendency to learn spurious features instead of the true signals of the task @xcite @xcite . this often leads to corrosive outcomes, from degraded performance on data in which the features no longer present @xcite , to pernicious biases in model decisions @xcite , and to overall reduced trust in technology @xcite . this work proposes to address the spurious features caused by the textual surfaces of noun phrases. here, we mainly consider noun phrases because they are often highly variable and replacing their forms would not make the sentence unreasonable. as a motivating example, consider the task to identify the sentiment polarity of i toward apples based on the textual expression i like eating apples, which belongs to the positive class. here, "i" and "apples" can be changed to many other forms (e.g., "i" → "they", "many people", etc.) with the resulting sentences still reasonable. if the model is trained on such an example, it may over-fit the spurious correlations between the positive class and i or apples. these spurious correlations will result in overfitting, degrading the generalization performance and interpretability of the learned model. such a problem can be mitigated by preprocessing the original expression into x like eating y before feeding it as input to the learning model and reformulating the task to predict the sentiment of x toward y, where x and y denote two variables. in this way, the processed expression, together with its label, captures abstract knowledge independent of the specific forms of x and y-x is positive toward y no matter x is i or they and y is apples or bananas. in addition, such a pre-processing can make the learned model more interpretable and facilitate symbolic learning. with this consideration, we propose the idea of sentence normalization, which aims to replace as many noun phrases in the input sentence as possible with specifically designed base forms. figure 1 shows a normalized input sentence for an intuitive understanding. to implement the idea, we introduce a pretrained language model (plm) based network, normnet. given an input sentence, normnet will first identify noun phrases in the sentence. then, it applies a plm to evaluate the variability of every noun phrase conditional on its context. phrases with high variability will be normalized to a base form, ranging from "a" to "z". the resulting sentence will then be fed as input to the learning model to perform label prediction. we tested the effectiveness of normnet on three typical nlp tasks, i.e., aspect sentiment classification (asc), semantic text similarity (sts), and natural language inference (nli). the experimental results show that our normalization strategy can improve both the models' in-domain and cross-domain performance. the contribution of this work is three-fold: ## methodology figure 1 shows the general architecture of the proposed method. it adopts a prompt-tuning-based learning model to perform the end tasks. and compared with the traditional prompt-tuning-based method, it additionally introduces a normnet to pre-process the input sentence, which replaces some noun phrases in the input sentence with special tokens (we use "a"-"z" in this work). the resulting sentence is then fed as input to the prompt-tuning-based learning model to perform model training and inference. in the following, we illustrate the detail of the prompt-tuning-based learning model and normnet, respectively. ## conclusion this work proposes a normalization strategy to overcome the spurious features caused by noun phrase surfaces. experimental studies on aspect sentiment classification (asc), semantic text similarity (sts), and natural language inference (nli) show that the proposed strategy can improve both models' in-domain and crossdomain performance. a potential extension of this work is extending the strategy to other types of phrases.