Looking inside Noun Compounds: Unsupervised Prepositional and Free Paraphrasing (2020)

Abstract: A noun compound is a sequence of contiguous nouns that acts as a single noun, although the predicate denoting the semantic relation between its components is dropped. Noun Compound Interpretation is the task of uncovering the relation, in the form of a preposition or a free paraphrase. Prepositional paraphrasing refers to the use of preposition to explain the semantic relation, whereas free paraphrasing refers to invoking an appropriate predicate denoting the semantic relation. In this paper, we propose an unsupervised methodology for these two types of paraphrasing. We use pre-trained contextualized language models to uncover the ‘missing’ words (preposition or predicate). These language models are usually trained to uncover the missing word/words in a given input sentence. Our approach uses templates to prepare the input sequence for the language model. The template uses a special token to indicate the missing predicate. As the model has already been pre-trained to uncover a missing word (or a sequence of words), we exploit it to predict missing words for the input sequence. Our experiments using four datasets show that our unsupervised approach (a) performs comparably to supervised approaches for prepositional paraphrasing, and (b) outperforms supervised approaches for free paraphrasing. Paraphrasing (prepositional or free) using our unsupervised approach is potentially helpful for NLP tasks like machine translation and information extraction.

## introduction noun compounds-contiguous sequences of nounsare common linguistic constructs. a compound is called compositional if the meaning of the compounds can be derived from the meaning of its components. the component nouns are related through a semantic relation that is constituents dependent. for instance, 'student protest' and 'university protest' are protests. however, the student(s) are ag e n t (doer of an event), whereas university is lo c a t i o n of the protest. the task of identifying such relations between the components of a noun compound is called noun compound interpretation @xcite . such interpretation can help a wide variety of nlp tasks, like machine translation @xcite @xcite , question answering @xcite , text entailment @xcite , and semantic parsing @xcite . for instance, to translate the english noun compound 'cow milk' to hindi, a machine translation system needs to generate the postposition ka (of ) in addition to translating the individual nouns. the correct translation of the compound is 'gāya ka dūdha' (lit. 'cow -of milk'; 'milk of cow'). without understanding the underlying relation, a machine translation system might fail. interpretation via abstract labels (representing semantic relations) is popular in the literature. given a noun compound, the task is to assign an abstract label from a predefined set, e.g., 'student protest': pr o t e s t e r. past work has proposed a wide variety of inventories for semantic relations @xcite @xcite @xcite @xcite @xcite @xcite ; however, there is no community agreed standard inventory. interpretation can be done via paraphrasing as well. here, one can use extract words (along with component nouns) to paraphrase a noun compound, e.g., 'student protest': 'protest by student', 'protest held by students', etc. the paraphrase reveals the underlying relation. a simpler version of paraphrasing, also known as prepositional paraphras-ing, uses only a preposition to paraphrase a noun compound. a set of 8 prepositions by @xcite is widely used for prepositional paraphrasing, and the task is to identify a preposition which can paraphrase the given noun compound. another way of paraphrasing, also known as free paraphrasing, allows any word(s) for paraphrasing. one can use multiple paraphrases to represent the semantic relation collectively. this is a more complex and challenging task. in this paper, we show how contextualized language models can be used for unsupervised paraphrasing of noun compounds. specifically, we propose two unsupervised approaches for paraphrasing of noun compounds: one for prepositional paraphrasing and another for free paraphrasing. we use contextualized language models and feed template to generate possible paraphrases. our results show that the proposed unsupervised approach gives results comparable to supervised systems for prepositional paraphrasing and outperforms supervised approaches for free paraphrasing. ## background with the introduction of the transformer networks @xcite , pre-trained language models have become a key component in advancing the state-of-the-art for many nlp tasks. bert @xcite , a transformer-based encoder, has advanced the state-of-the-art for various nlp tasks. for pre-training, bert uses two self-supervised objectives: next sentence prediction (nsp), and masked language model (mlm). for nsp, bert is trained to predict whether the second text segment follows the first text segment. this is hypothesized to improve bert's understanding of the relationship between two text sentences. for mlm, given the input token sequence, a portion of tokens are replaced by a special symbol @xcite , and the model is trained to recover the original tokens from the corrupted version. this allows representations to be conditioned on the left and right context. note that bert predicts plausible words for each [mask] token independently. the success of bert inspired many variants such as training on domain/application specific corpus @xcite @xcite @xcite , training on monolingual corpora @xcite , incorporating knowledge graph in the input @xcite , etc. bert requires a task-specific output layer. so, one needs to modify bert's architecture to adapt it for a new task. recent text-to-text models, such as t5 @xcite and bart @xcite , use encoder-decoder architectures which share output layer for all tasks effectively eliminating the requirement to modify architecture for a new task. these models convert all nlp problems into a text-to-text format, i.e., input and output for any nlp task (including classification task) are sequences. a text-to-text model can generate a variable length span for a single masked token because of encoder-decoder architecture. we use the t5 model to generate free paraphrases for noun compounds. ## our approach our approach benefits from the mlm objective of contextualized language models. we use templates to rephrase a noun compound. the template uses a mask-token foot_1 to indicate the missing word(s). we feed the phrase to a pre-trained model and ask it to predict the missing word(s), which can replace the mask-token. we use bert and roberta to uncover a single word and t5 to uncover variablelength sequences. we use the transformers library @xcite .8) foot_2 for the experiments. ## experiments in this section, we discuss the datasets, evaluation metrics used in our experiments. ## conclusion and future work a noun compound can be paraphrased using the components nouns along with the predicate. the predicate indicates the semantic relation between the component nouns. we use a simple pattern for generating the predicate using a fixed pattern, i.e., w 1 w 2 → 'w 2 <extra-@xmath0. one can exploit recent pre-trained language models to uncover the connecting extra-words for paraphrasing. these language models have been trained with one of the training objective being uncovering the missing words. in this paper, we propose an approach that performs noun compound paraphrasing by using these pre-trained models to uncover the missing extra words. our approach uses these pre-trained models as is without any task-specific training or fine-tuning. our approach is tested for both prepositional paraphrasing and free paraphrasing of noun compounds on various datasets. with simple patterns, our approach gives results closer to supervised systems for prepositional paraphrasing and outperforms supervised systems for free paraphrasing. in the future, we will investigate whether finetuning the language models would lead to better paraphrasing. we will also study the setting where context is crucial for correct paraphrasing. we believe that given this approach is language-agnostic, it should work for other languages too. so we will also verify this belief holds for other languages.