“Covid vaccine is against Covid but O xford vaccine is made at O xford!” Semantic Interpretation of Proper Noun Compounds (2022)

Abstract: Proper noun compounds, e.g., “Covid vaccine”, convey information in a succinct manner (a “Covid vaccine” is a “vaccine that immunizes against the Covid disease”). These are commonly used in short-form domains, such as news headlines, but are largely ignored in information-seeking applications. To address this limitation, we release a new manually annotated dataset, ProNCI, consisting of 22.5K proper noun compounds along with their free-form semantic interpretations. ProNCI is 60 times larger than prior noun compound datasets and also includes non-compositional examples, which have not been previously explored. We experiment with various neural models for automatically generating the semantic interpretations from proper noun compounds, ranging from few-shot prompting to supervised learning, with varying degrees of knowledge about the constituent nouns. We find that adding targeted knowledge, particularly about the common noun, results in performance gains of upto 2.8%. Finally, we integrate our model generated interpretations with an existing Open IE system and observe an 7.5% increase in yield at a precision of 85%. The dataset and code are available at https://github.com/dair-iitd/pronci.

## introduction proper noun compounds (pncs) @xcite 1 are grammatical constructions where a proper noun is followed by a common noun, for example: covid vaccines or buddhist monks. these often serve as a compact way to convey information about an already known entity, omitting predicates that are interpreted by the reader using surrounding context, common sense, and world knowledge. for example, a reader is likely to interpret that "buddhist monks" are "religious people who are buddhists". in other cases, pncs are used to identify specific entities, and do not provide additional 1 also referred to as proper noun modified compounds. information. for example, watergate scandal and kawasaki disease do not have any implicit relation between the proper and common noun as they refer to a specific instance of a scandal and a disease. thanks to their brevity, pncs are commonly used to shorten descriptions in space-constrained domains, such as news articles headlines @xcite . however, we find that prior work on compound noun interpretations only considered cases where the constituents are common nouns (e.g. baby oil), thus missing all of the information conveyed in proper noun compounds @xcite . to address this limitation in current systems, we begin by defining the task of pnc interpretation as two subsequent stages (section 3). the first stage requires identifying whether a given pnc is compositional or not, while the second stage is the generation of an interpretation, where applicable. in section 4, we present pronci, a crowdsourced dataset over wikipedia containing 22.5k proper noun compounds and their annotated semantic interpretations. candidates pncs are found using syntactic parsing, and are then presented to crowdworkers who are asked to interpret them. our annotation interface marks whether workers needed to read the full sentence, thus identifying pncs whose interpretation relies on context. we will make the pronci dataset publicly available to spur future research into pncs. in section 5, we develop two approaches for pnc interpretation: (1) a multi-task neural model that performs classification and sequence generation in two distinct stages and (2) a text-to-text approach, using a sequence-to-sequence model for both classification and generation. in addition, we experiment with different methods for injecting various sources of world knowledge, which seems crucial for the task, using external resources like wikipedia and wordnet @xcite , that type example ## related work noun compounds are commonly used in english language, constituting 3.9% of the tokens in the reuters corpus @xcite . they can be arbitrary length phrases, such as split air conditioner, but most prior work on interpreting noun compounds has primarily looked at two word noun compounds of the type noun-noun, where both are common nouns. to the best of our knowledge challenges in interpretation where the first word is a proper noun (i.e., proper noun compounds) have not been addressed, although their functional analysis and prevalence in certain domains have been studied in linguistics @xcite @xcite . we briefly summarise the various types of nouncompound interpretations in literature and discuss their uses in applications. ## problem definition interpretations of noun compounds are meant to expose the expressed implicit relation. free-form paraphrases as interpretations provide flexibility for expressing relations implied in noun compounds, overcoming the limitations associated with choosing from a fixed set of classes or templates at the cost of a possibly non-consolidated representation, i.e., where similar-meaning noun compounds are represented differently. hence, we define semantic interpretation of a pnc as a free-form paraphrase that exposes the implicit relation between the constituent nouns, if any relation exists, else identify it as @xcite ). ## pronci dataset to facilitate research in semantic understanding of proper noun compounds, we collect and release a supervised dataset as part of this work, which we call the pronci dataset. it contains 22,500 pncs and their semantic interpretations which are annotated by human workers hired from amazon mechanical turk (amt). the scale of the dataset is orders of magnitude greater than previously published free-paraphrase (common) noun compound datasets like semeval 2013 task 4 @xcite that have only considered 355 noun compounds. for handling the evaluation of generated interpretations where multiple correct answers are possible, prior datasets choose to annotate multiple interpretations for each noun compound (varying from 30-50). on the other hand, pronci dataset only contains one interpretation per noun compound as we choose to invest our annotation budget in breadth rather than depth, relying on recent advances in semantic text similarity (e.g. bleurt @xcite ) to help evaluate the generated interpretations. moreover, prior datasets consider noun compounds out of context, while pronci also contains the sentence in which the proper noun compound is used. providing this additional context helps to limit the ambiguity associated with multiple possible interpretations of the noun compound. for example, u.s. sanctions can mean either sanctions imposed by u.s. or sanctions imposed on u.s. the exact case can be determined based on the context in which it is used. "u.s. sanctions on iran have crippled the country", implies the former and "u.s. sanctions by iran..." implies the latter. to prepare the pronci dataset, we randomly sample sentences from wikipedia, and retain sentences which contain two-word proper noun compounds as identified by the spacy dependency parser @xcite . for every word, spacy identifies the root word along with the dependency tag. the "compound" dependency tag is used if the word and its root are part of a compound word. then the parts of speech of the first and second word of the compound are checked. if they are proper noun ("propn") and common noun ("noun") respectively, we identify it as a proper noun compound and include it. if any word pairs have been identified incorrectly as proper noun compounds, they are marked by annotators to indicate the absence of any relation. after the collection of proper noun compounds and corresponding sentences in which they appear, we posted hits on the amt platform for identification of relation between two words. the hits were accompanied by task instructions, summarized in to check the quality of annotation, we randomly sample 100 examples and find them to be correct 93% of the time. this represents an acceptable level, considering the difficulty of understanding certain compounds that need technical knowledge (aes key) or cultural background (abner characters), as well as the subjectiveness in determining non-compositionality. ## models the task of semantic interpretation of proper noun compounds involves generating valid paraphrases that explicate the relation in cases which are compositional. so a model designed for this task needs to first identify if the given noun compound is compositional ([cmp]) or not ([non-cmp]), and generate a paraphrase accordingly. we experiment with (1) supervised neural models, (2) adding external infor- knowledge example none buddhist monks sentence recent visitors to the campus include buddhist monks who installed an environmental artwork at lower pond. [sep] buddhist monks wordnet-nn buddhist meaning: buddhism is a widespread asian religion based on a series of original teachings attributed to gautama buddha. [sep] buddhist monks wiki-nnp monks meaning: a male religious living in a cloister and devoting himself to contemplation and prayer and work [sep] buddhist monks ner-nnp buddhist belongs to nationalities or religious groups [sep] buddhist monks supervised neural models: we use two types of supervised neural models: (1) a multi-task and in the unified generation model, we follow the recent advances in nlp where multiple tasks are posed in a common text-to-text format and are handled by a single seq2seq model like t5 @xcite . for this purpose, we pose the task as a simple string generation problem that outputs either the paraphrase itself in cases where it is interpretable or generates the string "proper noun compound is non-compositional" in the remaining cases. we refer to this model as unigen. external information: since the task of interpretation requires knowledge of the noun compound, we also experiment with adding different types of knowledge to the model that help it in generating accurate interpretations. various methods have been proposed to incorporate external knowledge into pre-trained language models @xcite @xcite . we use a simple strategy of concatenating the knowledge along with the proper noun compound before passing it to the model. a [sep] token is added as a demarcator to differentiate the added knowledge. we use four sources of knowledge that provide further information about the noun compound. they include information of the proper noun, from (1) the first paragraph of wikipedia that an entity linking system links it to (wiki-nnp), (2) tags assigned to it by the named entity recognition system (ner-nnp), or include information about the common noun using (3) the corresponding synset definitions provided in wordnet (wordnet-nn), or information about the entire compound based on the (4) sentence in which it is used. an example of each type of knowledge is shown in zero/few-shot prompting: prior techniques for noun compound interpretation such as @xcite have proposed zero-shot generation using pre-trained language models to achieve stateof-art performance on semeval 2013 task 4 @xcite and semeval 2010 task 9 @xcite . we therefore evaluate the performance of such techniques along with some extensions using few-shot learning on the pronci dataset. we find that there exists a significant gap compared to finetuning on the supervised dataset, demonstrating the importance of having a large scale dataset for the task of pnc interpretation. ## experimental setup the 22,500 examples of pronci are split into train, validation and test such that all compounds with the same common noun occur exclusively in a single set. such splitting ensures that there is no intersecting common noun in either the train or evaluation splits. this results in a more challenging setting than splitting the examples randomly, whose results are shown in appendix b. further, we also consider subsets that contain only compositional examples (cmp) or only non-compositional examples (non-cmp). the number of examples in each case are shown in score(g, p) = sem-match(g, p), if cmp ex-match(g, p), if non-cmp. in particular, we compare two alternatives for sem-match: (1) the rule-based popular bleu score, relies on n-gram overlap, and often used in machine translations; and (2) bleurt, which is finetuned over pretrained language model and represents a recent trend in trained evaluation metrics for text generation tasks. in both alternatives, we use the entire paraphrase to evaluate the semantic score as evaluating only the relations does not suit metrics such as bleurt, which expects a full-formed sentence to infer the semantic meaning. evaluation of the quality of bluert for similarity between predicted and gold paraphrases using 1k human annotated judgements indicates a 0.57 pearson and 0.56 kendall correlation. we follow standard protocols in evaluating metric quality, as used in wmt metrics shared tasks, and ask human annotators to rate the compositional model predictions as good, average and bad and see how these judgement scores correlate with the bleu and bleurt scores. further details are provided in appendix a. we denote the final evaluation metric as sem/ex-match. when using bleu or bleurt as the semantic matcher, the metric is also referred to as bleu/ex or bleurt/ex, respectively. to understand the effect of each type of match, we also report the ex-match classification accuracy over all the examples, where the compositional type is assigned the positive class, and the noncompositional type is assigned as the negative class. along with binary accuracy, we compute the precision and recall as well. since the sem-match cannot be computed over all examples, we report the scores averaged over only the cases where both gold and prediction are compositional. pre-trained models: for all our experiments, we use the t5-base @xcite as the default initialization, unless explicitly mentioned otherwise. it contains 220m parameters. for checking the statistical consistency, every model is trained 5 times with different seeds and their mean and standard deviation are reported. hyper-parameters and computational resources: we run all our experiments using a v100 gpu. we use the standard hyper-parameters recommended in t5 for all the experiments, using a batch size of 16, initial learning rate of 2e-5. the final model is chosen using early stopping on the validation set after training for 10 epochs. each round of training and evaluation takes around 1 hr. ## experiments in this section, we address the following questions: ## conclusion in this work we develop the novel task of semantic intepretation of proper noun compounds. we present the pronci dataset for the task and test performance of various neural models. we show the downstream utility of generated interpretations by integrating it with an open ie system that results in generation of new extractions involving implicit relations. linguistic characterization of noncompositionality and utilizing additional sources of knowledge present scope for further work.