P i C : A Phrase-in-Context Dataset for Phrase Understanding and Semantic Search (2023)

Abstract: While contextualized word embeddings have been a de-facto standard, learning contextualized phrase embeddings is less explored and being hindered by the lack of a human-annotated benchmark that tests machine understanding of phrase semantics given a context sentence or paragraph (instead of phrases alone). To fill this gap, we propose PiC—a dataset of ∼28K of noun phrases accompanied by their contextual Wikipedia pages and a suite of three tasks for training and evaluating phrase embeddings. Training on PiC improves ranking-models’ accuracy and remarkably pushes span selection (SS) models (i.e., predicting the start and end index of the target phrase) near human accuracy, which is 95% Exact Match (EM) on semantic search given a query phrase and a passage. Interestingly, we find evidence that such impressive performance is because the SS models learn to better capture the common meaning of a phrase regardless of its actual context. SotA models perform poorly in distinguishing two senses of the same phrase in two contexts (∼60% EM) and in estimating the similarity between two different phrases in the same context (∼70% EM).

## introduction understanding phrases in context is a key to learning new vocabularies @xcite , disambiguation (pilehvar and camacho-collados, 2019), and many downstream tasks, including semantic search @xcite ). yet, the contextualized phrase embeddings @xcite in existing systems mostly capture the common meaning of a phrase, i.e. without strong dependence on its context @xcite . while there are word-sense disambiguation datasets @xcite , no such benchmarks exist for phrases. existing phrase-similarity benchmarks @xcite @xcite @xcite compare phrases alone (without context) and some of them @xcite contain a large, undesired amount (∼15% to 99%) of phrase pairs that have lexical overlap ( others generated the context for a phrase by querying gpt-2 @xcite or by retrieving from wikipedia @xcite ). yet, there was no human verification of the realism of generated text @xcite and no human annotation of how a phrase's meaning changes w.r.t. the context @xcite . all above drawbacks are limiting the evaluation of phrase understanding. to advance the development of contextualized phrase embeddings, we propose phrase-in-context (pic), a suite of three tasks: (1) phrase similarity (ps), i.e. compare the semantic similarity of two phrases in the same context sentence (fig. 1b ); (2) phrase retrieval (pr), which is divided into prpass and pr-page (fig. 1c-d ), i.e. from a passage or a wikipedia page, retrieve a phrase semanticallysimilar to a given query phrase; and (3) phrase-sense disambiguation (psd), i.e. find the target phrase p semantically similar to the query phrase from a 2-paragraph document where p appears twice, each time in a different context paragraph that provides a unique meaning to p (fig. 1e ). our ∼28k-example dataset is rigorously (a) annotated and verified by two groups of annotators: linguistics experts on upwork.com and non-experts on amazon mechanical turk (mturk); and then @xcite tested by models, linguists, and graduate students. our contributions are: ## related work each of our tasks (ps-phrase similarity; prphrase retrieval; and psd-disambiguation) is related to a separate research area discussed below. phrase similarity first, most existing phrase similarity datasets-e.g. ppdb-annotated @xcite , ppdb-filtered @xcite , bird @xcite , and paws-short @xcite -contain a large percent of instances with lexical overlap between two paired phrases while our ps contains the least percent (5.34%; question answering (qa) our phrase-retrieval tasks-pr and psd-follow the format of qa datasets except that our queries are phrases instead of questions and hence shorter ( without the need for predefined senses, wic (pilehvar and camacho-collados, 2019) poses disambiguation as a binary classification task where the goal is to predict whether the same target word in two different sentences carries the same or different meanings. compared to wic ps is also a binary classification task, but with two major differences: (1) in samson is the emblem of lungau, salzburg and parades in his honor are held annually in ten villages of the lungau and two villages in the north-west styria (austria). during the parade, a young bachelor from the community carries a massive figure made of wood or aluminum said to represent samson. the tradition, which was first documented in 1635, was entered into the unesco list of intangible cultural heritage in austria in 2010………………………… wic, the same target word appears in two different sentences while in ps, two different phrases appear in the same context sentence; (2) ps compares phrases composed @xmath0. while word senses are defined in wordnet and babelnet dictionaries @xcite , there are no english dictionaries of senses for multi-word noun phrases (mnps). thus, it is more challenging to acquire and learn the senses of mnps, hence the importance of our pic dataset. like wic, psd tests disambiguating the meanings of the same ngram in two different contexts. yet, psd is a phrase search task, which involves many more phrase comparisons per example than ps or wic. before the deep learning era, phrase-sense disambiguation was already proposed @xcite but only as an auxiliary task for training machine-translation models. and their phrase senses were not annotated by humans but inferred by performing word-alignment on a bilingual corpus. here, our psd is the first phrase-sense disambiguation task annotated by experts and requires understanding of phrase-senses in a passage. ## pic dataset construction we first collect a set of phrases with context and human annotations. then, we derive the examples and labels for three main tasks: ps, pr, and psd (fig. 1 ). our idea is to mine a set of triplets (p, page 1 , page 2 ) from wikipedia where the phrase p is a polysemous mnp that carries two different senses in two wikipedia pages (e.g., "massive figure" means a large number in page 1 but a huge physical shape in page 2 ; fig. 1a ). then, we ask experts to rephrase p into two paraphrases q 1 and q 2 , maintaining the two original senses of p in page 1 and page 2 , respectively. the resultant set of 5-tuples (p, q 1 , q 2 , page 1 , page 2 ) enables the tests for (1) comparing the semantic similarity of two phrases given the same context sentence (ps; fig. 1b); (2) finding a semantically similar phrase in a document (pr-pass & pr-page; fig. 1c); (3) disambiguating the senses of the same target mnp given two context paragraphs (psd; fig. 1e ). ## three phrase understanding tasks using the human-annotated data, we construct three tasks of ps, pr, and psd (as summarized in fig. 1 ) for evaluating contextualized phraseembeddings and semantic-search models. ## experiments and results we test sota models on ps, pr-pass, pr-page, and psd to (1) assess how the models are able to leverage context to improve accuracy; and (2) quantify the headroom for future research. phrase embeddings besides training and testing sota bert-based classifiers, we also test a ranking approach that involves computing the cosine similarity between the query's and each candidate's embedding. to compute a contextualized phrase embedding, following yu and ettinger 2020, we feed the entire phrase-containing sentence (e.g. s 1 in fig. 1b ) into a model, e.g. bert, and then take the mean pooling of the last-layer embeddings over the words of the given phrase only. for non-contextualized phrase embeddings, we repeat the same process but input to the model only the phrase (instead of the entire sentence). models we choose sota models in (a) phrase similarity: phrasebert @xcite ; @xcite sentence similarity: use-v5 @xcite , sentencebert @xcite , and simcse @xcite ); (c) questionanswering: longformer @xcite , densephrase @xcite ; and (d) contextualized embeddings: spanbert @xcite and bert @xcite . for densephrase, we use their phrase-encoder (as opposed to the query-encoder) to compute phrase embeddings. use-v5 is only available via public apis (tensorflow, 2022) that do not support extraction of contextualized phrase embeddings. ## discussion and conclusion while wic and english wsd rely exclusively on dictionaries (pilehvar and camacho-collados, 2019) to obtain word senses and example sentences, our data collection depends on wikipedia, wic, & nlp models and our annotation depends on experts. in sum, we present pic, the first 3-task suite for evaluating phrases in context. ss models can obtain high accuracy on semantic search after training on our pr-pass and pr-page datasets. yet, their capability is limited to finding a semantically-similar phrase given a single context that contains the target phrase (in pr-pass). the results on ps and psd show that sota phrase embeddings are still limited in encoding contextualized phrases. it is interesting future work to improve these models for disambiguating the senses of a phrase in context (ps and psd).