Learning High-Quality and General-Purpose Phrase Representations (2024)

Abstract: Phrase representations play an important role in data science and natural language processing, benefiting various tasks like Entity Alignment, Record Linkage, Fuzzy Joins, and Paraphrase Classification.The current state-of-the-art method involves fine-tuning pre-trained language models for phrasal embeddings using contrastive learning. However, we have identified areas for improvement. First, these pre-trained models tend to be unnecessarily complex and require to be pre-trained on a corpus with context sentences.Second, leveraging the phrase type and morphology gives phrase representations that are both more precise and more flexible.We propose an improved framework to learn phrase representations in a context-free fashion.The framework employs phrase type classification as an auxiliary task and incorporates character-level information more effectively into the phrase representation.Furthermore, we design three granularities of data augmentation to increase the diversity of training samples.Our experiments across a wide range of tasks reveal that our approach generates superior phrase embeddings compared to previous methods while requiring a smaller model size.

## introduction a phrase is a group of words (or a single word) with a special meaning. they may denote recognizable entities: names of people (albert einstein), organizations (the new york times), dates @xcite , and events (2024 summer olympics). beyond these typical contexts, phrases also appear as column names in tabular data (average_wage), as user queries (black pant men), or even as a nonnoun phrase in clinical reports (more than 63kg). phrases are thus an important building block in many applications of both data science and natural language processing (nlp), e.g., in tasks such input entity name: the new york times figure 1 : an example of entity retrieval. given the input entity name "the new york times", we show the cosine similarity obtained by different models. the ranking of scores is listed in parentheses. as entity alignment @xcite , fuzzy joins @xcite , question answering @xcite , record linkage @xcite , and syntactic parsing @xcite . central to these applications is the assessment of the semantic similarity between two distinct phrases. today, the main tool to assess the similarity of phrases is phrase embeddings, i.e., learned vector representations that capture the semantics of the phrases in such a way that phrases with similar meanings are close in representation space. the difficulty of learning such representations arises from the fact that phrases often appear without context (e.g., in user queries), and exhibit diverse morphological variations. for example, given the entity "the new york times (q9684)", the knowledge base wikidata @xcite offers multiple aliases (alternate names) 1 . three of them are shown in the first rows of figure 1 . the last two rows show names of other entities: "new york post (q211374)" and "new york (q1384)". while all five of these phrases look very much alike, only the first three are associated with "the new york times". this versatility of phrases makes it hard to use rule-based or string-distance methods for semantic similarities. sentence-bert @xcite was the first approach to fine-tune pre-trained language models like bert @xcite and roberta @xcite to derive meaningful sentence embeddings. however, sentence-bert is given entire sentences during training (no special focus on short texts or phrases), so that its capabilities to embed phrases remain limited. phrase-bert @xcite was explicitly designed to embed phrases and adopts contrastive learning to fine-tune bert on lexically diverse phrasal paraphrase pairs and their surrounding context, yielding more powerful phrase embeddings. another context-aware approach, uctopic @xcite , further improved phrase representations by using cluster-assisted negative sampling i.e., leveraging clustering results as pseudo-labels. however, this prior work faces several limitations. first, phrases frequently appear devoid of context cues, especially in tabular data, and are often characterized by short lengths. consequently, we might not actually need the complex reasoning abilities of large (or deep) language models. a small (or shallow) neural architecture could suffice for the purpose of capturing phrase semantics. also, we need a model that works well in the absence of context. second, existing work partially neglects the type information of phrases. for example, although "the new york times" and "new york" have a high lexical overlap, a good representation model should distinguish them since the first phrase pertains to an organization while the second is linked to a geopolitical entity. third, existing sub-word embeddings are not robust against out-of-vocabulary words @xcite , and this vulnerability entails the necessity of using character-level features and morphological information. indeed, as figure 1 shows, phrase-bert and uctopic fail to recognize that "nytimes" is an abbreviation of the original phrase, and wrongly rank "new york post" (a different newspaper) as closest to "the new york times". in this paper, we present a context-free contrastive learning framework called pearl foot_1 , which enriches existing language models by incorporating phrase type and character-level features. additionally, pearl uses a range of data augmentation techniques to increase training samples. pearl has the following advantages: first, it is able to discern between phrases that share similar surface forms but are of different semantic types. for example, a model using our framework sees "new york" as a poor match for "the new york times" as it is of a different type: a geopolitical entity versus an organization (figure 1 ). second, our approach captures morphology in phrases better. in figure 1 , our method correctly ranks all three positive candidates, including those with acronyms, as nytimes. third, a pearl model of relatively small size (40m parameters) can outperform existing larger models (phrase-bert and uctopic) and it learns phrase embeddings in a context-free fashion. this results in shorter training times and less resource consumption, which makes our approach more accessible in low-resource scenarios and reduces its carbon footprint. we conduct extensive experiments with pearl across various phrase and short text tasks, including paraphrase classification, phrase similarity, entity retrieval, entity clustering, fuzzy join, and short text classification. we can show that our method outperforms other competitors across all these tasks -despite a smaller model size. ## related work phrases are fundamental linguistic units, pivotal to understanding languages. hence, learning their representations has attracted quite some attention in the research community. early works mostly use compositional transformation to obtain phrasal embeddings, i.e., they derive phrase representations from word embddings @xcite @xcite @xcite . with the advent of large pre-trained models, recent approaches fine-tune transformer models like bert @xcite to obtain generalized text embeddings, e.g. sentence-bert @xcite and e5 @xcite . however, a recent study suggests that phrase representations in these language models heavily rely on lexical content while struggling to capture the sophisticated compositional semantics @xcite . to develop more powerful models dedicated to phrasal representations, phrase-bert @xcite fine-tunes bert on lexically diverse datasets by using both phraselevel paraphrases and context sentences around phrases. this allows the production of embeddings that go beyond simple lexical overlap. another context-aware model, uctopic @xcite , proposes cluster-assisted contrastive learning for inducing phrasal representations for topic mining. mcphrasy @xcite incorporates context information into phrase embeddings during inference. although these methods can effectively generate semantically meaningful phrasal representations, they ignore the phrase type and morphological information, which are crucial for understanding phrases. in this paper, we show that our approach can outperform these models with a much smaller model. in the field of data science, a task closely related to phrase representation is string matching. it is widely used across diverse applications, including fuzzy join @xcite , entity resolution (papadakis et al., 2020) or alignment @xcite , and ontology matching @xcite . a simple yet effective solution for this task is similarity functions such as the edit distance and jaccard similarity, which assess either token-level or character-level (or n-gram) similarity. more refined methods resort to word embeddings like glove @xcite and fasttext @xcite to better capture lexical meaning. in this work, we show that models trained by our framework can be used for a series of database or knowledge base related tasks and achieve competitive results at little cost. ## our approach our objective is to learn representations for arbitrary input phrases. for this, we design a novel contrastive-learning framework named pearl, as shown in figure 2 . the input for pearl is contextfree phrases. this is different from other existing models like phrase-bert @xcite and uctopic @xcite which take phrases with context as input. given a specific phrase, pearl first applies data augmentation in order to obtain similar phrases that will serve as positive samples. for example, "the new york times" becomes "the new york timse" by using a character-level augmentation (character swap). next, embeddings are generated by both phrase-level and characterlevel encoders. we then learn embeddings with the help of contrastive loss, which aims to pull close positive pairs while pushing apart in-batch negative samples. in order to learn more expressive representations, we add a certain number of hard negatives to each batch. for example, "new york post" and "new york" can serve as hard negatives, given their high lexical overlap with the original phrase coupled with very distinct semantics. to integrate phrase structural information into the representations, we force the framework to assign tags of a lexical class and a named entity type to each phrase. for example, the framework learns to assign a np-org tag to the phrase "the new york times", meaning that the phrase is a noun phrase associated with an organization. the negative sample "new york", in contrast, receives a np-gpe tag, meaning that the phrase is a noun phrase linked to a geopolitical entity. this augmentation with entity type information allows the model to distinguish "the new york times" and "new york" in the representation space. ## conclusion in this study, we have presented pearl, a novel contrastive learning framework for more powerful phrase representations. pearl incorporates phrase type information and morphological features, and thereby captures better the nuances of phrases. furthermore, pearl enriches training samples with distinct granularities of data augmentations. our empirical results show that it improves phrase embeddings for a wide range of tasks, from paraphrase classification to entity retrieval, useful in applications across nlp and data engineering. adding character-level support to language models appears crucial to success on short texts. indeed, these provide much less context than full paragraphs and thus it is important to go beyond the tokens of the original language model that mainly capture word stems.