Low-resource Entity Set Expansion: A Comprehensive Study on User-generated Text (2022)

Abstract: Entity set expansion (ESE) aims at obtaining a more complete set of entities given a textual corpus and a seed set of entities of a concept. Although it is a critical task in many NLP applications, existing benchmarks are limited to well-formed text (e.g., Wikipedia) and well-defined concepts (e.g., countries and diseases). Furthermore, only a small number of predictions are evaluated compared to the actual size of an entity set. A rigorous assessment of ESE methods warrants more comprehensive benchmarks and evaluation. In this paper, we consider user-generated text to understand the generalizability of ESE methods. We develop new benchmarks and propose more rigorous evaluation metrics for assessing the performance of ESE methods. Additionally, we identify phenomena such as non-named entities, multifaceted entities, vague concepts that are more prevalent in user-generated text than well-formed text, and use them to profile ESE methods. We observe that the strong performance of state-of-the-art ESE methods does not generalize well to user-generated text. We conduct comprehensive empirical analysis and draw insights from the findings.

## introduction entities are integral to applications that require understanding natural language text such as semantic search @xcite , question answering @xcite and knowledge base construction @xcite . to this end, entity set expansion (ese) is a crucial task that uses a textual corpus to enhance a set of seed entities (e.g., 'mini bar', 'tv unit') with since training data in new domains is scarce, many existing ese methods expand a small seed set by learning to rank new entity candidates with limited supervision. broadly speaking, there are two types of such low-resource ese methods: (a) corpus-based methods @xcite @xcite ) that bootstrap the seed set using contextual features and patterns, and (b) language model-based methods @xcite ) that probe a pre-trained language model with prompts to rank the entity candidates. despite the recent progress, reported success of ese methods is largely limited to benchmarks focusing on named entities (e.g., countries, diseases) and well-written text such as wikipedia. furthermore, the evaluation is limited to top 10-50 predictions regardless of the actual size of the entity set. as a result, it is unclear whether the reported effectiveness of ese methods is conditional to datasets, domains, and/or evaluation methods. in this paper, we conduct a comprehensive study to investigate the generalizability of ese methods in low-resource settings. specifically, we focus on domains with user-generated text such as reviews. user-generated text data is abundant and is largely unlabeled. enabling nlp applications including semantic search and question answering @xcite @xcite over user-generated text requires entities mined from these largely unlabeled data. furthermore, user-generated text has distinctive characteristics than well-written text, making it appropriate for this study. due to lack of benchmarks on usergenerated text, we create new benchmarks from three domains -hotels, restaurants and jobs. we found that these benchmarks exhibit characteristics (illustrated in figure 1 ) distinct from existing benchmarks: (a) multifaceted entities (entities that belong to multiple concepts -e.g., 'venice beach' can belong to concepts location and nearby attractions); (b) non-named entities (entities that are typically noun phrases but not proper names -e.g., 'coffee'); and (c) vague entities (human annotators have subjective disagreement on their concept labels -e.g., 'casino' for nearby attraction). we explain why these characteristics emerge in user-generated text in section 3. we found that user-generated text can have up to 10× more multifaceted entities and 2× more nonnamed entities compared to well-curated benchmarks. furthermore, concepts that do not have well-defined semantics result in vague entities. we use these characteristics to profile ese methods, showing that the performance difference between well-curated and user-generated text can partially be attributed to these characteristics. contributions. to summarize, our key contributions include: a) identifying and verifying several important new characteristics in user-generated text that are not explored in evaluation of existing ese methods, b) constructing three new usergenerated text benchmarks (we publicly release two foot_0 ), c) proposing new metrics for evaluating ese methods, d) deriving insights through a crossdomain (user-generated text vs. well-curated) comparison study on different ese methods. key findings. our main findings are listed below: • widely used evaluation metrics such as (mean average precision (map) at @xmath0. evaluating top-k g foot_1 predictions is potentially more robust, especially for benchmarking. • performance of state-of-the-art (sota) ese methods drops dramatically on user-generated text compared to well-curated text. • deviating from prior observations, simple corpus-based and language model-based methods that underperform sota methods on well-curated text can outperform sota methods on user-generated text. • simple rank-based ensemble methods can provide further improvements on user-generated text. the degree of overlap of correct predictions from candidate methods is indicative of the effectiveness of their ensemble. ## background and related work we now introduce the task of entity set expansion (ese), existing paradigms and evaluation methods. ## case study existing work suggest that user-generated text differs from well-curated text in writing style @xcite and cleanliness @xcite . in this section, we discuss one of the use cases of ese for a downstream nlp application and highlight new characteristics in user-generated text that are particularly relevant to the ese task. ## experimental set-up we now outline our experiment set-up designed to explore the suitability of existing benchmarks, metrics, and methods. ## findings we next share our findings from analyzing the ese methods. note that all the results are obtained from single run of each experiment. ## discussion we now discuss the implications of the proposed benchmark, metrics, and experiment observations. capturing the silent majority. recent work @xcite shows that the majority of the entities in wikipedia articleswhich feeds knowledge-bases such as dbpedia @xcite and yago @xcite -are non-named and recommends adding the silent majority to these kbs for completeness. to this end, our proposed benchmark highlights the importance of capturing multifaceted, vague, and non-named entities present in user-generated text. for example, domain-specific kbs such as the amazon product knowledge graph (karamanolakis et al., 2020) rely on user-generated text to collect entities for concepts of interest. these kbs power many downstream tasks such as semantic search, question answering, and conversational ai. therefore, these kbs would remain incomplete without capturing the different types of entities identified in our benchmark. practical usage. the goal of our evaluation metrics (evaluation at k g ) is to characterize the performance of ese methods in the presence of entity types that are typically present in user-generated text. note that we do not recommend replacing the existing metric map@@xmath1. our proposed map@k g metric is complementary and is designed to stress test ese methods in scenarios where coverage is an important criteria (e.g., kb population.) top-20 predictions do not have enough representation of non-named, multifaceted, and vague entities. therefore, when evaluating ese methods designed for user-generated text on our benchmark, the proposed evaluation at k g metric may help practitioners measure the suitability of a method. ## conclusion we conduct a comprehensive study to analyze the performance of ese in user-generated text. we observe that user-generated text has characteristics that are not captured in existing benchmarks, and propose new benchmarks and evaluation metrics. our findings indicate that state-of-the-art methods are not very effective in user-generated text and are often outperformed by simpler baselines.