A Query-Parallel Machine Reading Comprehension Framework for Low-resource NER (2023)

Abstract: Named entity recognition (NER) is a fundamental task in natural language processing. Recently, NER has been formulated as a machine reading comprehension (MRC) task, in which manually-crafted queries are used to extract entities of different types. However, current MRC-based NER techniques are limited to extracting a single type of entities at a time and are largely geared towards resource-rich settings. This renders them inefficient during the inference phase, while also leaving their potential untapped for utilization in low-resource settings. We suggest a query-parallel MRC-based approach to address these issues, which is capable of extracting multiple entity types concurrently and is applicable to both resource-rich and resource-limited settings. Specifically, we propose a query-parallel encoder which uses a query-segmented attention mechanism to isolate the semantics of queries and model the query-context interaction with a unidirectional flow. This allows for easier generalization to new entity types or transfer to new domains. After obtaining the query and context representations through the encoder, they are fed into a query-conditioned biaffine predictor to extract multiple entities at once. The model is trained with parameter-efficient tuning technique, making it more data-efficient. We conduct extensive experiments and demonstrate that our model performs competitively against strong baseline methods in resource-rich settings, and achieves state-of-the-art results in low-resource settings, including training-from-scratch, in-domain transfer and cross-domain transfer tasks.

## introduction named entity recognition (ner) is a fundamental task in natural language processing, aiming at detecting the text spans that refer to entities. it has been widely used in various downstream tasks, such as entity linking @xcite , relation extraction (miwa and @xcite ) and information retrieval @xcite . traditionally, ner has been formalized as a sequence labeling task, assigning a single tag class to each token in a sentence @xcite @xcite @xcite . recently, several new ner paradigms have been proposed, which conceptualize ner as span classification @xcite @xcite @xcite , sequence generation @xcite @xcite @xcite and constituency parsing @xcite tasks. in spite of achieving promising performance, these approaches require large amounts of well-annotated in-domain data @xcite . the data annotation usually involves carefully defined guideline and annotators with domain expertise, which could be quite time-consuming and costprohibitive. as a result, the development of ner systems is a costly endeavor in real-world scenarios where usually only a small amount of labeled data is available for new domains @xcite . recently, some works reformulate ner as a ma-chine reading comprehension (mrc) task @xcite . as illustrated in figure 1(a), the input sentence is regarded as the context, a collection of queries that have been manually crafted to represent various entity types can be viewed as questions, then extracting entities from the context could be solved by employing question answer techniques. prior mrc-based ner approaches are limited to extracting one type of entities at a time and have largely been applied to the standard supervised setting. in this context, they have exhibited superior performance in comparison to conventional sequence labeling techniques and have demonstrated comparable performance with reduced training data. nonetheless, the extraction of a singular entity type at a time is not particularly efficient, and their potential for deployment in low-resource settings remains largely unexplored. to alleviate these issues, this paper presents a query-parallel machine reading comprehension framework for named entity recognition (qpmrc-ner). it is suitable for both resource-rich and resource-limited settings, and is characterized by enhanced efficiency, as illustrated in figure 1 @xcite . it consists of two key components: the query-parallel encoder and the query-conditioned biaffine predictor. the query-parallel encoder takes the combination of the context sentence and the queries with shared continuous prompt as input, and utilizes a query-segmented attention mechanism to separate the queries from one another and model the query-context interaction with a unidirectional flow, thus facilitating easier generalization to new entity types and domain transfer. after obtaining the contextualized representations from the query-parallel encoder, we feed them to the query-conditioned biaffine predictor to extract entities of multiple types simultaneously. the model is trained with parameter-efficient techniques for data-efficiency @xcite . we conduct extensive experiments, and in most cases, our model outperforms the present sota methods. the main contributions of this paper are summarized as follows: (1) we introduce a novel query-parallel machine reading comprehension framework qpmrc-ner that is capable of handling low-resource ner tasks effectively and efficiently. (2) our mrc-based architecture facilitates simultaneous extraction of entities belonging to multiple categories, resulting in faster inference speed. (3) we conduct extensive evaluations of qpmrc-ner across a diverse range of ner tasks. our model exhibits competitive performance in resource-rich settings and achieves state-of-the-art results in low-resource settings, including trainingfrom-scratch, in-domain transfer, and cross-domain transfer tasks. ## method this section presents the task formulation in section 2.1, then introduces the proposed method qpmrc-ner, illustrated in figure 2 . qpmrc-ner consists of two components: the queryparallel encoder and the query-conditioned biaffine predictor, explained in section 2.2 and section 2.3 respectively. ## related work although ner is usually formalized as a sequence labeling task @xcite @xcite , several new ner paradigms have been proposed recently, conceptualizing ner as span classification @xcite @xcite @xcite , sequence generation @xcite @xcite , constituency parsing @xcite and machine reading comprehension @xcite tasks and achieving impressive results. however, these approaches are mainly focus on standard supervised setting, which is not suitable for low-resource scenarios. another line of research sought to address the low-resource ner task using techniques such as prototype-based learning @xcite @xcite , template-based learning @xcite and contrastive learning @xcite . but they often fail to fully exploit the potential of pretrained language models (plms) and perform inferior to standard sequence labeling ner methods in resource-rich settings. to bridge the gap, we propose a machine reading comprehension based method, which is effective in both resource-rich and resource-limited ner settings. ## conclusion in this paper, we propose a query-parallel machine reading comprehension framework, which predicts all entities simultaneously and is applicable to both resource-rich and resource-limited settings. specifically, we introduce the query-parallel encoder, which leverages the query-segmented attention mechanism to facilitate more straightforward generalization to new entity types. additionally, we propose the query-conditioned biaffine predictor, which enables parallel prediction of entities. the model is trained with parameter-efficient technique for data-efficiency. extensive experimentation has shown that our approach attains a faster inference speed, exhibits competitive performance against strong baselines in resource-rich setting, and yields state-of-the-art outcomes in low-resource contexts, including training-from-scratch, in-domain transfer, and cross-domain transfer tasks.