E con BERT a: Towards Robust Extraction of Named Entities in Economics (2023)

Abstract: Adapting general-purpose language models has proven to be effective in tackling downstream tasks within specific domains. In this paper, we address the task of extracting entities from the economics literature on impact evaluation. To this end, we release EconBERTa, a large language model pretrained on scientific publications in economics, and ECON-IE, a new expert-annotated dataset of economics abstracts for Named Entity Recognition (NER). We find that EconBERTa reaches state-of-the-art performance on our downstream NER task. Additionally, we extensively analyze the model’s generalization capacities, finding that most errors correspond to detecting only a subspan of an entity or failure to extrapolate to longer sequences. This limitation is primarily due to an inability to detect part-of-speech sequences unseen during training, and this effect diminishes when the number of unique instances in the training set increases. Examining the generalization abilities of domain-specific language models paves the way towards improving the robustness of NER models for causal knowledge extraction.

## introduction implementing robust systems capable of automatically extracting structured information from unstructured text is critical for a variety of applications, ranging from event detection @xcite @xcite @xcite to building knowledge databases @xcite @xcite . in particular, named entity recognition (ner) is a canonical information extraction task which consists of detecting text spans and classifying them into a predetermined set of entity types @xcite @xcite . in the past decade, numerous bench-mark datasets have enabled researchers to compare and improve the performances of ner models within specific domains such as science @xcite , medicine @xcite , law @xcite , finance @xcite , and social media @xcite ; in some cases, these datasets have spanned multiple domains @xcite or languages @xcite . such datasets are crucial for building models capable of handling a wide range of downstream applications. in medicine for instance, information extraction systems can allow clinician and medical researchers to easily determine which medication is an effective treatment for a specific disease. analogously, knowing which policy intervention produces a certain economic outcome is imperative for evidence-based policymaking. however, while much work has been done on identifying relevant causal entities in biomedical research @xcite , this question has been unexplored for impact evaluation (ie) in economics. to fill this gap, we introduce econ-ie, a ner dataset of 1,000 abstracts from economic research papers annotated for entities describing the causal effects of policy interventions: intervention, outcome, population, effect size, and coreference. to the best of our knowledge, this is the first dataset of this kind in economics, thereby laying foundations for causal knowledge extraction in the field. the introduction of pretrained language models, sometimes referred to as foundation models @xcite , had a profound impact on the field of information extraction, enabling substantial gains in performance across domains by fine-tuning these general-purpose systems on downstream tasks @xcite . in this paradigm, such models are often fine-tuned and deployed to perform a domainspecific task, introducing a distribution shift between the pretraining data and examples seen at inference time. existing work shows that pretraining a language model on documents from the same domain as the downstream task improves performance @xcite , where this domain adaptation can either be implemented by in-domain pretraining from scratch @xcite @xcite , or by further in-domain pretraining starting from the weights of an existing pretrained general-purpose model @xcite . while pretrained language models are available across a wide range of domains @xcite @xcite , economics has yet to follow suit. therefore we contribute econberta, a language model pretrained on 1.5 million scientific papers from economics, demonstrating that it outperforms opensource general-purpose pretrained models on the econ-ie benchmark dataset. while assessing the quality of a model is often achieved by measuring performance after finetuning, the knowledge captured by such transfer learning approaches is opaque. therefore, it remains unclear, which aspects of a model require improvement in order to increase robustness during deployment. understanding the weaknesses of state-of-the-art models on downstream tasks requires operationalizing the notion of generalization. however, we lack a precise definition of generalization in spite of recent attempts to re-explore the concept @xcite . for instance, high performance on a held-out test set does not necessarily imply that a model is achieving robust generalization @xcite . in this context, we challenge the surface performance of our pretrained model, and seek cases where it fails to generalize on the ner task. our examination reveals three common error patterns. first, we show that most errors occur when the model detects an entity but fails to properly delimit its boundaries. we further find that, as the number of tokens in a span increases, the model is more likely to incorrectly predict boundaries. lastly, our model extrapolates more easily to part-of-speech sequences seen during training, a tendency exacerbated by a low number of unique spans for a given entity type in the training dataset. taken together, our contributions can be summarized as follows: the pretrained econberta models, the econ-ie dataset, the metadata of the economic pretraining corpus and the source code to replicate the experiments are available via https://github.com/ worldbank/econberta-econie 2 related work ## experimental setup in this section, we introduce the econ-ie dataset for ner ( §3.1), present the training procedure for econberta, and describe the baseline models ( §3.2). ## model diagnosis and results in this section, we present the performance of the different pretrained models on the ner task, and analyze their generalization capabilities. ## discussion memorization: a core component of learning or a sign of undergeneralization? throughout this study, we tested several memorization effects ranging from the shallow memorization of lexical patterns to that of lexicosyntactic sequences. while memorization is often opposed to generalization and considered to be a weakness of models @xcite , its role in the learning process is a source of debate. for instance, some authors have argued that memorization and in particular lexical memorization is a key component of language acquisition and linguistic knowledge @xcite @xcite . while relying too much on memorized associations might be detrimental to learning a task properly @xcite (see results on coreferences in section 4.4), penalizing this phenomenon entirely could also be too prescriptive. memorization could also play a crucial role either in early or in late stages of the learning process, especially if the task is characterized by a multiplicity of implicit rules and exceptions. learning from mistakes uncovering how a model reaches a decision is helpful to understand why it makes certain mistakes and implement modeling strategies to improve its performance. recent work proposed to teach fine-tuned models to learn from their mistakes on nlu tasks @xcite . similarly, @xcite attempted to debias ner models by using a counterfactual data-augmentation technique. together with the diagnoses presented in this paper, these studies pro-vide a blueprint on how to improve models used for deployment, which constitute interesting directions for future work. to improve models' ability to extrapolate to longer sequences, future work could also explore the impact of the splitting strategy or of augmenting the training data with difficult examples. ## conclusion in this work, we presented econberta, a novel language model adapted to the economic domain. we demonstrated that it achieves state-of-the-art performance on econ-ie, a novel ner dataset of 1,000 abstracts from economic research annotated for entities describing the causal effects of policy interventions. in doing so, we demonstrated the advantages of domain adaptation, specifically further pretraining from existing checkpoints. additionally, we diagnosed the model's generalization abilities in a series of fine-grained tests. we found that most mistakes stem from its inability to elicit the exact boundaries of detected entities, and that this phenomenon is due to the model relying on memorization. the memorization effect is mild and extends beyond lexical memorization, as the model is able to extrapolate to entities which have the same part-of-speech sequence as entities seen during training. finally, we found that the effect of memorization is more pronounced for entities which are less diverse in the training data. taken together, these findings pave the way towards developing more robust downstream information extraction systems.