CORE : A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation. (2023)

Abstract: We introduce CORE, a dataset for few-shot relation classification (RC) focused on company relations and business entities. CORE includes 4,708 instances of 12 relation types with corresponding textual evidence extracted from company Wikipedia pages. Company names and business entities pose a challenge for few-shot RC models due to the rich and diverse information associated with them. For example, a company name may represent the legal entity, products, people, or business divisions depending on the context. Therefore, deriving the relation type between entities is highly dependent on textual context. To evaluate the performance of state-of-the-art RC models on the CORE dataset, we conduct experiments in the few-shot domain adaptation setting. Our results reveal substantial performance gaps, confirming that models trained on different domains struggle to adapt to CORE. Interestingly, we find that models trained on CORE showcase improved out-of-domain performance, which highlights the importance of high-quality data for robust domain generalization. Specifically, the information richness embedded in business entities allows models to focus on contextual nuances, reducing their reliance on superficial clues such as relation-specific verbs. In addition to the dataset, we provide relevant code snippets to facilitate reproducibility and encourage further research in the field. The CORE dataset and code are publicly available at https://github.com/pnborchert/CORE .

## introduction relation classification (rc) is a fundamental task in natural language processing that involves identifying the corresponding relation between a pair of entities based on the surrounding textual context. several studies highlight the need for more realistic dataset configurations that include none-of-the-above (nota) detection @xcite , few-shot evaluation (baldini soares coke's e1 advertising is pervasive, as one of woodruff's stated goals was to ensure that everyone on earth drank coca-cola as their preferred beverage. ## related work recent research on datasets for few-shot rc has been focused on improving data quality. several approaches have been proposed, including extensions and augmentations of existing datasets to incorporate features such as nota detection or domain adaptation, as well as correction of labeling errors @xcite . in this section, we present an overview of recent rc datasets that encompass sentence-level textual context. one of such datasets is tacred, which is a human-annotated dataset based on online news, focusing on people and organizations as entities @xcite . it preserves a realistic relation type distribution and includes 79.5% nota instances. @xcite re-evaluated the original dataset, analyzing labeling errors, and investigating sources for incorrect model predictions. another prominent dataset is fewrel, introduced by @xcite , which consists of text passages extracted from english wikipedia pages. it is annotated using distant supervision and the wikidata knowledge base. the model evaluation is primarily focused on the k-way n-shot setting. @xcite extended the dataset by incorporating nota examples and introducing out-of-domain test sets. the crossre dataset specifically targets the domain adaptation setting in relation classification @xcite . it incorporates text from wikipedia and news sources, covering six domains. notably, the dataset excludes the business domain and is limited to named entities. in terms of relation types, crossre adopts a shared approach across domains, omitting any domain-specific relation types. @xcite acknowledge the value of business relations extracted from multilingual web sources and present their dataset, which is collected using distant supervision. this dataset focuses primarily on named entities and subsequent human annotation. it contains five types of undirected business relationships. domain adaptation refers to the task of transferring information from a source domain to a specific target domain. domain adaptation models learn to extract domain-invariant information from the source domain, which can be applied to the target domain. popular approaches include domain adversarial networks @xcite and leveraging external information such as task descriptions or knowledge bases @xcite . recent work has also explored prompt tuning approaches @xcite @xcite . in this study, we address the problem of domain adaptation by exposing the model to a limited number of instances from the target domain using few-shot learning. to achieve effective domain adaptation, models combine domain-invariant information learned from the source domain with the information available from the target domain. ## core 3.1 data collection to gather corporate wikipedia pages in english, we conducted a query on dbpedia @xcite using the ontology class "dbo:company", which corresponds to the wikidata identifier "q4830453". the dataset was constructed using wikipedia content corresponding to april 2022. we extract candidate entity pairs based on the company corresponding to the wikipedia page and named entities in the text. we use a named entity recognition (ner) model based on roberta-large @xcite that was fine-tuned on the conll2003 dataset to retain entities tagged as organizations or miscellaneous. the model achieved state-of-the-art performance on conll2003 with a f1-score of 97% on the full dataset and 96% on entities labeled as organizations. this extraction procedure allows the company corresponding to the wikipedia page to take any entity type, including named entities, pronouns, and nouns. we use spacy to create a dictionary of nouns and pronouns in the dataset and manually validate entity types @xcite . for each candidate entity pair, we extract the corresponding paragraph as supporting context characterizing the relation between the entities. we remove duplicate entity pairs with overlapping context to reduce the number of candidate samples and manually filter out miscellaneous entities that do not belong to the category of business entities, such as "german" or "dutch". finally, we randomly sampled 14,000 candidate samples for subsequent annotation. the annotation platform is created by the authors, offering a similar interface as commercial services such as mturk (see appendix a.5). the dataset is annotated by 50 students in business and management who are presented with candidate entity pairs and the corresponding textual context. the annotation consists of selecting one of the predefined relations (see appendix a.1) and the textual context relevant to determine the selected relation (see figure 2 ). since relations such as "acquired by" imply a direction, annotators are also asked to indicate whether the entity order is reversed: (e 1 , r, e 2 ) → (e 2 , r, e 1 ). the relation annotation and the position of e1 and e2 in the textual context are validated in the subsequent annotation step. thereby, the validation process includes the removal of candidate instances that contain nonbusiness entities, incorrect relation type selection, insufficient context, or inappropriate data. the annotators were recruited through an open application process. for their contribution to the annotation process, the students received a reimbursement of @xmath06,600. ## results and discussion we evaluate the performance of the models using two evaluation metrics: micro f1 and macro f1. micro f1 provides an overall assessment of model performance by treating instances equally and accounting for the impact of an imbalanced nota ratio. in contrast, macro f1 focuses on individual class performance, treating each class equally and remaining unaffected by the nota ratio. we report micro f1 and macro f1 scores in percentage format in tables 3 , 4 , and 5. the best results in each table are highlighted in bold, while the best performance measures for each domain are underlined. to ensure robust evaluation, the models were evaluated on ten sets of training and test relations for each dataset and n-way k-shot setting. the resulting evaluation metrics were then averaged. ## conclusion in this paper, we introduced core, a few-shot rc dataset that specifically focuses on company relations and business entities. to the best of our knowledge, there are no other datasets that specifically target company relations for few-shot rc, making core a valuable addition to the existing benchmark resources. by extending the availability of datasets for few-shot domain adaptation settings, core contributes towards more robust and realistic rc evaluation results. despite the progress made in addressing challenges in rc datasets, such as incorporating more realistic entity types and relation distributions, there is still a need for further research and attention in adapting models to out-of-domain datasets using diverse datasets and modeling approaches. our findings illustrate that models trained on other rc datasets often rely on shortcuts that fail to generalize to core. moreover, models trained on core display superior out-of-domain performance. the information richness in the training data enables models to focus on contextual nuances, reducing their reliance on superficial clues such as relationspecific verbs. this underscores the significance of high-quality datasets that include challenging entity types and require a deeper contextual understanding. in conclusion, core serves as a valuable resource for the few-shot domain adaptation setting in rc. it highlights the importance of incorporating diverse and challenging datasets and encourages future research efforts to focus on improving model adaptation to out-of-domain data. by addressing these challenges, we can advance the development of more effective and robust rc models.