A uto RC : Improving BERT Based Relation Classification Models via Architecture Search (2021)

Abstract: Although BERT based relation classification (RC) models have achieved significant improvements over the traditional deep learning models, it seems that no consensus can be reached on what is the optimal architecture, since there are many design choices available. In this work, we design a comprehensive search space for BERT based RC models and employ a modified version of efficient neural architecture search (ENAS) method to automatically discover the design choices mentioned above. Experiments on eight benchmark RC tasks show that our method is efficient and effective in finding better architectures than the baseline BERT based RC models. Ablation study demonstrates the necessity of our search space design and the effectiveness of our search method. We also show that our framework can also apply to other entity related tasks like coreference resolution and span based named entity recognition (NER).

## introduction the task of relation classification (rc) is to predict semantic relations between pairs of entities inside a context. it is an important nlp task since it serves as an intermediate step in variety of nlp applications. there are many works that apply deep neural networks (dnn) to relation classification @xcite @xcite . with the rise of pre-trained language models (plms) @xcite , a series of literature have incorporated plms such as bert in rc tasks @xcite @xcite , and shows significant improvements over the traditional dnn models. despite great success, there is yet no consensus reached on how to represent the entity pair and their @xcite and @xcite use different aggregation methods of entity representations and contexts. third, choosing which features should be considered for the classfication layer should also be determined @xcite . in addition, previous literature does not consider the interactions between the feature vectors. in this work, we experiment on making the design choices in the bert based rc model automatically, so that one can obtain an architecture that better suits the task at hand (figure 1 ). throughout this work, we will refer to our framework as autorc, which includes our search space and search method. firstly, a comprehensive search space for the design choices that should be considered in a bert based rc model is established. second, to navigate on our search space, we employ reinforcement learning (rl) strategy following enas @xcite . that is, a controller generates new rc architectures, receives rewards, and updates its policy via policy gradient method. to stabilize and improve the search results, three non-trivial modifications to enas are proposed: a) heterogeneous parameter sharing, which is to share parameters more deeply than enas if the modules play similar role, and not to share if not; b) maintain multiple copies of the shared parameters which will be drawn randomly to the child models; c) search warm-ups, which is to generate and update child models without updating the controller at the beginning of the search stage. experiments on eight benchmark rc tasks show that our method can outperform the standard bert based rc models. transfer of the learned architecture across different tasks is investigated, which shows the transferred architectures can outperform the baseline models but cannot outperform the architecture learned on this task. ablation study of the search space demonstrates the validity of the search space design. in addition, ablation studies on the search space show the validity of our search space design, and experiments show that our proposed modifications to enas are effective. we also show our framework can work effectively on other entity related tasks like coreference resolution and span based ner. the contributions of the paper can be summarized as: ## search space for rc model an overall architecture design for a rc model is shown in figure 1 . following its bottom-up workflow, we will define the search space for autorc. ## search method in this section, we first formally formulate the problem of architecture search with reinforcement learning. then, , we discuss the search algorithm based on policy gradient. at the last part, we discuss our modifications to stabilize the search outputs. ## experiments due to resource limitations, we assign up to 2 n-vidia v100 gpu cards to each tasks. ## conclusion in this work, we first construct a comprehensive search space to include many import design choices for a bert based rc model. then we design an efficient search method with the help of rl to navigate on this search space. to improve the search results, parameter sharing strategies different from enas are designed. to avoid over-fitting, we maintain multiple copies of shared weights during search. wiki80 (denoted as wiki80) this dataset @xcite is derived from fewrel @xcite , a large scale few-shot dataset. since wiki80 only has a train/val split, we randomly split the train set into a train set and val set (with 8:2 ratio), and treat the original validation set as the test set. kbp-37 @xcite (denoted as kbp37). this dataset is a revision of miml-re annotation dataset, provided by gabor @xcite . they use both the 2010 and 2013 kbp official document collections, as well as a july 2013 dump of wikipedia as the text corpus for annotation. deft-2020 subtask 3 (denoted as deft2020) this dataset also serves as the task 6 of semeval 2020 shared tasks. this rc task have to overcome longer contexts, longer entity mentions, and more imbalanced relation types. @xcite i2b2 2010 (denoted as i2b2) shared task collection consists of 170 medical documents for training and 256 documents for testing, which is the subset of the original dataset @xcite . chemprot (denoted as chemprot) consists of 1,820 pubmed abstracts with chemical-protein interactions annotated by domain experts and was used in the biocreative vi text mining chemicalprotein interactions shared task @xcite 4 . ddi extraction 2013 corpus (denoted as ddi) is a collection of 792 texts selected from the drugbank database and other 233 medline abstracts @xcite . foot_4