Discovering Better Model Architectures for Medical Query Understanding (2021)

Abstract: In developing an online question-answering system for the medical domains, natural language inference (NLI) models play a central role in question matching and intention detection. However, which models are best for our datasets? Manually selecting or tuning a model is time-consuming. Thus we experiment with automatically optimizing the model architectures on the task at hand via neural architecture search (NAS). First, we formulate a novel architecture search space based on the previous NAS literature, supporting cross-sentence attention (cross-attn) modeling. Second, we propose to modify the ENAS method to accelerate and stabilize the search results. We conduct extensive experiments on our two medical NLI tasks. Results show that our system can easily outperform the classical baseline models. We compare different NAS methods and demonstrate our approach provides the best results.

## introduction nowadays, online medical question answering (qa) systems are becoming more and more popular. since the breakout of covid-19, people grapple with going to the hospital, and hospitals' emergency rooms in some cities are even empty during the daytime. 1 thus, medical qa systems are of essential importance. nli models play a central role in such a qa system @xcite . in our qa scenario, we usually use nli models to determine whether a query has the same intention as some of our labeled questions. for in-domain tasks like ours, modeling experiences are scarce, so selecting a suitable model for our medical nli tasks becomes a time-consuming procedure. from our experience, different datasets have different optimal models. thus when a new dataset comes, our engineers usually devote 4 to 5 days experimenting on model tuning and hyper-parameter search with multiple gpu cards. to speed up the development of our medical qa system and free up the nlp engineers from laborious work, we propose developing a neural architecture search (nas) framework. neural architecture search (nas) has recently attracted intensive attention, both in computer vision and nlp. new rnn models are learned in nasnet @xcite , enas @xcite , darts @xcite , improved darts @xcite for language modeling. evolved transformer @xcite use the evolution-based nas algorithm to search for better transformer architectures. textnas @xcite design a new search space for nlu tasks. we will refer to our system as nasqu, shorted for neural architecture search for query understanding. nasqu consists of two parts. first, we design a search space that is suited for nli tasks. the search space is an extension of the search spaces of textnas @xcite . first, for sentence pair modeling, cross-sentence attention plays a central role in aligning the two sentences' contexts and providing a more in-depth understanding of the semantic relations between two sentences @xcite . we add cross-sentence attention operations in the search space, enabling nasqu to search for models with cross-sentence attention mechanisms. second, aggregating the encoder's outputs to a fixed vector is essential for an nli model's performance. in this work, we use nas to decide which layers' outputs are fed into the aggregator and the specific operation in the aggregator layer. second, for improving the search results, we employ two modifications to the search method. our search method mainly follows enas @xcite , a reinforcement learning (rl) based search approach. an lstm controller is employed to generate a novel child model, and it will receive a reward based on the child model's performance. then the controller can update its parameters to improve its ability to generate better child models. a key ingredient for enas is weight sharing. we propose further to enhance the weight sharing strategies during architecture search. besides, we suggest that search warm-ups can stabilize the search processes and provide better search results. we conduct experiments on two medical nli datasets designated for intent identification in our medical qa system. the experimental results show that nasqu can learn novel models that perform better than baseline models and meet efficiency requirements. also, a comparison among the search methods shows that our nasqu obtains better results than other search methods. besides, we show that the search space design of nasqu is essential. our work contributes to the field by the following aspects: ## nasqu in this section, we first describe the architecture framework of nli tasks. then we extend the search space of textnas to support cross-attn modeling and aggregator search. and finally, we elaborate on the search algorithm in nasqu. ## conclusion and future work in this work, we experiment on modeling nli tasks via nas on our medical nli tasks. our search space is an extension to textnas and enas, which enable us to search for novel models with cross sentence attention and suitable aggregators. to improve the search results, we also propose a more in-depth weight sharing strategy than enas and search warm-up steps. experiments on our nli tasks demonstrate that our search space is beneficial for nli tasks.