How You Ask Matters: The Effect of Paraphrastic Questions to BERT Performance on a Clinical SQ u AD Dataset (2020)

Abstract: Reading comprehension style question-answering (QA) based on patient-specific documents represents a growing area in clinical NLP with plentiful applications. Bidirectional Encoder Representations from Transformers (BERT) and its derivatives lead the state-of-the-art accuracy on the task, but most evaluation has treated the data as a pre-mixture without systematically looking into the potential effect of imperfect train/test questions. The current study seeks to address this gap by experimenting with full versus partial train/test data consisting of paraphrastic questions. Our key findings include 1) training with all pooled question variants yielded best accuracy, 2) the accuracy varied widely, from 0.74 to 0.80, when trained with each single question variant, and 3) questions of similar lexical/syntactic structure tended to induce identical answers. The results suggest that how you ask questions matters in BERT-based QA, especially at the training stage.

## introduction in clinical nlp, there has been vital interest in developing question-answering (qa) systems, e.g., askhermes @xcite , mipacq @xcite @xcite . one specific type of clinical qa targets on locating any suitable answer within a given document (a.k.a. reading comprehension), which is helpful for answering patient-specific questions based on information mentioned in clinical notes. recently, bert @xcite and its derivatives have struck impressive success in this task for general english, represented by squad @xcite , and for clinical text with promising results @xcite . however, an under-explored area in performing bert-assisted qa is: how the system would behave if the input question is asked in a different (paraphrastic) way? most existing experiments have assumed that the train and test data belong to a closed space with pre-assembled syntactic and lexical diversity (i.e., paraphrastic questions) representing what the users could ever ask, and faithfully evaluate the both diverse train/test sets in a symmetric manner. in practice, there are at least two possible scenarios concerning the potential effect from a differently-asked test question: 1) the system is trained with, or has seen, the question construct, 2) the system has never seen the question construct during training. here by "construct" we refer to paraphrases like: "why is the patient prescribed medication-x?" versus "why does the patient take medication-x?". more examples are in ideally, a bert qa model is supposed to provide a consistent answer as long as the user asked a semantically-equivalent (paraphrastic) question. this is important in a production system because a user should not be required to ask only questions conforming to "template" constructs. therefore, in this study we set to understand how such paraphrastic perturbation in asking would affect a bert-based qa model. we used a dataset that contained finite question constructs, but purposefully injected experiments with using limited constructs in the training and/or testing to simulate the asymmetric perturbations of interest. for example, training on only one question construct and testing on the other different constructs (i.e., unseen ways of asking). our major findings can be summarized as follows: besides the relevant backgrounds above, there have been nlp studies that reported the effect of paraphrastic questions in qa system performance. @xcite and @xcite developed approaches to paraphrasing questions for optimal answer accuracy in retrieval-based qa, where candidate answers were searched and ranked from a large set of documents. the closest work for reading comprehension qa we identified was by @xcite , which investigated the effect of question variants on a general english squad dataset. they demonstrated that unseen paraphrastic test questions hurt the accuracy of deep learning qa models, and proposed a countermeasure by pre-augmenting the training data with machine-generated paraphrastic questions. ## conclusion we found that how you ask matters in a bertbased clinical qa task, especially at the training stage. by controlling the train and test questions to individual lexical/syntactic constructs, our crossover evaluation showed that certain question constructs consistently yielded higher accuracy. accordingly, it suggests that the most effective way to secure robust performance is still by training with diverse, sizable questions. our results also brought up a somewhat nuanced inquiry: how come some question constructs seem to act "linguistically superior" to others, and whether it is a universal or bert-dependent phenomenon?