Where’s the Question? A Multi-channel Deep Convolutional Neural Network for Question Identification in Textual Data (2020)

Abstract: In most clinical practice settings, there is no rigorous reviewing of the clinical documentation, resulting in inaccurate information captured in the patient medical records. The gold standard in clinical data capturing is achieved via “expert-review”, where clinicians can have a dialogue with a domain expert (reviewers) and ask them questions about data entry rules. Automatically identifying “real questions” in these dialogues could uncover ambiguities or common problems in data capturing in a given clinical setting. In this study, we proposed a novel multi-channel deep convolutional neural network architecture, namely Quest-CNN, for the purpose of separating real questions that expect an answer (information or help) about an issue from sentences that are not questions, as well as from questions referring to an issue mentioned in a nearby sentence (e.g., can you clarify this?), which we will refer as “c-questions”. We conducted a comprehensive performance comparison analysis of the proposed multi-channel deep convolutional neural network against other deep neural networks. Furthermore, we evaluated the performance of traditional rule-based and learning-based methods for detecting question sentences. The proposed Quest-CNN achieved the best F1 score both on a dataset of data entry-review dialogue in a dialysis care setting, and on a general domain dataset.

## introduction in healthcare, real-world data (rwd) refers to patient data routinely collected during clinic visits, hospitalization, as well as patient-reported results. in recent years, rwd's volume has become enormous, and invaluable insights and real-world evidence can be generated from these datasets using the latest data processing and analytical techniques. however, rwd's quality remains one of the main challenges that prevent novel machine learning methods from being readily adopted in healthcare. therefore, creating data quality tools is of great importance in health care and health data sciences. erroneous data in healthcare systems could jeopardize a patient's clinical outcomes and affect the care provider's ability to optimize its performance. common data quality issues include missing critical information about medical history, wrong coding of a condition, and inconsistency in documentation across different care sites. manual review by domain experts is the gold standard for achieving the highest data quality but is unattainable in regular care practices. recent developments in the field of natural language processing (nlp) has attracted great interest in the healthcare community since algorithms for identifying variables of interest and classification algorithm for diseases have been recently developed @xcite . in this paper, we presented a novel model for the extraction of queries (questions) in a corpus of dialogue between data entry clinicians and expert reviewers in a multi-site dialysis environment. the main contributions of this work are: (i) to the best of our knowledge, we are the first to benchmark the performance of different rule and learning-based methods for the extraction of question sentences from logs of real-world (medical) systems by providing specific misclassification cases that emphasize the limitation of each technique. (ii) we proposed a new deep neural network architecture, namely quest-cnn which unifies syntactic, semantic and statistical features and is capable of identifying real questions that expect an answer, questions referring to an issue mentioned in a nearby sentence (c-questions) and sentences that are not questions. (iii) we examined the importance of the above mentioned features and we experimented extensively with different state of the art deep learning models in order to determine the best architecture for this particular task. (iv) we investigated the effect of incorporating domain knowledge on the performance of a model by examining whether word embeddings and semantic features (that will be described in section 4.1) which are pre-trained in a domain-specific dataset rather than in a general dataset are more beneficial for the model. finally, in addition to evaluating our model's performance in a medical context, we also experimented in section 5 with a general-domain dataset (questions in the twitter social platform) to show our model's generalizability. the rest of the paper is organized as follows. related work is presented in section 2. the different question detection methods that will be examined, are described in section 3. section 4 details the characteristics of the proposed multi-channel cnn model. finally, the results of the experiments are reported in section 5 and a conclusion and a plan for future work are given in section 6. ## related work different question-detection methods have mainly been focused on the extraction of questions in social online settings (e.g. emails, twitter) @xcite . these methods can be classified into two categories: (i) rule based methods that make use of rules like 5w1h words (what, who, where, when, why, how) or question marks (qm) to identify questions (ii) learning-based methods, which train a classifier based on the patterns of sentences. recently, different deep-neural networks have achieved a state-of-the-art results in text classification. in the kim-cnn model @xcite t filters are applied to the concatenated word embeddings of each document in order to produce t feature maps, which are fed to a max pooling layer, in order to create a t-dimensional representation of the document. in addition, in @xcite the xml-cnn network was introduced, where a dynamic max-pooling scheme and a hidden bottleneck layer were used to achieve a better representation of documents. another state-of-the art deep model is seq-cnn @xcite where each word is represented as a v -dimensional one-hot vector where v is the vocabulary of the dataset and the concatenation of the word vectors are passed through a convolutional layer, followed by a special dynamic pooling layer. furthermore, in fasttext @xcite the embedding of the words that appear in a document were averaged to create a document representation. finally, a comprehensive analysis of clinical-domain embedding methods is presented in @xcite . ## question detection as mentioned above, the task of identifying sentences that contain questions and c-questions can be broken into two sub-problems: (i) detecting question sentences from unstructured data (logs) (ii) correctly identifying the c-questions and the real questions that require an answer. in order to create the corpus of candidate questions, we explored both rule and learning based approaches. we also compared the performance of each method on the task of identifying questions on (medical) unstructured text and analyzed the most common misclassification cases. rule-based approach: in particular, we employed the following rules: (i) the last character of a sentence is a question mark (ii) the rules that were introduced in @xcite (e.g i* [try*,like, need] to [find, know]) (iii) the sentence contains a 5w1h word (iv) the refined 5w1h rules from @xcite : the sentence contains a 5w1h word at its beginning or it contains auxiliary words (e.g. "what is" instead of what or "how does" instead of how) learning-based approach: these approaches learn specific patterns that can be used to identify sentences that are questions. in this study, we evaluated the following methods: (i) a naive bayes classifier which was trained on the nps chat corpus that consists of over 10,000 posts from instant messaging sessions @xcite . as these posts have been labeled with dialogue act types, such as "statement","ynquestion", we used the classifier without any further training. (ii) the syntactic parser from the stanford corenlp natural language processing toolkit @xcite which provides a full syntactic analysis of an input sentence. the question-sentences were identified, by examining the syntactic structure of each sentence in the dataset. ## quest-cnn architecture in this section, we describe our proposed model which has as input all the sentences that the above question extraction method identified as questions and it distinguished between the sentences that are actual questions and c-questions. the architecture of our model is depicted in figure 1 . the key characteristic of our model is its capability of unifying the syntactic properties of questions, the semantic features of words by incorporating domain knowledge and statistical features (section 4.1) in a multi-channel cnn scenario. let x @xmath0. each sentence can be presented as the concatenation of the word embeddings of its words. by using multiple t convolutional filters the model can capture multiple features for each sentence, as each filter @xmath1:i+h-1 (of size h): finally, our model consists of multiple channels that represent different features. each filter i is applied to all the channels and the feature map c i is calculated by adding the results in each channel. in the ablation study (section 5.2.3), we presented the performance of different variations of the model which consist of one to three channels and use (or not) the information of the statistical features. ## experiments in this section, we presented the result of an empirical evaluation on the question-detection methods that were described in section 3 and our proposed model in section 4. in particular, we analyzed the performance of the rule and learning based methods and we provided specific examples that highlighted their limitations. we also provided a comparison between different deep learning text classification methods that were described in section 2 in order to show the efficiency of our proposed model. finally, we conducted an ablation study that demonstrated the importance of the features (channels) for achieving high-accuracy results. datasets. the main dataset used in this study is a review log consists of the query-answer dialogue in the dialysis measurement, analysis and reporting system (dmar ® ). dmar ® is a web-based application that collects patient-level, clinical data within a renal program @xcite and its review module facilitates the "query-answer" style of communication between the reviewer (neurologist) and users (renal coordinators) during routine care process. a user would post a question to the reviewer only when she/he was unsure about the correctness of patient data (for example why the hernia was considered absolute contraindications?'). therefore, the questions in this dialogue dataset provided a good indicator of data quality issues. the dataset used in this study was extracted from dmar ® between 2013-2019. 1 this dataset offered a rare (in terms of quantity) opportunity to examine a vast array of different questions types during medical data review processes over a fairly long period of time in a multi-sites real-world clinical setting. the annotations of the sentences were created by manually checking each sentence, as a ground truth was non existent. unfortunately, due to the sensitive nature of the dataset (contains medical information of patients), we cannot provide a link to a downloadable version of the data without the approval of the research ethics boards. in addition, in order to test our model in a general setting, we experiment with the twitter dataset which was created and analyzed in @xcite . this dataset contains 2462 tweets which were annotated by two human annotators as questions (conveying an information need) or non-questions. this dataset was used to evaluate the models in the binary classification task to separate actual questions from sentences that do not require an answer. for the evaluation of the question detection methods, the accuracy, recall and f1 score (the harmonic mean between recall and precision) were reported. for the evaluation of the deep-learning models, we reported the micro-averaged f1 score and the f1 score for the multi-class and binary-class dataset respectively on the testing and the validation set. ## conclusion and future work in this paper, we have provided an analysis of the performance of existing methods for question extraction with real-world misclassification examples that showed the weak point of each method. furthermore, we have proposed a novel approach for the automatic identification of real questions and c-questions. we have also shown empirically that the proposed architecture of unifying syntactic, semantic and statistical features achieved a state-ofthe-art f1 score for this particular task. finally, we have presented the relevance of exploiting domain knowledge in the overall performance of a model. we are in the process of obtaining access to datasets from different application contexts in order to examine the generalizability of our model. as for future work, we plan to extend our work by calculating the similarity of questions in order to create groups of questions that represent the most impactful "problems" of a given application environment. finally, we plan to compare our model with recent language representation models like the bert model in @xcite both for the task of question identification and for the task of creating the above mentioned "problem" groups.