pwc-archive/papers-with-abstracts huggingface dataset:
NMF Ensembles? Not for Text Summarization!
Authors: Alka Khurana, Vasudha Bhatnagar
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Text Summarization

Non-negative Matrix Factorization (NMF) has been used for text analytics with promising results. Instability of results arising due to stochastic variations during initialization makes a case for use of ensemble technology. However, our extensive empirical investigation indicates otherwise. In this paper, we establish that ensemble summary for single document using NMF is no better than the best base model summary.
If You Build Your Own NER Scorer, Non-replicable Results Will Come
Authors: Constantine Lignos, Marjan Kamyab
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

We attempt to replicate a named entity recognition (NER) model implemented in a popular toolkit and discover that a critical barrier to doing so is the inconsistent evaluation of improper label sequences. We define these sequences and examine how two scorers differ in their handling of them, finding that one approach produces F1 scores approximately 0.5 points higher on the CoNLL 2003 English development and test sets. We propose best practices to increase the replicability of NER evaluations by increasing transparency regarding the handling of improper label sequences.
Embedding Structured Dictionary Entries
Authors: Steven Wilson, Walid Magdy, Barbara McGillivray, Gareth Tyson
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Learning Word Embeddings, Multi-Task Learning, Word Embeddings

Previous work has shown how to effectively use external resources such as dictionaries to improve English-language word embeddings, either by manipulating the training process or by applying post-hoc adjustments to the embedding space. We experiment with a multi-task learning approach for explicitly incorporating the structured elements of dictionary entries, such as user-assigned tags and usage examples, when learning embeddings for dictionary headwords. Our work generalizes several existing models for learning word embeddings from dictionaries. However, we find that the most effective representations overall are learned by simply training with a skip-gram objective over the concatenated text of all entries in the dictionary, giving no particular focus to the structure of the entries.
Interpretability Rules: Jointly Bootstrapping a Neural Relation Extractorwith an Explanation Decoder
Authors: Zheng Tang, Mihai Surdeanu
Date: 2021-06-01 00:00:00
Proceeding: NAACL (TrustNLP) 2021 6
Tasks: Decoder, Relation, Relation Extraction

We introduce a method that transforms a rule-based relation extraction (RE) classifier into a neural one such that both interpretability and performance are achieved. Our approach jointly trains a RE classifier with a decoder that generates explanations for these extractions, using as sole supervision a set of rules that match these relations. Our evaluation on the TACRED dataset shows that our neural RE classifier outperforms the rule-based one we started from by 9 F1 points; our decoder generates explanations with a high BLEU score of over 90%; and, the joint learning improves the performance of both the classifier and decoder.
Measuring Biases of Word Embeddings: What Similarity Measures and Descriptive Statistics to Use?
Authors: Hossein Azarpanah, Mohsen Farhadloo
Date: 2021-06-01 00:00:00
Proceeding: NAACL (TrustNLP) 2021 6
Tasks: Descriptive, Word Embeddings

Word embeddings are widely used in Natural Language Processing (NLP) for a vast range of applications. However, it has been consistently proven that these embeddings reflect the same human biases that exist in the data used to train them. Most of the introduced bias indicators to reveal word embeddings’ bias are average-based indicators based on the cosine similarity measure. In this study, we examine the impacts of different similarity measures as well as other descriptive techniques than averaging in measuring the biases of contextual and non-contextual word embeddings. We show that the extent of revealed biases in word embeddings depends on the descriptive statistics and similarity measures used to measure the bias. We found that over the ten categories of word embedding association tests, Mahalanobis distance reveals the smallest bias, and Euclidean distance reveals the largest bias in word embeddings. In addition, the contextual models reveal less severe biases than the non-contextual word embedding models.
Private Release of Text Embedding Vectors
Authors: Oluwaseyi Feyisetan, Shiva Kasiviswanathan
Date: 2021-06-01 00:00:00
Proceeding: NAACL (TrustNLP) 2021 6
Tasks: Privacy Preserving

Ensuring strong theoretical privacy guarantees on text data is a challenging problem which is usually attained at the expense of utility. However, to improve the practicality of privacy preserving text analyses, it is essential to design algorithms that better optimize this tradeoff. To address this challenge, we propose a release mechanism that takes any (text) embedding vector as input and releases a corresponding private vector. The mechanism satisfies an extension of differential privacy to metric spaces. Our idea based on first randomly projecting the vectors to a lower-dimensional space and then adding noise in this projected space generates private vectors that achieve strong theoretical guarantees on its utility. We support our theoretical proofs with empirical experiments on multiple word embedding models and NLP datasets, achieving in some cases more than 10% gains over the existing state-of-the-art privatization techniques.
xER: An Explainable Model for Entity Resolution using an Efficient Solution for the Clique Partitioning Problem
Authors: Samhita Vadrevu, Rakesh Nagi, JinJun Xiong, Wen-mei Hwu
Date: 2021-06-01 00:00:00
Proceeding: NAACL (TrustNLP) 2021 6
Tasks: Entity Resolution, graph partitioning

In this paper, we propose a global, self- explainable solution to solve a prominent NLP problem: Entity Resolution (ER). We formu- late ER as a graph partitioning problem. Every mention of a real-world entity is represented by a node in the graph, and the pairwise sim- ilarity scores between the mentions are used to associate these nodes to exactly one clique, which represents a real-world entity in the ER domain. In this paper, we use Clique Partition- ing Problem (CPP), which is an Integer Pro- gram (IP) to formulate ER as a graph partition- ing problem and then highlight the explainable nature of this method. Since CPP is NP-Hard, we introduce an efficient solution procedure, the xER algorithm, to solve CPP as a combi- nation of finding maximal cliques in the graph and then performing generalized set packing using a novel formulation. We discuss the advantages of using xER over the traditional methods and provide the computational exper- iments and results of applying this method to ER data sets.
Gender Bias in Natural Language Processing Across Human Languages
Authors: Abigail Matthews, Isabella Grasso, Christopher Mahoney, Yan Chen, Esma Wali, Thomas Middleton, Mariama Njie, Jeanna Matthews
Date: 2021-06-01 00:00:00
Proceeding: NAACL (TrustNLP) 2021 6
Tasks: Decision Making

Natural Language Processing (NLP) systems are at the heart of many critical automated decision-making systems making crucial recommendations about our future world. Gender bias in NLP has been well studied in English, but has been less studied in other languages. In this paper, a team including speakers of 9 languages - Chinese, Spanish, English, Arabic, German, French, Farsi, Urdu, and Wolof - reports and analyzes measurements of gender bias in the Wikipedia corpora for these 9 languages. We develop extensions to profession-level and corpus-level gender bias metric calculations originally designed for English and apply them to 8 other languages, including languages that have grammatically gendered nouns including different feminine, masculine, and neuter profession words. We discuss future work that would benefit immensely from a computational linguistics perspective.
Interpreting Text Classifiers by Learning Context-sensitive Influence of Words
Authors: Sawan Kumar, Kalpit Dixit, Kashif Shah
Date: 2021-06-01 00:00:00
Proceeding: NAACL (TrustNLP) 2021 6
Tasks: Sentiment Analysis, text-classification, Text Classification

Many existing approaches for interpreting text classification models focus on providing importance scores for parts of the input text, such as words, but without a way to test or improve the interpretation method itself. This has the effect of compounding the problem of understanding or building trust in the model, with the interpretation method itself adding to the opacity of the model. Further, importance scores on individual examples are usually not enough to provide a sufficient picture of model behavior. To address these concerns, we propose MOXIE (MOdeling conteXt-sensitive InfluencE of words) with an aim to enable a richer interface for a user to interact with the model being interpreted and to produce testable predictions. In particular, we aim to make predictions for importance scores, counterfactuals and learned biases with MOXIE. In addition, with a global learning objective, MOXIE provides a clear path for testing and improving itself. We evaluate the reliability and efficiency of MOXIE on the task of sentiment analysis.
Various Approaches for Predicting Stroke Prognosis using Magnetic Resonance Imaging Text Records
Authors: Tak-Sung Heo, Chulho Kim, Jeong-Myeong Choi, Yeong-Seok Jeong, Yu-Seop Kim
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Prognosis, Sentence

Stroke is one of the leading causes of death and disability worldwide. Stroke is treatable, but it is prone to disability after treatment and must be prevented. To grasp the degree of disability caused by stroke, we use magnetic resonance imaging text records to predict stroke and measure the performance according to the document-level and sentence-level representation. As a result of the experiment, the document-level representation shows better performance.
Incorporating Risk Factor Embeddings in Pre-trained Transformers Improves Sentiment Prediction in Psychiatric Discharge Summaries
Authors: Xiyu Ding, Mei-Hua Hall, Timothy Miller
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11

Reducing rates of early hospital readmission has been recognized and identified as a key to improve quality of care and reduce costs. There are a number of risk factors that have been hypothesized to be important for understanding re-admission risk, including such factors as problems with substance abuse, ability to maintain work, relations with family. In this work, we develop Roberta-based models to predict the sentiment of sentences describing readmission risk factors in discharge summaries of patients with psychosis. We improve substantially on previous results by a scheme that shares information across risk factors while also allowing the model to learn risk factor-specific information.
Evaluation of Transfer Learning for Adverse Drug Event (ADE) and Medication Entity Extraction
Authors: Sankaran Narayanan, Kaivalya Mannam, Sreeranga P Rajan, P Venkat Rangan
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Language Modeling, Language Modelling, Sentence, Transfer Learning

We evaluate several biomedical contextual embeddings (based on BERT, ELMo, and Flair) for the detection of medication entities such as Drugs and Adverse Drug Events (ADE) from Electronic Health Records (EHR) using the 2018 ADE and Medication Extraction (Track 2) n2c2 data-set. We identify best practices for transfer learning, such as language-model fine-tuning and scalar mix. Our transfer learning models achieve strong performance in the overall task (F1=92.91%) as well as in ADE identification (F1=53.08%). Flair-based embeddings out-perform in the identification of context-dependent entities such as ADE. BERT-based embeddings out-perform in recognizing clinical terminology such as Drug and Form entities. ELMo-based embeddings deliver competitive performance in all entities. We develop a sentence-augmentation method for enhanced ADE identification benefiting BERT-based and ELMo-based models by up to 3.13% in F1 gains. Finally, we show that a simple ensemble of these models out-paces most current methods in ADE extraction (F1=55.77%).
BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition
Authors: Elisa Terumi Rubel Schneider, João Vitor Andrioli de Souza, Julien Knafou, Lucas Emanuel Silva e Oliveira, Jenny Copara, Yohan Bonescki Gumiel, Lucas Ferro Antunes de Oliveira, Emerson Cabrera Paraiso, Douglas Teodoro, Cláudia Maria Cabral Moro Barra
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Language Modeling, Language Modelling, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Transfer Learning

With the growing number of electronic health record data, clinical NLP tasks have become increasingly relevant to unlock valuable information from unstructured clinical text. Although the performance of downstream NLP tasks, such as named-entity recognition (NER), in English corpus has recently improved by contextualised language models, less research is available for clinical texts in low resource languages. Our goal is to assess a deep contextual embedding model for Portuguese, so called BioBERTpt, to support clinical and biomedical NER. We transfer learned information encoded in a multilingual-BERT model to a corpora of clinical narratives and biomedical-scientific papers in Brazilian Portuguese. To evaluate the performance of BioBERTpt, we ran NER experiments on two annotated corpora containing clinical narratives and compared the results with existing BERT models. Our in-domain model outperformed the baseline model in F1-score by 2.72%, achieving higher performance in 11 out of 13 assessed entities. We demonstrate that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. The transfer learning process enhanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model.
Classification of Syncope Cases in Norwegian Medical Records
Authors: Ildiko Pilan, Pål H. Brekke, Fredrik A. Dahl, Tore Gundersen, Haldor Husby, Øystein Nytrø, Lilja Øvrelid
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Classification, Prognosis

Loss of consciousness, so-called syncope, is a commonly occurring symptom associated with worse prognosis for a number of heart-related diseases. We present a comparison of methods for a diagnosis classification task in Norwegian clinical notes, targeting syncope, i.e. fainting cases. We find that an often neglected baseline with keyword matching constitutes a rather strong basis, but more advanced methods do offer some improvement in classification performance, especially a convolutional neural network model. The developed pipeline is planned to be used for quantifying unregistered syncope cases in Norway.
Comparison of Machine Learning Methods for Multi-label Classification of Nursing Education and Licensure Exam Questions
Authors: John Langton, Krishna Srihasam, Junlin Jiang
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Multi Label Text Classification, Multi-Label Text Classification, text-classification, Text Classification

In this paper, we evaluate several machine learning methods for multi-label classification of text questions. Every nursing student in the United States must pass the National Council Licensure Examination (NCLEX) to begin professional practice. NCLEX defines a number of competencies on which students are evaluated. By labeling test questions with NCLEX competencies, we can score students according to their performance in each competency. This information helps instructors measure how prepared students are for the NCLEX, as well as which competencies they may need help with. A key challenge is that questions may be related to more than one competency. Labeling questions with NCLEX competencies, therefore, equates to a multi-label, text classification problem where each competency is a label. Here we present an evaluation of several methods to support this use case along with a proposed approach. While our work is grounded in the nursing education domain, the methods described here can be used for any multi-label, text classification use case.
Automatic recognition of abdominal lymph nodes from clinical text
Authors: Yifan Peng, SungWon Lee, Daniel C. Elton, Thomas Shen, Yu-Xing Tang, Qingyu Chen, Shuai Wang, Yingying Zhu, Ronald Summers, Zhiyong Lu
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11

Lymph node status plays a pivotal role in the treatment of cancer. The extraction of lymph nodes from radiology text reports enables large-scale training of lymph node detection on MRI. In this work, we first propose an ontology of 41 types of abdominal lymph nodes with a hierarchical relationship. We then introduce an end-to-end approach based on the combination of rules and transformer-based methods to detect these abdominal lymph node mentions and classify their types from the MRI radiology reports. We demonstrate the superior performance of a model fine-tuned on MRI reports using BlueBERT, called MriBERT. We find that MriBERT outperforms the rule-based labeler (0.957 vs 0.644 in micro weighted F1-score) as well as other BERT-based variations (0.913 - 0.928). We make the code and MriBERT publicly available at https://github.com/ncbi-nlp/bluebert, with the hope that this method can facilitate the development of medical report annotators to produce labels from scratch at scale.
How You Ask Matters: The Effect of Paraphrastic Questions to BERT Performance on a Clinical SQuAD Dataset
Authors: Sungrim (Riea) Moon, Jungwei Fan
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Question Answering, Reading Comprehension

Reading comprehension style question-answering (QA) based on patient-specific documents represents a growing area in clinical NLP with plentiful applications. Bidirectional Encoder Representations from Transformers (BERT) and its derivatives lead the state-of-the-art accuracy on the task, but most evaluation has treated the data as a pre-mixture without systematically looking into the potential effect of imperfect train/test questions. The current study seeks to address this gap by experimenting with full versus partial train/test data consisting of paraphrastic questions. Our key findings include 1) training with all pooled question variants yielded best accuracy, 2) the accuracy varied widely, from 0.74 to 0.80, when trained with each single question variant, and 3) questions of similar lexical/syntactic structure tended to induce identical answers. The results suggest that how you ask questions matters in BERT-based QA, especially at the training stage.
Relative and Incomplete Time Expression Anchoring for Clinical Text
Authors: Louise Dupuis, Nicol Bergou, Hegler Tissot, Sumithra Velupillai
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: Classification, Relation Classification

Extracting and modeling temporal information in clinical text is an important element for developing timelines and disease trajectories. Time information in written text varies in preciseness and explicitness, posing challenges for NLP approaches that aim to accurately anchor temporal information on a timeline. Relative and incomplete time expressions (RI-Timexes) are expressions that require additional information for their temporal anchor to be resolved, but few studies have addressed this challenge specifically. In this study, we aimed to reproduce and verify a classification approach for identifying anchor dates and relations in clinical text, and propose a novel relation classification approach for this task.
Assessment of DistilBERT performance on Named Entity Recognition task for the detection of Protected Health Information and medical concepts
Authors: Macarious Abadeer
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

Bidirectional Encoder Representations from Transformers (BERT) models achieve state-of-the-art performance on a number of Natural Language Processing tasks. However, their model size on disk often exceeds 1 GB and the process of fine-tuning them and using them to run inference consumes significant hardware resources and runtime. This makes them hard to deploy to production environments. This paper fine-tunes DistilBERT, a lightweight deep learning model, on medical text for the named entity recognition task of Protected Health Information (PHI) and medical concepts. This work provides a full assessment of the performance of DistilBERT in comparison with BERT models that were pre-trained on medical text. For Named Entity Recognition task of PHI, DistilBERT achieved almost the same results as medical versions of BERT in terms of F1 score at almost half the runtime and consuming approximately half the disk space. On the other hand, for the detection of medical concepts, DistilBERT’s F1 score was lower by 4 points on average than medical BERT variants.
Distinguishing between Dementia with Lewy bodies (DLB) and Alzheimer’s Disease (AD) using Mental Health Records: a Classification Approach
Authors: Zixu Wang, Julia Ive, Sinead Moylett, Christoph Mueller, Rudolf Cardinal, Sumithra Velupillai, John O’Brien, Robert Stewart
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: text-classification, Text Classification

While Dementia with Lewy Bodies (DLB) is the second most common type of neurodegenerative dementia following Alzheimer’s Disease (AD), it is difficult to distinguish from AD. We propose a method for DLB detection by using mental health record (MHR) documents from a (3-month) period before a patient has been diagnosed with DLB or AD. Our objective is to develop a model that could be clinically useful to differentiate between DLB and AD across datasets from different healthcare institutions. We cast this as a classification task using Convolutional Neural Network (CNN), an efficient neural model for text classification. We experiment with different representation models, and explore the features that contribute to model performances. In addition, we apply temperature scaling, a simple but efficient model calibration method, to produce more reliable predictions. We believe the proposed method has important potential for clinical applications using routine healthcare records, and for generalising to other relevant clinical record datasets. To the best of our knowledge, this is the first attempt to distinguish DLB from AD using mental health records, and to improve the reliability of DLB predictions.