pwc-archive/papers-with-abstracts huggingface dataset:
Should You Fine-Tune BERT for Automated Essay Scoring?
Authors: Elijah Mayfield, Alan W. black
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Automated Essay Scoring

Most natural language processing research now recommends large Transformer-based models with fine-tuning for supervised classification tasks; older strategies like bag-of-words features and linear models have fallen out of favor. Here we investigate whether, in automated essay scoring (AES) research, deep neural models are an appropriate technological choice. We find that fine-tuning BERT produces similar performance to classical models at significant additional cost. We argue that while state-of-the-art strategies do match existing best results, they come with opportunity costs in computational resources. We conclude with a review of promising areas for research on student essays where the unique characteristics of Transformers may provide benefits over classical methods to justify the costs.
Interpreting Neural CWI Classifiers' Weights as Vocabulary Size
Authors: Yo Ehara
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Complex Word Identification

Complex Word Identification (CWI) is a task for the identification of words that are challenging for second-language learners to read. Even though the use of neural classifiers is now common in CWI, the interpretation of their parameters remains difficult. This paper analyzes neural CWI classifiers and shows that some of their parameters can be interpreted as vocabulary size. We present a novel formalization of vocabulary size measurement methods that are practiced in the applied linguistics field as a kind of neural classifier. We also contribute to building a novel dataset for validating vocabulary testing and readability via crowdsourcing.
Automated Scoring of Clinical Expressive Language Evaluation Tasks
Authors: Yiyi Wang, Emily Prud{'}hommeaux, Meysam Asgari, Jill Dolata
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Diversity, Machine Translation, Sentence, Transfer Learning, Translation, Word Embeddings

Many clinical assessment instruments used to diagnose language impairments in children include a task in which the subject must formulate a sentence to describe an image using a specific target word. Because producing sentences in this way requires the speaker to integrate syntactic and semantic knowledge in a complex manner, responses are typically evaluated on several different dimensions of appropriateness yielding a single composite score for each response. In this paper, we present a dataset consisting of non-clinically elicited responses for three related sentence formulation tasks, and we propose an approach for automatically evaluating their appropriateness. We use neural machine translation to generate correct-incorrect sentence pairs in order to create synthetic data to increase the amount and diversity of training data for our scoring model. Our scoring model uses transfer learning to facilitate automatic sentence appropriateness evaluation. We further compare custom word embeddings with pre-trained contextualized embeddings serving as features for our scoring model. We find that transfer learning improves scoring accuracy, particularly when using pretrained contextualized embeddings.
Context-based Automated Scoring of Complex Mathematical Responses
Authors: Aoife Cahill, James H Fife, Brian Riordan, Avijit Vajpayee, Dmytro Galochkin
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Explainable Models

The tasks of automatically scoring either textual or algebraic responses to mathematical questions have both been well-studied, albeit separately. In this paper we propose a method for automatically scoring responses that contain both text and algebraic expressions. Our method not only achieves high agreement with human raters, but also links explicitly to the scoring rubric {--} essentially providing explainable models and a way to potentially provide feedback to students in the future.
Predicting the Difficulty and Response Time of Multiple Choice Questions Using Transfer Learning
Authors: Kang Xue, Victoria Yaneva, Christopher Runyon, Peter Baldwin
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Multiple-choice, Transfer Learning

This paper investigates whether transfer learning can improve the prediction of the difficulty and response time parameters for 18,000 multiple-choice questions from a high-stakes medical exam. The type the signal that best predicts difficulty and response time is also explored, both in terms of representation abstraction and item component used as input (e.g., whole item, answer options only, etc.). The results indicate that, for our sample, transfer learning can improve the prediction of item difficulty when response time is used as an auxiliary task but not the other way around. In addition, difficulty was best predicted using signal from the item stem (the description of the clinical case), while all parts of the item were important for predicting the response time.
A Comparative Study of Synthetic Data Generation Methods for Grammatical Error Correction
Authors: Max White, Alla Rozovskaya
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Grammatical Error Correction, Machine Translation, Synthetic Data Generation, Translation

Grammatical Error Correction (GEC) is concerned with correcting grammatical errors in written text. Current GEC systems, namely those leveraging statistical and neural machine translation, require large quantities of annotated training data, which can be expensive or impractical to obtain. This research compares techniques for generating synthetic data utilized by the two highest scoring submissions to the restricted and low-resource tracks in the BEA-2019 Shared Task on Grammatical Error Correction.
Quantifying 60 Years of Gender Bias in Biomedical Research with Word Embeddings
Authors: Anthony Rios, Reenam Joshi, Hejin Shin
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Articles, Word Embeddings

Gender bias in biomedical research can have an adverse impact on the health of real people. For example, there is evidence that heart disease-related funded research generally focuses on men. Health disparities can form between men and at-risk groups of women (i.e., elderly and low-income) if there is not an equal number of heart disease-related studies for both genders. In this paper, we study temporal bias in biomedical research articles by measuring gender differences in word embeddings. Specifically, we address multiple questions, including, How has gender bias changed over time in biomedical research, and what health-related concepts are the most biased? Overall, we find that traditional gender stereotypes have reduced over time. However, we also find that the embeddings of many medical conditions are as biased today as they were 60 years ago (e.g., concepts related to drug addiction and body dysmorphia).
Sequence-to-Set Semantic Tagging for Complex Query Reformulation and Automated Text Categorization in Biomedical IR using Self-Attention
Authors: Manirupa Das, Juanxi Li, Eric Fosler-Lussier, Simon Lin, Steve Rust, Yungui Huang, Rajiv Ramnath
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Retrieval, Text Categorization

Novel contexts, comprising a set of terms referring to one or more concepts, may often arise in complex querying scenarios such as in evidence-based medicine (EBM) involving biomedical literature. These may not explicitly refer to entities or canonical concept forms occurring in a fact-based knowledge source, e.g. the UMLS ontology. Moreover, hidden associations between related concepts meaningful in the current context, may not exist within a single document, but across documents in the collection. Predicting semantic concept tags of documents can therefore serve to associate documents related in unseen contexts, or categorize them, in information filtering or retrieval scenarios. Thus, inspired by the success of sequence-to-sequence neural models, we develop a novel sequence-to-set framework with attention, for learning document representations in a unique unsupervised setting, using no human-annotated document labels or external knowledge resources and only corpus-derived term statistics to drive the training, that can effect term transfer within a corpus for semantically tagging a large collection of documents. Our sequence-to-set modeling approach to predict semantic tags, gives to the best of our knowledge, the state-of-the-art for both, an unsupervised query expansion (QE) task for the TREC CDS 2016 challenge dataset when evaluated on an Okapi BM25{--}based document retrieval system; and also over the MLTM system baseline baseline (Soleimani and Miller, 2016), for both supervised and semi-supervised multi-label prediction tasks on the del.icio.us and Ohsumed datasets. We make our code and data publicly available.
Improving Biomedical Analogical Retrieval with Embedding of Structural Dependencies
Authors: Am Paullada, alynne, Bethany Percha, Trevor Cohen
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Retrieval, Word Embeddings

Inferring the nature of the relationships between biomedical entities from text is an important problem due to the difficulty of maintaining human-curated knowledge bases in rapidly evolving fields. Neural word embeddings have earned attention for an apparent ability to encode relational information. However, word embedding models that disregard syntax during training are limited in their ability to encode the structural relationships fundamental to cognitive theories of analogy. In this paper, we demonstrate the utility of encoding dependency structure in word embeddings in a model we call Embedding of Structural Dependencies (ESD) as a way to represent biomedical relationships in two analogical retrieval tasks: a relationship retrieval (RR) task, and a literature-based discovery (LBD) task meant to hypothesize plausible relationships between pairs of entities unseen in training. We compare our model to skip-gram with negative sampling (SGNS), using 19 databases of biomedical relationships as our evaluation data, with improvements in performance on 17 (LBD) and 18 (RR) of these sets. These results suggest embeddings encoding dependency path information are of value for biomedical analogy retrieval.
DeSpin: a prototype system for detecting spin in biomedical publications
Authors: Anna Koroleva, Sanjay Kamath, Patrick Bossuyt, Patrick Paroubek
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Articles, Relation Extraction, Semantic Similarity, Semantic Textual Similarity, Sentence, Sentence Classification

Improving the quality of medical research reporting is crucial to reduce avoidable waste in research and to improve the quality of health care. Despite various initiatives aiming at improving research reporting {--} guidelines, checklists, authoring aids, peer review procedures, etc. {--} overinterpretation of research results, also known as spin, is still a serious issue in research reporting. In this paper, we propose a Natural Language Processing (NLP) system for detecting several types of spin in biomedical articles reporting randomized controlled trials (RCTs). We use a combination of rule-based and machine learning approaches to extract important information on trial design and to detect potential spin. The proposed spin detection system includes algorithms for text structure analysis, sentence classification, entity and relation extraction, semantic similarity assessment. Our algorithms achieved operational performance for the these tasks, F-measure ranging from 79,42 to 97.86{\%} for different tasks. The most difficult task is extracting reported outcomes. Our tool is intended to be used as a semi-automated aid tool for assisting both authors and peer reviewers to detect potential spin. The tool incorporates a simple interface that allows to run the algorithms and visualize their output. It can also be used for manual annotation and correction of the errors in the outputs. The proposed tool is the first tool for spin detection. The tool and the annotated dataset are freely available.
Towards Visual Dialog for Radiology
Authors: Olga Kovaleva, Chaitanya Shivade, Satyan Kashyap, a, Karina Kanjaria, Joy Wu, Deddeh Ballah, Adam Coy, Alex Karargyris, ros, Yufan Guo, David Beymer Beymer, Anna Rumshisky, V Mukherjee, ana Mukherjee
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Question Answering, Visual Dialog, Visual Question Answering, Visual Question Answering (VQA)

Current research in machine learning for radiology is focused mostly on images. There exists limited work in investigating intelligent interactive systems for radiology. To address this limitation, we introduce a realistic and information-rich task of Visual Dialog in radiology, specific to chest X-ray images. Using MIMIC-CXR, an openly available database of chest X-ray images, we construct both a synthetic and a real-world dataset and provide baseline scores achieved by state-of-the-art models. We show that incorporating medical history of the patient leads to better performance in answering questions as opposed to conventional visual question answering model which looks only at the image. While our experiments show promising results, they indicate that the task is extremely challenging with significant scope for improvement. We make both the datasets (synthetic and gold standard) and the associated code publicly available to the research community.
A BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction
Authors: Chen Lin, Timothy Miller, Dmitriy Dligach, Farig Sadeque, Steven Bethard, Guergana Savova
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Multi-Task Learning, Relation, Relation Classification, Relation Extraction, Temporal Relation Extraction

Recently BERT has achieved a state-of-the-art performance in temporal relation extraction from clinical Electronic Medical Records text. However, the current approach is inefficient as it requires multiple passes through each input sequence. We extend a recently-proposed one-pass model for relation classification to a one-pass model for relation extraction. We augment this framework by introducing global embeddings to help with long-distance relation inference, and by multi-task learning to increase model performance and generalizability. Our proposed model produces results on par with the state-of-the-art in temporal relation extraction on the THYME corpus and is much {``}greener{''} in computational cost.
Evaluating the Utility of Model Configurations and Data Augmentation on Clinical Semantic Textual Similarity
Authors: Yuxia Wang, Fei Liu, Karin Verspoor, Timothy Baldwin
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Data Augmentation, Semantic Textual Similarity, STS

In this paper, we apply pre-trained language models to the Semantic Textual Similarity (STS) task, with a specific focus on the clinical domain. In low-resource setting of clinical STS, these large models tend to be impractical and prone to overfitting. Building on BERT, we study the impact of a number of model design choices, namely different fine-tuning and pooling strategies. We observe that the impact of domain-specific fine-tuning on clinical STS is much less than that in the general domain, likely due to the concept richness of the domain. Based on this, we propose two data augmentation techniques. Experimental results on N2C2-STS 1 demonstrate substantial improvements, validating the utility of the proposed methods.
Neural Transduction of Letter Position Dyslexia using an Anagram Matrix Representation
Authors: Avi Bleiweiss
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Position

Research on analyzing reading patterns of dyslectic children has mainly been driven by classifying dyslexia types offline. We contend that a framework to remedy reading errors inline is more far-reaching and will help to further advance our understanding of this impairment. In this paper, we propose a simple and intuitive neural model to reinstate migrating words that transpire in letter position dyslexia, a visual analysis deficit to the encoding of character order within a word. Introduced by the anagram matrix representation of an input verse, the novelty of our work lies in the expansion from one to a two dimensional context window for training. This warrants words that only differ in the disposition of letters to remain interpreted semantically similar in the embedding space. Subject to the apparent constraints of the self-attention transformer architecture, our model achieved a unigram BLEU score of 40.6 on our reconstructed dataset of the Shakespeare sonnets.
Domain Adaptation and Instance Selection for Disease Syndrome Classification over Veterinary Clinical Notes
Authors: Brian Hur, Timothy Baldwin, Karin Verspoor, Laura Hardefeldt, James Gilkerson
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Document Classification, Domain Adaptation, General Classification

Identifying the reasons for antibiotic administration in veterinary records is a critical component of understanding antimicrobial usage patterns. This informs antimicrobial stewardship programs designed to fight antimicrobial resistance, a major health crisis affecting both humans and animals in which veterinarians have an important role to play. We propose a document classification approach to determine the reason for administration of a given drug, with particular focus on domain adaptation from one drug to another, and instance selection to minimize annotation effort.
Multilogue-Net: A Context-Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation
Authors: Aman Shenoy, Ashish Sardana
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Multimodal Sentiment Analysis, Sentiment Analysis

Sentiment Analysis and Emotion Detection in conversation is key in several real-world applications, with an increase in modalities available aiding a better understanding of the underlying emotions. Multi-modal Emotion Detection and Sentiment Analysis can be particularly useful, as applications will be able to use specific subsets of available modalities, as per the available data. Current systems dealing with Multi-modal functionality fail to leverage and capture - the context of the conversation through all modalities, the dependency between the listener(s) and speaker emotional states, and the relevance and relationship between the available modalities. In this paper, we propose an end to end RNN architecture that attempts to take into account all the mentioned drawbacks. Our proposed model, at the time of writing, out-performs the state of the art on a benchmark dataset on a variety of accuracy and regression metrics.
AI Sensing for Robotics using Deep Learning based Visual and Language Modeling
Authors: yuvaram singh, Kameshwar Rao JV
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Image Captioning, Language Modeling, Language Modelling

An artificial intelligence(AI) system should be capable of processing the sensory inputs to extract both task-specific and general information about its environment. However, most of the existing algorithms extract only task specific information. In this work, an innovative approach to address the problem of processing visual sensory data is presented by utilizing convolutional neural network (CNN). It recognizes and represents the physical and semantic nature of the surrounding in both human readable and machine processable format. This work utilizes the image captioning model to capture the semantics of the input image and a modular design to generate a probability distribution for semantic topics. It gives any autonomous system the ability to process visual information in a human-like way and generates more insights which are hardly possible with a conventional algorithm. Here a model and data collection method are proposed.
Improving Intent Classification in an E-commerce Voice Assistant by Using Inter-Utterance Context
Authors: Arpit Sharma
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Classification, General Classification, intent-classification, Intent Classification

In this work, we improve the intent classification in an English based e-commerce voice assistant by using inter-utterance context. For increased user adaptation and hence being more profitable, an e-commerce voice assistant is desired to understand the context of a conversation and not have the users repeat it in every utterance. For example, let a user{'}s first utterance be {`}find apples{'}. Then, the user may say {`}i want organic only{'} to filter out the results generated by an assistant with respect to the first query. So, it is important for the assistant to take into account the context from the user{'}s first utterance to understand her intention in the second one. In this paper, we present our approach for contextual intent classification in Walmart{'}s e-commerce voice assistant. It uses the intent of the previous user utterance to predict the intent of her current utterance. With the help of experiments performed on real user queries we show that our approach improves the intent classification in the assistant.
Semi-Supervised Iterative Approach for Domain-Specific Complaint Detection in Social Media
Authors: Akash Gautam, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7

In this paper, we present a semi-supervised bootstrapping approach to detect product or service related complaints in social media. Our approach begins with a small collection of annotated samples which are used to identify a preliminary set of linguistic indicators pertinent to complaints. These indicators are then used to expand the dataset. The expanded dataset is again used to extract more indicators. This process is applied for several iterations until we can no longer find any new indicators. We evaluated this approach on a Twitter corpus specifically to detect complaints about transportation services. We started with an annotated set of 326 samples of transportation complaints, and after four iterations of the approach, we collected 2,840 indicators and over 3,700 tweets. We annotated a random sample of 700 tweets from the final dataset and observed that nearly half the samples were actual transportation complaints. Lastly, we also studied how different features based on semantics, orthographic properties, and sentiment contribute towards the prediction of complaints.
Item-based Collaborative Filtering with BERT
Authors: Tian Wang, Yuyangzi Fu
Date: 2020-07-01 00:00:00
Proceeding: WS 2020 7
Tasks: Collaborative Filtering, Recommendation Systems

In e-commerce, recommender systems have become an indispensable part of helping users explore the available inventory. In this work, we present a novel approach for item-based collaborative filtering, by leveraging BERT to understand items, and score relevancy between different items. Our proposed method could address problems that plague traditional recommender systems such as cold start, and {``}more of the same{''} recommended content. We conducted experiments on a large-scale real-world dataset with full cold-start scenario, and the proposed approach significantly outperforms the popular Bi-LSTM model.