pwc-archive/papers-with-abstracts huggingface dataset:
Analyzing Linguistic Complexity and Accuracy in Academic Language Development of German across Elementary and Secondary School
Authors: Zarah Weiss, Detmar Meurers
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Sentence

We track the development of writing complexity and accuracy in German students{'} early academic language development from first to eighth grade. Combining an empirically broad approach to linguistic complexity with the high-quality error annotation included in the Karlsruhe Children{'}s Text corpus (Lavalley et al. 2015) used, we construct models of German academic language development that successfully identify the student{'}s grade level. We show that classifiers for the early years rely more on accuracy development, whereas development in secondary school is better characterized by increasingly complex language in all domains: linguistic system, language use, and human sentence processing characteristics. We demonstrate the generalizability and robustness of models using such a broad complexity feature set across writing topics.
Content Modeling for Automated Oral Proficiency Scoring System
Authors: Su-Youn Yoon, Chong MIn Lee
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

We developed an automated oral proficiency scoring system for non-native English speakers{'} spontaneous speech. Automated systems that score holistic proficiency are expected to assess a wide range of performance categories, and the content is one of the core performance categories. In order to assess the quality of the content, we trained a Siamese convolutional neural network (CNN) to model the semantic relationship between key points generated by experts and a test response. The correlation between human scores and Siamese CNN scores was comparable to human-human agreement (r=0.63), and it was higher than the baseline content features. The inclusion of Siamese CNN-based feature to the existing state-of-the-art automated scoring model achieved a small but statistically significant improvement. However, the new model suffered from score inflation for long atypical responses with serious content issues. We investigated the reasons of this score inflation by analyzing the associations with linguistic features and identifying areas strongly associated with the score errors.
Learning Outcomes and Their Relatedness in a Medical Curriculum
Authors: Sneha Mondal, Tejas Dhamecha, Shantanu Godbole, Smriti Pathak, Red Mendoza, K Gayathri Wijayarathna, Nabil Zary, Swarnadeep Saha, Malolan Chetlur
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

A typical medical curriculum is organized in a hierarchy of instructional objectives called Learning Outcomes (LOs); a few thousand LOs span five years of study. Gaining a thorough understanding of the curriculum requires learners to recognize and apply related LOs across years, and across different parts of the curriculum. However, given the large scope of the curriculum, manually labeling related LOs is tedious, and almost impossible to scale. In this paper, we build a system that learns relationships between LOs, and we achieve up to human-level performance in the LO relationship extraction task. We then present an application where the proposed system is employed to build a map of related LOs and Learning Resources (LRs) pertaining to a virtual patient case. We believe that our system can help medical students grasp the curriculum better, within classroom as well as in Intelligent Tutoring Systems (ITS) settings.
Measuring text readability with machine comprehension: a pilot study
Authors: Marc Benzahra, Fran{\c{c}}ois Yvon
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Reading Comprehension

This article studies the relationship between text readability indice and automatic machine understanding systems. Our hypothesis is that the simpler a text is, the better it should be understood by a machine. We thus expect to a strong correlation between readability levels on the one hand, and performance of automatic reading systems on the other hand. We test this hypothesis with several understanding systems based on language models of varying strengths, measuring this correlation on two corpora of journalistic texts. Our results suggest that this correlation is rather small that existing comprehension systems are far to reproduce the gradual improvement of their performance on texts of decreasing complexity.
Metaphors in Text Simplification: To change or not to change, that is the question
Authors: Yulia Clausen, Vivi Nastase
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Text Simplification

We present an analysis of metaphors in news text simplification. Using features that capture general and metaphor specific characteristics, we test whether we can automatically identify which metaphors will be changed or preserved, and whether there are features that have different predictive power for metaphors or literal words. The experiments show that the Age of Acquisition is the most distinctive feature for both metaphors and literal words. Features that capture Imageability and Concreteness are useful when used alone, but within the full set of features they lose their impact. Frequency of use seems to be the best feature to differentiate metaphors that should be changed and those to be preserved.
Equity Beyond Bias in Language Technologies for Education
Authors: Elijah Mayfield, Michael Madaio, Shrimai Prabhumoye, David Gerritsen, Brittany McLaughlin, Ezekiel Dixon-Rom{\'a}n, Alan W. black
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Automated Essay Scoring, BIG-bench Machine Learning, Fairness, Sentiment Analysis

There is a long record of research on equity in schools. As machine learning researchers begin to study fairness and bias in earnest, language technologies in education have an unusually strong theoretical and applied foundation to build on. Here, we introduce concepts from culturally relevant pedagogy and other frameworks for teaching and learning, identifying future work on equity in NLP. We present case studies in a range of topics like intelligent tutoring systems, computer-assisted language learning, automated essay scoring, and sentiment analysis in classrooms, and provide an actionable agenda for research.
Equipping Educational Applications with Domain Knowledge
Authors: Tarek Sakakini, Hongyu Gong, Jong Yoon Lee, Robert Schloss, JinJun Xiong, Suma Bhat
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Distractor Generation, Language Modeling, Language Modelling, Word Embeddings

One of the challenges of building natural language processing (NLP) applications for education is finding a large domain-specific corpus for the subject of interest (e.g., history or science). To address this challenge, we propose a tool, Dexter, that extracts a subject-specific corpus from a heterogeneous corpus, such as Wikipedia, by relying on a small seed corpus and distributed document representations. We empirically show the impact of the generated corpus on language modeling, estimating word embeddings, and consequently, distractor generation, resulting in better performances than while using a general domain corpus, a heuristically constructed domain-specific corpus, and a corpus generated by a popular system: BootCaT.
Automated Essay Scoring with Discourse-Aware Neural Models
Authors: Farah Nadeem, Huy Nguyen, Yang Liu, Mari Ostendorf
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Automated Essay Scoring, Feature Engineering

Automated essay scoring systems typically rely on hand-crafted features to predict essay quality, but such systems are limited by the cost of feature engineering. Neural networks offer an alternative to feature engineering, but they typically require more annotated data. This paper explores network structures, contextualized embeddings and pre-training strategies aimed at capturing discourse characteristics of essays. Experiments on three essay scoring tasks show benefits from all three strategies in different combinations, with simpler architectures being more effective when less training data is available.
Modeling language learning using specialized Elo rating
Authors: Jue Hou, Koppatz Maximilian, Jos{\'e} Mar{\'\i}a Hoya Quecedo, Nataliya Stoyanova, Roman Yangarber
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Automatic assessment of the proficiency levels of the learner is a critical part of Intelligent Tutoring Systems. We present methods for assessment in the context of language learning. We use a specialized Elo formula used in conjunction with educational data mining. We simultaneously obtain ratings for the proficiency of the learners and for the difficulty of the linguistic concepts that the learners are trying to master. From the same data we also learn a graph structure representing a domain model capturing the relations among the concepts. This application of Elo provides ratings for learners and concepts which correlate well with subjective proficiency levels of the learners and difficulty levels of the concepts.
Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays
Authors: Yanjun Gao, Alex Driban, Brennan Xavier McManus, Elena Musi, Patricia Davies, Smar Muresan, a, Rebecca J. Passonneau
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

We present a unique dataset of student source-based argument essays to facilitate research on the relations between content, argumentation skills, and assessment. Two classroom writing assignments were given to college students in a STEM major, accompanied by a carefully designed rubric. The paper presents a reliability study of the rubric, showing it to be highly reliable, and initial annotation on content and argumentation annotation of the essays.
Segmentation of Argumentative Texts with Contextualised Word Representations
Authors: Georgios Petasis
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Argument Mining, Contextualised Word Representations, Feature Engineering

The segmentation of argumentative units is an important subtask of argument mining, which is frequently addressed at a coarse granularity, usually assuming argumentative units to be no smaller than sentences. Approaches focusing at the clause-level granularity, typically address the task as sequence labeling at the token level, aiming to classify whether a token begins, is inside, or is outside of an argumentative unit. Most approaches exploit highly engineered, manually constructed features, and algorithms typically used in sequential tagging {--} such as Conditional Random Fields, while more recent approaches try to exploit manually constructed features in the context of deep neural networks. In this context, we examined to what extend recent advances in sequential labelling allow to reduce the need for highly sophisticated, manually constructed features, and whether limiting features to embeddings, pre-trained on large corpora is a promising approach. Evaluation results suggest the examined models and approaches can exhibit comparable performance, minimising the need for feature engineering.
A Cascade Model for Proposition Extraction in Argumentation
Authors: Yohan Jo, Jacky Visser, Chris Reed, Eduard Hovy
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Argument Mining, Segmentation, Text Segmentation

We present a model to tackle a fundamental but understudied problem in computational argumentation: proposition extraction. Propositions are the basic units of an argument and the primary building blocks of most argument mining systems. However, they are usually substituted by argumentative discourse units obtained via surface-level text segmentation, which may yield text segments that lack semantic information necessary for subsequent argument mining processes. In contrast, our cascade model aims to extract complete propositions by handling anaphora resolution, text segmentation, reported speech, questions, imperatives, missing subjects, and revision. We formulate each task as a computational problem and test various models using a corpus of the 2016 U.S. presidential debates. We show promising performance for some tasks and discuss main challenges in proposition extraction.
Aligning Discourse and Argumentation Structures using Subtrees and Redescription Mining
Authors: Laurine Huber, Yannick Toussaint, Charlotte Roze, Mathilde Dargnat, Chlo{\'e} Braud
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

In this paper, we investigate similarities between discourse and argumentation structures by aligning subtrees in a corpus containing both annotations. Contrary to previous works, we focus on comparing sub-structures and not only relations matches. Using data mining techniques, we show that discourse and argumentation most often align well, and the double annotation allows to derive a mapping between structures. Moreover, this approach enables the study of similarities between discourse structures and differences in their expressive power.
Transferring Knowledge from Discourse to Arguments: A Case Study with Scientific Abstracts
Authors: Pablo Accuosto, Horacio Saggion
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Argument Mining, Discourse Parsing, Transfer Learning

In this work we propose to leverage resources available with discourse-level annotations to facilitate the identification of argumentative components and relations in scientific texts, which has been recognized as a particularly challenging task. In particular, we implement and evaluate a transfer learning approach in which contextualized representations learned from discourse parsing tasks are used as input of argument mining models. As a pilot application, we explore the feasibility of using automatically identified argumentative components and relations to predict the acceptance of papers in computer science venues. In order to conduct our experiments, we propose an annotation scheme for argumentative units and relations and use it to enrich an existing corpus with an argumentation layer.
The Swedish PoliGraph: A Semantic Graph for Argument Mining of Swedish Parliamentary Data
Authors: Stian R{\o}dven Eide
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Argument Mining, Entity Resolution, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

As part of a larger project on argument mining of Swedish parliamentary data, we have created a semantic graph that, together with named entity recognition and resolution (NER), should make it easier to establish connections between arguments in a given debate. The graph is essentially a semantic database that keeps track of Members of Parliament (MPs), in particular their presence in the parliament and activity in debates, but also party affiliation and participation in commissions. The hope is that the Swedish PoliGraph will enable us to perform named entity resolution on debates in the Swedish parliament with a high accuracy, with the aim of determining to whom an argument is directed.
Lexicon Guided Attentive Neural Network Model for Argument Mining
Authors: Jian-Fu Lin, Kuo Yu Huang, Hen-Hsen Huang, Hsin-Hsi Chen
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Argument Mining, Sentiment Analysis

Identification of argumentative components is an important stage of argument mining. Lexicon information is reported as one of the most frequently used features in the argument mining research. In this paper, we propose a methodology to integrate lexicon information into a neural network model by attention mechanism. We conduct experiments on the UKP dataset, which is collected from heterogeneous sources and contains several text types, e.g., microblog, Wikipedia, and news. We explore lexicons from various application scenarios such as sentiment analysis and emotion detection. We also compare the experimental results of leveraging different lexicons.
Argument Component Classification by Relation Identification by Neural Network and TextRank
Authors: Mamoru Deguchi, Kazunori Yamaguchi
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Articles, Classification, Component Classification, Document Classification, General Classification, Relation, Sentence

In recent years, argumentation mining, which automatically extracts the structure of argumentation from unstructured documents such as essays and debates, is gaining attention. For argumentation mining applications, argument-component classification is an important subtask. The existing methods can be classified into supervised methods and unsupervised methods. Supervised document classification performs classification using a single sentence without relying on the whole document. On the other hand, unsupervised document classification has the advantage of being able to use the whole document, but accuracy of these methods is not so high. In this paper, we propose a method for argument-component classification that combines relation identification by neural networks and TextRank to integrate relation informations (i.e. the strength of the relation). This method can use argumentation-specific knowledge by employing a supervised learning on a corpus while maintaining the advantage of using the whole document. Experiments on two corpora, one consisting of student essay and the other of Wikipedia articles, show the effectiveness of this method.
Argumentative Evidences Classification and Argument Scheme Detection Using Tree Kernels
Authors: Davide Liga
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: General Classification

The purpose of this study is to deploy a novel methodology for classifying different argumentative support (supporting evidences) in arguments, without considering the context. The proposed methodology is based on the idea that the use of Tree Kernel algorithms can be a good way to discriminate between different types of argumentative stances without the need of highly engineered features. This can be useful in different Argumentation Mining sub-tasks. This work provides an example of classifier built using a Tree Kernel method, which can discriminate between different kinds of argumentative support with a high accuracy. The ability to distinguish different kinds of support is, in fact, a key step toward Argument Scheme classification.
The Utility of Discourse Parsing Features for Predicting Argumentation Structure
Authors: Freya Hewett, Roshan Prakash Rane, Nina Harlacher, Manfred Stede
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Discourse Parsing

Research on argumentation mining from text has frequently discussed relationships to discourse parsing, but few empirical results are available so far. One corpus that has been annotated in parallel for argumentation structure and for discourse structure (RST, SDRT) are the {`}argumentative microtexts{'} (Peldszus and Stede, 2016a). While results on perusing the gold RST annotations for predicting argumentation have been published (Peldszus and Stede, 2016b), the step to automatic discourse parsing has not yet been taken. In this paper, we run various discourse parsers (RST, PDTB) on the corpus, compare their results to the gold annotations (for RST) and then assess the contribution of automatically-derived discourse features for argumentation parsing. After reproducing the state-of-the-art Evidence Graph model from Afantenos et al. (2018) for the microtexts, we find that PDTB features can indeed improve its performance.
Detecting Argumentative Discourse Acts with Linguistic Alignment
Authors: Timothy Niven, Hung-Yu Kao
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

We report the results of preliminary investigations into the relationship between linguistic alignment and dialogical argumentation at the level of discourse acts. We annotated a proof of concept dataset with illocutions and transitions at the comment level based on Inference Anchoring Theory. We estimated linguistic alignment across discourse acts and found significant variation. Alignment features calculated at the dyad level are found to be useful for detecting a range of argumentative discourse acts.
Annotation of Rhetorical Moves in Biochemistry Articles
Authors: Mohammed Alliheedi, Robert E. Mercer, Robin Cohen
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Articles, Descriptive

This paper focuses on the real world application of scientific writing and on determining rhetorical moves, an important step in establishing the argument structure of biomedical articles. Using the observation that the structure of scholarly writing in laboratory-based experimental sciences closely follows laboratory procedures, we examine most closely the Methods section of the texts and adopt an approach of identifying rhetorical moves that are procedure-oriented. We also propose a verb-centric frame semantics with an effective set of semantic roles in order to support the analysis. These components are designed to support a computational model that extends a promising proposal of appropriate rhetorical moves for this domain, but one which is merely descriptive. Our work also contributes to the understanding of argument-related annotation schemes. In particular, we conduct a detailed study with human annotators to confirm that our selection of semantic roles is effective in determining the underlying rhetorical structure of existing biomedical articles in an extensive dataset. The annotated dataset that we produce provides the important knowledge needed for our ultimate goal of analyzing biochemistry articles.
Evaluation of Scientific Elements for Text Similarity in Biomedical Publications
Authors: Mariana Neves, Daniel Butzke, Barbara Grune
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: text similarity

Rhetorical elements from scientific publications provide a more structured view of the document and allow algorithms to focus on particular parts of the text. We surveyed the literature for previously proposed schemes for rhetorical elements and present an overview of its current state of the art. We also searched for available tools using these schemes and applied four tools for our particular task of ranking biomedical abstracts based on text similarity. Comparison of the tools with two strong baselines shows that the predictions provided by the ArguminSci tool can support our use case of mining alternative methods for animal experiments.
Ranking Passages for Argument Convincingness
Authors: Peter Potash, Adam Ferguson, Timothy J. Hazen
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

In data ranking applications, pairwise annotation is often more consistent than cardinal annotation for learning ranking models. We examine this in a case study on ranking text passages for argument convincingness. Our task is to choose text passages that provide the highest-quality, most-convincing arguments for opposing sides of a topic. Using data from a deployed system within the Bing search engine, we construct a pairwise-labeled dataset for argument convincingness that is substantially more comprehensive in topical coverage compared to existing public resources. We detail the process of extracting topical passages for queries submitted to a search engine, creating annotated sets of passages aligned to different stances on a topic, and assessing argument convincingness of passages using pairwise annotation. Using a state-of-the-art convincingness model, we evaluate several methods for using pairwise-annotated data examples to train models for ranking passages. Our results show pairwise training outperforms training that regresses to a target score for each passage. Our results also show a simple {`}win-rate{'} score is a better regression target than the previously proposed page-rank target. Lastly, addressing the need to filter noisy crowd-sourced annotations when constructing a dataset, we show that filtering for transitivity within pairwise annotations is more effective than filtering based on annotation confidence measures for individual examples.