pwc-archive/papers-with-abstracts huggingface dataset:
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Authors: Canming Huang, Weinan He, Yongmei Liu
Date: 2021-11-01 00:00:00
Proceeding: Findings (EMNLP) 2021 11
Tasks: Natural Language Inference, Transfer Learning, Winowhy

Recent methods based on pre-trained language models have shown strong supervised performance on commonsense reasoning. However, they rely on expensive data annotation and time-consuming training. Thus, we focus on unsupervised commonsense reasoning. We show the effectiveness of using a common framework, Natural Language Inference (NLI), to solve diverse commonsense reasoning tasks. By leveraging transfer learning from large NLI datasets, and injecting crucial knowledge from commonsense sources such as ATOMIC 2020 and ConceptNet, our method achieved state-of-the-art unsupervised performance on two commonsense reasoning tasks: WinoWhy and CommonsenseQA. Further analysis demonstrated the benefits of multiple categories of knowledge, but problems about quantities and antonyms are still challenging.
Negation Scope Resolution for Chinese as a Second Language
Authors: Mengyu Zhang, Weiqi Wang, Shuqiao Sun, Weiwei Sun
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Cross-Lingual Transfer, Negation, Negation Scope Resolution

This paper studies Negation Scope Resolution (NSR) for Chinese as a Second Language (CSL), which shows many unique characteristics that distinguish itself from “standard” Chinese. We annotate a new moderate-sized corpus that covers two background L1 languages, viz. English and Japanese. We build a neural NSR system, which achieves a new state-of-the-art accuracy on English benchmark data. We leverage this system to gauge how successful NSR for CSL can be. Different native language backgrounds of language learners result in unequal cross-lingual transfer, which has a significant impact on processing second language data. In particular, manual annotation, empirical evaluation and error analysis indicate two non-obvious facts: 1) L2-Chinese, L1-Japanese data are more difficult to analyze and thus annotate than L2-Chinese, L1-English data; 2) computational models trained on L2-Chinese, L1-Japanese data perform better than models trained on L2-Chinese, L1-English data.
Broad Linguistic Complexity Analysis for Greek Readability Classification
Authors: Savvas Chatzipanagiotidis, Maria Giagkou, Detmar Meurers
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Classification, Cross-corpus

This paper explores the linguistic complexity of Greek textbooks as a readability classification task. We analyze textbook corpora for different school subjects and textbooks for Greek as a Second Language, covering a very wide spectrum of school age groups and proficiency levels. A broad range of quantifiable linguistic complexity features (lexical, morphological and syntactic) are extracted and calculated. Conducting experiments with different feature subsets, we show that the different linguistic dimensions contribute orthogonal information, each contributing towards the highest result achieved using all linguistic feature subsets. A readability classifier trained on this basis reaches a classification accuracy of 88.16% for the Greek as a Second Language corpus. To investigate the generalizability of the classification models, we also perform cross-corpus evaluations. We show that the model trained on the most varied text collection (for Greek as a school subject) generalizes best. In addition to advancing the state of the art for Greek readability analysis, the paper also contributes insights on the role of different feature sets and training setups for generalizable readability classification.
Character Set Construction for Chinese Language Learning
Authors: Chak Yan Yeung, John Lee
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Clustering, Semantic Similarity, Semantic Textual Similarity

To promote efficient learning of Chinese characters, pedagogical materials may present not only a single character, but a set of characters that are related in meaning and in written form. This paper investigates automatic construction of these character sets. The proposed model represents a character as averaged word vectors of common words containing the character. It then identifies sets of characters with high semantic similarity through clustering. Human evaluation shows that this representation outperforms direct use of character embeddings, and that the resulting character sets capture distinct semantic ranges.
Identifying negative language transfer in learner errors using POS information
Authors: Leticia Farias Wanderley, Carrie Demmans Epp
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: POS

A common mistake made by language learners is the misguided usage of first language rules when communicating in another language. In this paper, n-gram and recurrent neural network language models are used to represent language structures and detect when Chinese native speakers incorrectly transfer rules from their first language (i.e., Chinese) into their English writing. These models make it possible to inform corrective error feedback with error causes, such as negative language transfer. We report the results of our negative language detection experiments with n-gram and recurrent neural network models that were trained using part-of-speech tags. The best performing model achieves an F1-score of 0.51 when tasked with recognizing negative language transfer in English learner data.
Document-level grammatical error correction
Authors: Zheng Yuan, Christopher Bryant
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Grammatical Error Correction, NMT, Sentence

Document-level context can provide valuable information in grammatical error correction (GEC), which is crucial for correcting certain errors and resolving inconsistencies. In this paper, we investigate context-aware approaches and propose document-level GEC systems. Additionally, we employ a three-step training strategy to benefit from both sentence-level and document-level data. Our system outperforms previous document-level and all other NMT-based single-model systems, achieving state of the art on a common test set.
Essay Quality Signals as Weak Supervision for Source-based Essay Scoring
Authors: Haoran Zhang, Diane Litman
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Automated Essay Scoring

Human essay grading is a laborious task that can consume much time and effort. Automated Essay Scoring (AES) has thus been proposed as a fast and effective solution to the problem of grading student writing at scale. However, because AES typically uses supervised machine learning, a human-graded essay corpus is still required to train the AES model. Unfortunately, such a graded corpus often does not exist, so creating a corpus for machine learning can also be a laborious task. This paper presents an investigation of replacing the use of human-labeled essay grades when training an AES system with two automatically available but weaker signals of essay quality: word count and topic distribution similarity. Experiments using two source-based essay scoring (evidence score) corpora show that while weak supervision does not yield a competitive result when training a neural source-based AES model, it can be used to successfully extract Topical Components (TCs) from a source text, which are required by a supervised feature-based AES model. In particular, results show that feature-based AES performance is comparable with either automatically or manually constructed TCs.
Parsing Argumentative Structure in English-as-Foreign-Language Essays
Authors: Jan Wira Gotama Putra, Simone Teufel, Takenobu Tokunaga
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Argument Mining, Sentence

This paper presents a study on parsing the argumentative structure in English-as-foreign-language (EFL) essays, which are inherently noisy. The parsing process consists of two steps, linking related sentences and then labelling their relations. We experiment with several deep learning architectures to address each task independently. In the sentence linking task, a biaffine model performed the best. In the relation labelling task, a fine-tuned BERT model performed the best. Two sentence encoders are employed, and we observed that non-fine-tuning models generally performed better when using Sentence-BERT as opposed to BERT encoder. We trained our models using two types of parallel texts: original noisy EFL essays and those improved by annotators, then evaluate them on the original essays. The experiment shows that an end-to-end in-domain system achieved an accuracy of .341. On the other hand, the cross-domain system achieved 94% performance of the in-domain system. This signals that well-written texts can also be useful to train argument mining system for noisy texts.
Training and Domain Adaptation for Supervised Text Segmentation
Authors: Goran Glavaš, Ananya Ganesh, Swapna Somasundaran
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Domain Adaptation, Hierarchical Text Segmentation, Segmentation, Text Segmentation

Unlike traditional unsupervised text segmentation methods, recent supervised segmentation models rely on Wikipedia as the source of large-scale segmentation supervision. These models have, however, predominantly been evaluated on the in-domain (Wikipedia-based) test sets, preventing conclusions about their general segmentation efficacy. In this work, we focus on the domain transfer performance of supervised neural text segmentation in the educational domain. To this end, we first introduce K12Seg, a new dataset for evaluation of supervised segmentation, created from educational reading material for grade-1 to college-level students. We then benchmark a hierarchical text segmentation model (HITS), based on RoBERTa, in both in-domain and domain-transfer segmentation experiments. While HITS produces state-of-the-art in-domain performance (on three Wikipedia-based test sets), we show that, subject to the standard full-blown fine-tuning, it is susceptible to domain overfitting. We identify adapter-based fine-tuning as a remedy that substantially improves transfer performance.
Data Strategies for Low-Resource Grammatical Error Correction
Authors: Simon Flachs, Felix Stahlberg, Shankar Kumar
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Grammatical Error Correction

Grammatical Error Correction (GEC) is a task that has been extensively investigated for the English language. However, for low-resource languages the best practices for training GEC systems have not yet been systematically determined. We investigate how best to take advantage of existing data sources for improving GEC systems for languages with limited quantities of high quality training data. We show that methods for generating artificial training data for GEC can benefit from including morphological errors. We also demonstrate that noisy error correction data gathered from Wikipedia revision histories and the language learning website Lang8, are valuable data sources. Finally, we show that GEC systems pre-trained on noisy data sources can be fine-tuned effectively using small amounts of high quality, human-annotated data.
Towards a Data Analytics Pipeline for the Visualisation of Complexity Metrics in L2 writings
Authors: Thomas Gaillat, Anas Knefati, Antoine Lafontaine
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Relation

We present the design of a tool for the visualisation of linguistic complexity in second language (L2) learner writings. We show how metrics can be exploited to visualise complexity in L2 writings in relation to CEFR levels.
Estonian as a Second Language Teacher’s Tools
Authors: Tiiu Üksik, Jelena Kallas, Kristina Koppel, Katrin Tsepelina, Raili Pool
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4

The paper presents the results of the project “Teacher’s Tools” (et Õpetaja tööriistad) published as a subpage of the new language portal Sõnaveeb developed by the Institute of the Estonian Language. The toolbox includes four modules: vocabulary, grammar, communicative language activities and text evaluation. The tools are aimed to help teachers and specialists of Estonian as a second language plan courses, create new educational materials, exercises and tests based on CEFR level descriptions.
Assessing Grammatical Correctness in Language Learning
Authors: Anisia Katinskaia, Roman Yangarber
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Grammatical Error Detection, LEMMA

We present experiments on assessing the grammatical correctness of learners’ answers in a language-learning System (references to the System, and the links to the released data and code are withheld for anonymity). In particular, we explore the problem of detecting alternative-correct answers: when more than one inflected form of a lemma fits syntactically and semantically in a given context. We approach the problem with the methods for grammatical error detection (GED), since we hypothesize that models for detecting grammatical mistakes can assess the correctness of potential alternative answers in a learning setting. Due to the paucity of training data, we explore the ability of pre-trained BERT to detect grammatical errors and then fine-tune it using synthetic training data. In this work, we focus on errors in inflection. Our experiments show a. that pre-trained BERT performs worse at detecting grammatical irregularities for Russian than for English; b. that fine-tuned BERT yields promising results on assessing the correctness of grammatical exercises; and c. establish a new benchmark for Russian. To further investigate its performance, we compare fine-tuned BERT with one of the state-of-the-art models for GED (Bell et al., 2019) on our dataset and RULEC-GEC (Rozovskaya and Roth, 2019). We release the manually annotated learner dataset, used for testing, for general use.
On the application of Transformers for estimating the difficulty of Multiple-Choice Questions from text
Authors: Luca Benedetto, Giovanni Aradelli, Paolo Cremonesi, Andrea Cappelli, Andrea Giussani, Roberto Turrin
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Multiple-choice

Classical approaches to question calibration are either subjective or require newly created questions to be deployed before being calibrated. Recent works explored the possibility of estimating question difficulty from text, but did not experiment with the most recent NLP models, in particular Transformers. In this paper, we compare the performance of previous literature with Transformer models experimenting on a public and a private dataset. Our experimental results show that Transformers are capable of outperforming previously proposed models. Moreover, if an additional corpus of related documents is available, Transformers can leverage that information to further improve calibration accuracy. We characterize the dependence of the model performance on some properties of the questions, showing that it performs best on questions ending with a question mark and Multiple-Choice Questions (MCQs) with one correct choice.
Automatically Generating Cause-and-Effect Questions from Passages
Authors: Katherine Stasaski, Manav Rathod, Tony Tu, Yunfang Xiao, Marti A. Hearst
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Question Answering, Question Generation, Question-Generation

Automated question generation has the potential to greatly aid in education applications, such as online study aids to check understanding of readings. The state-of-the-art in neural question generation has advanced greatly, due in part to the availability of large datasets of question-answer pairs. However, the questions generated are often surface-level and not challenging for a human to answer. To develop more challenging questions, we propose the novel task of cause-and-effect question generation. We build a pipeline that extracts causal relations from passages of input text, and feeds these as input to a state-of-the-art neural question generator. The extractor is based on prior work that classifies causal relations by linguistic category (Cao et al., 2016; Altenberg, 1984). This work results in a new, publicly available collection of cause-and-effect questions. We evaluate via both automatic and manual metrics and find performance improves for both question generation and question answering when we utilize a small auxiliary data source of cause-and-effect questions for fine-tuning. Our approach can be easily applied to generate cause-and-effect questions from other text collections and educational material, allowing for adaptable large-scale generation of cause-and-effect questions.
Interventions Recommendation: Professionals’ Observations Analysis in Special Needs Education
Authors: Javier Muñoz, Felipe Bravo-Marquez
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4

We present a new task in educational NLP, recommend the best interventions to help special needs education professionals to work with students with different disabilities. We use the professionals’ observations of the students together with the students diagnosis and other chosen interventions to predict the best interventions for Chilean special needs students.
C-Test Collector: A Proficiency Testing Application to Collect Training Data for C-Tests
Authors: Christian Haring, Rene Lehmann, Andrea Horbach, Torsten Zesch
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4

We present the C-Test Collector, a web-based tool that allows language learners to test their proficiency level using c-tests. Our tool collects anonymized data on test performance, which allows teachers to gain insights into common error patterns. At the same time, it allows NLP researchers to collect training data for being able to generate c-test variants at the desired difficulty level.
Virtual Pre-Service Teacher Assessment and Feedback via Conversational Agents
Authors: Debajyoti Datta, Maria Phillips, James P. Bywater, Jennifer Chiu, Ginger S. Watson, Laura Barnes, Donald Brown
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Reading Comprehension

Conversational agents and assistants have been used for decades to facilitate learning. There are many examples of conversational agents used for educational and training purposes in K-12, higher education, healthcare, the military, and private industry settings. The most common forms of conversational agents in education are teaching agents that directly teach and support learning, peer agents that serve as knowledgeable learning companions to guide learners in the learning process, and teachable agents that function as a novice or less-knowledgeable student trained and taught by a learner who learns by teaching. The Instructional Quality Assessment (IQA) provides a robust framework to evaluate reading comprehension and mathematics instruction. We developed a system for pre-service teachers, individuals in a teacher preparation program, to evaluate teaching instruction quality based on a modified interpretation of IQA metrics. Our demonstration and approach take advantage of recent advances in Natural Language Processing (NLP) and deep learning for each dialogue system component. We built an open-source conversational agent system to engage pre-service teachers in a specific mathematical scenario focused on scale factor with the aim to provide feedback on pre-service teachers’ questioning strategies. We believe our system is not only practical for teacher education programs but can also enable other researchers to build new educational scenarios with minimal effort.
Automated Classification of Written Proficiency Levels on the CEFR-Scale through Complexity Contours and RNNs
Authors: Elma Kerz, Daniel Wiechmann, Yu Qiao, Emma Tseng, Marcus Ströbel
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4

Automatically predicting the level of second language (L2) learner proficiency is an emerging topic of interest and research based on machine learning approaches to language learning and development. The key to the present paper is the combined use of what we refer to as ‘complexity contours’, a series of measurements of indices of L2 proficiency obtained by a computational tool that implements a sliding window technique, and recurrent neural network (RNN) classifiers that adequately capture the sequential information in those contours. We used the EF-Cambridge Open Language Database (Geertzen et al. 2013) with its labelled Common European Framework of Reference (CEFR) levels (Council of Europe 2018) to predict six classes of L2 proficiency levels (A1, A2, B1, B2, C1, C2) in the assessment of writing skills. Our experiments demonstrate that an RNN classifier trained on complexity contours achieves higher classification accuracy than one trained on text-average complexity scores. In a secondary experiment, we determined the relative importance of features from four distinct categories through a sensitivity-based pruning technique. Our approach makes an important contribution to the field of automated identification of language proficiency levels, more specifically, to the increasing efforts towards the empirical validation of CEFR levels.
“Sharks are not the threat humans are”: Argument Component Segmentation in School Student Essays
Authors: Tariq Alhindi, Debanjan Ghosh
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Argument Mining, Classification, Multi-Task Learning, Sentence

Argument mining is often addressed by a pipeline method where segmentation of text into argumentative units is conducted first and proceeded by an argument component identification task. In this research, we apply a token-level classification to identify claim and premise tokens from a new corpus of argumentative essays written by middle school students. To this end, we compare a variety of state-of-the-art models such as discrete features and deep learning architectures (e.g., BiLSTM networks and BERT-based architectures) to identify the argument components. We demonstrate that a BERT-based multi-task learning architecture (i.e., token and sentence level classification) adaptively pretrained on a relevant unlabeled dataset obtains the best results.
Using Linguistic Features to Predict the Response Process Complexity Associated with Answering Clinical MCQs
Authors: Victoria Yaneva, Daniel Jurich, Le An Ha, Peter Baldwin
Date: 2021-04-01 00:00:00
Proceeding: EACL (BEA) 2021 4
Tasks: Clustering, Descriptive

This study examines the relationship between the linguistic characteristics of a test item and the complexity of the response process required to answer it correctly. Using data from a large-scale medical licensing exam, clustering methods identified items that were similar with respect to their relative difficulty and relative response-time intensiveness to create low response process complexity and high response process complexity item classes. Interpretable models were used to investigate the linguistic features that best differentiated between these classes from a descriptive and predictive framework. Results suggest that nuanced features such as the number of ambiguous medical terms help explain response process complexity beyond superficial item characteristics such as word count. Yet, although linguistic features carry signal relevant to response process complexity, the classification of individual items remains challenging.