pwc-archive/papers-with-abstracts huggingface dataset:
(Almost) Unsupervised Grammatical Error Correction using Synthetic Comparable Corpus
Authors: Satoru Katsumata, Mamoru Komachi
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Machine Translation, Translation

We introduce unsupervised techniques based on phrase-based statistical machine translation for grammatical error correction (GEC) trained on a pseudo learner corpus created by Google Translation. We verified our GEC system through experiments on a low resource track of the shared task at BEA2019. As a result, we achieved an F0.5 score of 28.31 points with the test data.
Erroneous data generation for Grammatical Error Correction
Authors: Shuyao Xu, Jiehao Zhang, Jin Chen, Long Qin
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction

It has been demonstrated that the utilization of a monolingual corpus in neural Grammatical Error Correction (GEC) systems can significantly improve the system performance. The previous state-of-the-art neural GEC system is an ensemble of four Transformer models pretrained on a large amount of Wikipedia Edits. The Singsound GEC system follows a similar approach but is equipped with a sophisticated erroneous data generating component. Our system achieved an F0:5 of 66.61 in the BEA 2019 Shared Task: Grammatical Error Correction. With our novel erroneous data generating component, the Singsound neural GEC system yielded an M2 of 63.2 on the CoNLL-2014 benchmark (8.4{\%} relative improvement over the previous state-of-the-art system).
The LAIX Systems in the BEA-2019 GEC Shared Task
Authors: Ruobing Li, Chuan Wang, Yefei Zha, Yonghong Yu, Shiman Guo, Qiang Wang, Yang Liu, Hui Lin
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Classification, General Classification, Grammatical Error Correction, Machine Translation, NMT, Translation

In this paper, we describe two systems we developed for the three tracks we have participated in the BEA-2019 GEC Shared Task. We investigate competitive classification models with bi-directional recurrent neural networks (Bi-RNN) and neural machine translation (NMT) models. For different tracks, we use ensemble systems to selectively combine the NMT models, the classification models, and some rules, and demonstrate that an ensemble solution can effectively improve GEC performance over single systems. Our GEC systems ranked the first in the Unrestricted Track, and the third in both the Restricted Track and the Low Resource Track.
The AIP-Tohoku System at the BEA-2019 Shared Task
Authors: Hiroki Asano, Masato Mita, Tomoya Mizumoto, Jun Suzuki
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Grammatical Error Detection, Sentence

We introduce the AIP-Tohoku grammatical error correction (GEC) system for the BEA-2019 shared task in Track 1 (Restricted Track) and Track 2 (Unrestricted Track) using the same system architecture. Our system comprises two key components: error generation and sentence-level error detection. In particular, GEC with sentence-level grammatical error detection is a novel and versatile approach, and we experimentally demonstrate that it significantly improves the precision of the base model. Our system is ranked 9th in Track 1 and 2nd in Track 2.
Noisy Channel for Low Resource Grammatical Error Correction
Authors: Simon Flachs, Oph{\'e}lie Lacroix, Anders S{\o}gaard
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Language Modeling, Language Modelling

This paper describes our contribution to the low-resource track of the BEA 2019 shared task on Grammatical Error Correction (GEC). Our approach to GEC builds on the theory of the noisy channel by combining a channel model and language model. We generate confusion sets from the Wikipedia edit history and use the frequencies of edits to estimate the channel model. Additionally, we use two pre-trained language models: 1) Google{'}s BERT model, which we fine-tune for specific error types and 2) OpenAI{'}s GPT-2 model, utilizing that it can operate with previous sentences as context. Furthermore, we search for the optimal combinations of corrections using beam search.
The BLCU System in the BEA 2019 Shared Task
Authors: Liner Yang, Chencheng Wang
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction

This paper describes the BLCU Group submissions to the Building Educational Applications (BEA) 2019 Shared Task on Grammatical Error Correction (GEC). The task is to detect and correct grammatical errors that occurred in essays. We participate in 2 tracks including the Restricted Track and the Unrestricted Track. Our system is based on a Transformer model architecture. We integrate many effective methods proposed in recent years. Such as, Byte Pair Encoding, model ensemble, checkpoints average and spell checker. We also corrupt the public monolingual data to further improve the performance of the model. On the test data of the BEA 2019 Shared Task, our system yields F0.5 = 58.62 and 59.50, ranking twelfth and fourth respectively.
TMU Transformer System Using BERT for Re-ranking at BEA 2019 Grammatical Error Correction on Restricted Track
Authors: Masahiro Kaneko, Kengo Hotate, Satoru Katsumata, Mamoru Komachi
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Re-Ranking, Sentence

We introduce our system that is submitted to the restricted track of the BEA 2019 shared task on grammatical error correction1 (GEC). It is essential to select an appropriate hypothesis sentence from the candidates list generated by the GEC model. A re-ranker can evaluate the naturalness of a corrected sentence using language models trained on large corpora. On the other hand, these language models and language representations do not explicitly take into account the grammatical errors written by learners. Thus, it is not straightforward to utilize language representations trained from a large corpus, such as Bidirectional Encoder Representations from Transformers (BERT), in a form suitable for the learner{'}s grammatical errors. Therefore, we propose to fine-tune BERT on learner corpora with grammatical errors for re-ranking. The experimental results of the W{\&}I+LOCNESS development dataset demonstrate that re-ranking using BERT can effectively improve the correction performance.
Neural and FST-based approaches to grammatical error correction
Authors: Zheng Yuan, Felix Stahlberg, Marek Rei, Bill Byrne, Helen Yannakoudakis
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Machine Translation, Multi-Task Learning, Re-Ranking, Translation

In this paper, we describe our submission to the BEA 2019 shared task on grammatical error correction. We present a system pipeline that utilises both error detection and correction models. The input text is first corrected by two complementary neural machine translation systems: one using convolutional networks and multi-task learning, and another using a neural Transformer-based system. Training is performed on publicly available data, along with artificial examples generated through back-translation. The n-best lists of these two machine translation systems are then combined and scored using a finite state transducer (FST). Finally, an unsupervised re-ranking system is applied to the n-best output of the FST. The re-ranker uses a number of error detection features to re-rank the FST n-best list and identify the final 1-best correction hypothesis. Our system achieves 66.75{\%} F 0.5 on error correction (ranking 4th), and 82.52{\%} F 0.5 on token-level error detection (ranking 2nd) in the restricted track of the shared task.
Improving Precision of Grammatical Error Correction with a Cheat Sheet
Authors: Mengyang Qiu, Xuejiao Chen, Maggie Liu, Krishna Parvathala, Apurva Patil, Jungyeul Park
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Machine Translation, Translation

In this paper, we explore two approaches of generating error-focused phrases and examine whether these phrases can lead to better performance in grammatical error correction for the restricted track of BEA 2019 Shared Task on GEC. Our results show that phrases directly extracted from GEC corpora outperform phrases from statistical machine translation phrase table by a large margin. Appending error+context phrases to the original GEC corpora yields comparably high precision. We also explore the generation of artificial syntactic error sentences using error+context phrases for the unrestricted track. The additional training data greatly facilitates syntactic error correction (e.g., verb form) and contributes to better overall performance.
Multi-headed Architecture Based on BERT for Grammatical Errors Correction
Authors: Bohdan Didenko, Julia Shaptala
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Sentence

In this paper, we describe our approach to GEC using the BERT model for creation of encoded representation and some of our enhancements, namely, {``}Heads{''} are fully-connected networks which are used for finding the errors and later receive recommendation from the networks on dealing with a highlighted part of the sentence only. Among the main advantages of our solution is increasing the system productivity and lowering the time of processing while keeping the high accuracy of GEC results.
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data
Authors: Roman Grundkiewicz, Marcin Junczys-Dowmunt, Kenneth Heafield
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction, Unsupervised Pre-training

Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre-train a Transformer sequence-to-sequence model, which not only improves over a strong baseline trained on authentic error-annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error-annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F$_{0.5}$ in the restricted and low-resource tracks respectively, both on the W{\&}I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state-of-the-art results of 64.16 M{\mbox{$^2$}} for the submitted system, and 61.30 M{\mbox{$^2$}} for the constrained system trained on the NUCLE and Lang-8 data.
Evaluation of automatic collocation extraction methods for language learning
Authors: Vishal Bhalla, Klara Klimcikova
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Dependency Parsing, POS

A number of methods have been proposed to automatically extract collocations, i.e., conventionalized lexical combinations, from text corpora. However, the attempts to evaluate and compare them with a specific application in mind lag behind. This paper compares three end-to-end resources for collocation learning, all of which used the same corpus but different methods. Adopting a gold-standard evaluation method, the results show that the method of dependency parsing outperforms regex-over-pos in collocation identification. The lexical association measures (AMs) used for collocation ranking perform about the same overall but differently for individual collocation types. Further analysis has also revealed that there are considerable differences between other commonly used AMs.
Anglicized Words and Misspelled Cognates in Native Language Identification
Authors: Ilia Markov, Vivi Nastase, Carlo Strapparava
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Language Identification, Native Language Identification

In this paper, we present experiments that estimate the impact of specific lexical choices of people writing in a second language (L2). In particular, we look at misspelled words that indicate lexical uncertainty on the part of the author, and separate them into three categories: misspelled cognates, {``}L2-ed{''} (in our case, anglicized) words, and all other spelling errors. We test the assumption that such errors contain clues about the native language of an essay{'}s author through the task of native language identification. The results of the experiments show that the information brought by each of these categories is complementary. We also note that while the distribution of such features changes with the proficiency level of the writer, their contribution towards native language identification remains significant at all levels.
Linguistically-Driven Strategy for Concept Prerequisites Learning on Italian
Authors: Alessio Miaschi, Chiara Alzetta, Franco Alberto Cardillo, Felice Dell{'}Orletta
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

We present a new concept prerequisite learning method for Learning Object (LO) ordering that exploits only linguistic features extracted from textual educational resources. The method was tested in a cross- and in- domain scenario both for Italian and English. Additionally, we performed experiments based on a incremental training strategy to study the impact of the training set size on the classifier performances. The paper also introduces ITA-PREREQ, to the best of our knowledge the first Italian dataset annotated with prerequisite relations between pairs of educational concepts, and describe the automatic strategy devised to build it.
Grammatical-Error-Aware Incorrect Example Retrieval System for Learners of Japanese as a Second Language
Authors: Mio Arai, Masahiro Kaneko, Mamoru Komachi
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Detection, Retrieval, Sentence, Sentence Retrieval

Existing example retrieval systems do not include grammatically incorrect examples or present only a few examples, if any. Even if a retrieval system has a wide coverage of incorrect examples along with the correct counterpart, learners need to know whether their query includes errors or not. Considering the usability of retrieving incorrect examples, our proposed method uses a large-scale corpus and presents correct expressions along with incorrect expressions using a grammatical error detection system so that the learner do not need to be aware of how to search for the examples. Intrinsic and extrinsic evaluations indicate that our method improves accuracy of example sentence retrieval and quality of learner{'}s writing.
Analytic Score Prediction and Justification Identification in Automated Short Answer Scoring
Authors: Tomoya Mizumoto, Hiroki Ouchi, Yoriko Isobe, Paul Reisert, Ryo Nagata, Satoshi Sekine, Kentaro Inui
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

This paper provides an analytical assessment of student short answer responses with a view to potential benefits in pedagogical contexts. We first propose and formalize two novel analytical assessment tasks: analytic score prediction and justification identification, and then provide the first dataset created for analytic short answer scoring research. Subsequently, we present a neural baseline model and report our extensive empirical results to demonstrate how our dataset can be used to explore new and intriguing technical challenges in short answer scoring. The dataset is publicly available for research purposes.
Content Customization for Micro Learning using Human Augmented AI Techniques
Authors: Ayush Shah, Tamer Abuelsaad, Jae-wook Ahn, Prasenjit Dey, Ravi Kokku, Ruhi Sharma Mittal, Aditya Vempaty, Mourvi Sharma
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Visual content has been proven to be effective for micro-learning compared to other media. In this paper, we discuss leveraging this observation in our efforts to build audio-visual content for young learners{'} vocabulary learning. We attempt to tackle two major issues in the process of traditional visual curation tasks. Generic learning videos do not necessarily satisfy the unique context of a learner and/or an educator, and hence may not result in maximal learning outcomes. Also, manual video curation by educators is a highly labor-intensive process. To this end, we present a customizable micro-learning audio-visual content curation tool that is designed to reduce the human (educator) effort in creating just-in-time learning videos from a textual description (learning script). This provides educators with control of the content while preparing the learning scripts, and in turn can also be customized to capture the desired learning objectives and outcomes. As a use case, we automatically generate learning videos with British National Corpus{'} (BNC) frequently spoken vocabulary words and evaluate them with experts. They positively recommended the generated learning videos with an average rating of 4.25 on a Likert scale of 5 points. The inter-annotator agreement between the experts for the video quality was substantial (Fleiss Kappa=0.62) with an overall agreement of 81{\%}.
Curio SmartChat : A system for Natural Language Question Answering for Self-Paced K-12 Learning
Authors: Srikrishna Raamadhurai, Ryan Baker, Vikraman Poduval
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Question Answering

During learning, students often have questions which they would benefit from responses to in real time. In class, a student can ask a question to a teacher. During homework, or even in class if the student is shy, it can be more difficult to receive a rapid response. In this work, we introduce Curio SmartChat, an automated question answering system for middle school Science topics. Our system has now been used by around 20,000 students who have so far asked over 100,000 questions. We present data on the challenge created by students{'} grammatical errors and spelling mistakes, and discuss our system{'}s approach and degree of effectiveness at disambiguating questions that the system is initially unsure about. We also discuss the prevalence of student {``}small talk{''} not related to science topics, the pluses and minuses of this behavior, and how a system should respond to these conversational acts. We conclude with discussions and point to directions for potential future work.
Supporting content evaluation of student summaries by Idea Unit embedding
Authors: Marcello Gecchele, Hiroaki Yamada, Takenobu Tokunaga, Yasuyo Sawaki
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

This paper discusses the computer-assisted content evaluation of summaries. We propose a method to make a correspondence between the segments of the source text and its summary. As a unit of the segment, we adopt {``}Idea Unit (IU){''} which is proposed in Applied Linguistics. Introducing IUs enables us to make a correspondence even for the sentences that contain multiple ideas. The IU correspondence is made based on the similarity between vector representations of IU. An evaluation experiment with two source texts and 20 summaries showed that the proposed method is more robust against rephrased expressions than the conventional ROUGE-based baselines. Also, the proposed method outperformed the baselines in recall. We im-plemented the proposed method in a GUI tool{``}Segment Matcher{''} that aids teachers to estab-lish a link between corresponding IUs acrossthe summary and source text.
On Understanding the Relation between Expert Annotations of Text Readability and Target Reader Comprehension
Authors: Sowmya Vajjala, Ivana Lucic
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Question Generation, Question-Generation, Reading Comprehension, Relation

Automatic readability assessment aims to ensure that readers read texts that they can comprehend. However, computational models are typically trained on texts created from the perspective of the text writer, not the target reader. There is little experimental research on the relationship between expert annotations of readability, reader{'}s language proficiency, and different levels of reading comprehension. To address this gap, we conducted a user study in which over a 100 participants read texts of different reading levels and answered questions created to test three forms of comprehension. Our results indicate that more than readability annotation or reader proficiency, it is the type of comprehension question asked that shows differences between reader responses - inferential questions were difficult for users of all levels of proficiency across reading levels. The data collected from this study will be released with this paper, which will, for the first time, provide a collection of 45 reader bench marked texts to evaluate readability assessment systems developed for adult learners of English. It can also potentially be useful for the development of question generation approaches in intelligent tutoring systems research.
Measuring Text Complexity for Italian as a Second Language Learning Purposes
Authors: Luciana Forti, Alfredo Milani, Luisa Piersanti, Filippo Santarelli, Valentino Santucci, Stefania Spina
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

The selection of texts for second language learning purposes typically relies on teachers{'} and test developers{'} individual judgment of the observable qualitative properties of a text. Little or no consideration is generally given to the quantitative dimension within an evidence-based framework of reproducibility. This study aims to fill the gap by evaluating the effectiveness of an automatic tool trained to assess text complexity in the context of Italian as a second language learning. A dataset of texts labeled by expert test developers was used to evaluate the performance of three classifier models (decision tree, random forest, and support vector machine), which were trained using linguistic features measured quantitatively and extracted from the texts. The experimental analysis provided satisfactory results, also in relation to which kind of linguistic trait contributed the most to the final outcome.
Simple Construction of Mixed-Language Texts for Vocabulary Learning
Authors: Adithya Renduchintala, Philipp Koehn, Jason Eisner
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Language Modeling, Language Modelling

We present a machine foreign-language teacher that takes documents written in a student{'}s native language and detects situations where it can replace words with their foreign glosses such that new foreign vocabulary can be learned simply through reading the resulting mixed-language text. We show that it is possible to design such a machine teacher without any supervised data from (human) students. We accomplish this by modifying a cloze language model to incrementally learn new vocabulary items, and use this language model as a proxy for the word guessing and learning ability of real students. Our machine foreign-language teacher decides which subset of words to replace by consulting this language model. We evaluate three variants of our student proxy language models through a study on Amazon Mechanical Turk (MTurk). We find that MTurk {``}students{''} were able to guess the meanings of foreign words introduced by the machine teacher with high accuracy for both function words as well as content words in two out of the three models. In addition, we show that students are able to retain their knowledge about the foreign words after they finish reading the document.