pwc-archive/papers-with-abstracts huggingface dataset:
Automatic Extraction of High-Quality Example Sentences for Word Learning Using a Determinantal Point Process
Authors: Arseny Tolmachev, Sadao Kurohashi
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Sentence, Sentence Similarity

Flashcard systems are effective tools for learning words but have their limitations in teaching word usage. To overcome this problem, we propose a novel flashcard system that shows a new example sentence on each repetition. This extension requires high-quality example sentences, automatically extracted from a huge corpus. To do this, we use a Determinantal Point Process which scales well to large data and allows to naturally represent sentence similarity and quality as features. Our human evaluation experiment on Japanese language indicates that the proposed method successfully extracted high-quality example sentences.
Distractor Generation for Chinese Fill-in-the-blank Items
Authors: Shu Jiang, John Lee
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Distractor Generation, Semantic Similarity, Semantic Textual Similarity

This paper reports the first study on automatic generation of distractors for fill-in-the-blank items for learning Chinese vocabulary. We investigate the quality of distractors generated by a number of criteria, including part-of-speech, difficulty level, spelling, word co-occurrence and semantic similarity. Evaluations show that a semantic similarity measure, based on the word2vec model, yields distractors that are significantly more plausible than those generated by baseline methods.
Investigating neural architectures for short answer scoring
Authors: Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, Chong MIn Lee
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Automated Essay Scoring, Reading Comprehension, Word Embeddings

Neural approaches to automated essay scoring have recently shown state-of-the-art performance. The automated essay scoring task typically involves a broad notion of writing quality that encompasses content, grammar, organization, and conventions. This differs from the short answer content scoring task, which focuses on content accuracy. The inputs to neural essay scoring models {--} ngrams and embeddings {--} are arguably well-suited to evaluate content in short answer scoring tasks. We investigate how several basic neural approaches similar to those used for automated essay scoring perform on short answer scoring. We show that neural architectures can outperform a strong non-neural baseline, but performance and optimal parameter settings vary across the more diverse types of prompts typical of short answer scoring.
Human and Automated CEFR-based Grading of Short Answers
Authors: Ana{\"\i}s Tack, Thomas Fran{\c{c}}ois, Sophie Roekhaut, C{\'e}drick Fairon
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9

This paper is concerned with the task of automatically assessing the written proficiency level of non-native (L2) learners of English. Drawing on previous research on automated L2 writing assessment following the Common European Framework of Reference for Languages (CEFR), we investigate the possibilities and difficulties of deriving the CEFR level from short answers to open-ended questions, which has not yet been subjected to numerous studies up to date. The object of our study is twofold: to examine the intricacy involved with both human and automated CEFR-based grading of short answers. On the one hand, we describe the compilation of a learner corpus of short answers graded with CEFR levels by three certified Cambridge examiners. We mainly observe that, although the shortness of the answers is reported as undermining a clear-cut evaluation, the length of the answer does not necessarily correlate with inter-examiner disagreement. On the other hand, we explore the development of a soft-voting system for the automated CEFR-based grading of short answers and draw tentative conclusions about its use in a computer-assisted testing (CAT) setting.
GEC into the future: Where are we going and how do we get there?
Authors: Keisuke Sakaguchi, Courtney Napoles, Joel Tetreault
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Grammatical Error Correction, Machine Translation, Position

The field of grammatical error correction (GEC) has made tremendous bounds in the last ten years, but new questions and obstacles are revealing themselves. In this position paper, we discuss the issues that need to be addressed and provide recommendations for the field to continue to make progress, and propose a new shared task. We invite suggestions and critiques from the audience to make the new shared task a community-driven venture.
Combining Textual and Speech Features in the NLI Task Using State-of-the-Art Machine Learning Techniques
Authors: Pavel Ircing, Jan {\v{S}}vec, Zbyn{\v{e}}k Zaj{\'\i}c, Barbora Hladk{\'a}, Martin Holub
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Acquisition, Language Identification

We summarize the involvement of our CEMI team in the {''}NLI Shared Task 2017{''}, which deals with both textual and speech input data. We submitted the results achieved by using three different system architectures; each of them combines multiple supervised learning models trained on various feature sets. As expected, better results are achieved with the systems that use both the textual data and the spoken responses. Combining the input data of two different modalities led to a rather dramatic improvement in classification performance. Our best performing method is based on a set of feed-forward neural networks whose hidden-layer outputs are combined together using a softmax layer. We achieved a macro-averaged F1 score of 0.9257 on the evaluation (unseen) test set and our team placed first in the main task together with other three teams.
Native Language Identification Using a Mixture of Character and Word N-grams
Authors: Elham Mohammadi, Hadi Veisi, Hessam Amini
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Acquisition, Language Identification, Native Language Identification

Native language identification (NLI) is the task of determining an author{'}s native language, based on a piece of his/her writing in a second language. In recent years, NLI has received much attention due to its challenging nature and its applications in language pedagogy and forensic linguistics. We participated in the NLI2017 shared task under the name UT-DSP. In our effort to implement a method for native language identification, we made use of a fusion of character and word N-grams, and achieved an optimal F1-Score of 77.64{\%}, using both essay and speech transcription datasets.
Ensemble Methods for Native Language Identification
Authors: Sophia Chan, Maryam Honari Jahromi, Benjamin Benetti, Aazim Lakhani, Alona Fyshe
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Acquisition, Language Identification, Native Language Identification

Our team{---}Uvic-NLP{---}explored and evaluated a variety of lexical features for Native Language Identification (NLI) within the framework of ensemble methods. Using a subset of the highest performing features, we train Support Vector Machines (SVM) and Fully Connected Neural Networks (FCNN) as base classifiers, and test different methods for combining their outputs. Restricting our scope to the closed essay track in the NLI Shared Task 2017, we find that our best SVM ensemble achieves an F1 score of 0.8730 on the test set.
Neural Networks and Spelling Features for Native Language Identification
Authors: Johannes Bjerva, Gintar{\.e} Grigonyt{\.e}, Robert {\"O}stling, Barbara Plank
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Identification, Native Language Identification, Word Embeddings

We present the RUG-SU team{'}s submission at the Native Language Identification Shared Task 2017. We combine several approaches into an ensemble, based on spelling error features, a simple neural network using word representations, a deep residual network using word and character features, and a system based on a recurrent neural network. Our best system is an ensemble of neural networks, reaching an F1 score of 0.8323. Although our system is not the highest ranking one, we do outperform the baseline by far.
A study of N-gram and Embedding Representations for Native Language Identification
Authors: Sowmya Vajjala, Sagnik Banerjee
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Feature Engineering, Language Acquisition, Language Identification, Native Language Identification, POS

We report on our experiments with N-gram and embedding based feature representations for Native Language Identification (NLI) as a part of the NLI Shared Task 2017 (team name: NLI-ISU). Our best performing system on the test set for written essays had a macro F1 of 0.8264 and was based on word uni, bi and trigram features. We explored n-grams covering word, character, POS and word-POS mixed representations for this task. For embedding based feature representations, we employed both word and document embeddings. We had a relatively poor performance with all embedding representations compared to n-grams, which could be because of the fact that embeddings capture semantic similarities whereas L1 differences are more stylistic in nature.
A Shallow Neural Network for Native Language Identification with Character N-grams
Authors: Yunita Sari, Muhammad Rifqi Fatchurrahman, Meisyarah Dwiastuti
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Identification, Native Language Identification

This paper describes the systems submitted by GadjahMada team to the Native Language Identification (NLI) Shared Task 2017. Our models used a continuous representation of character n-grams which are learned jointly with feed-forward neural network classifier. Character n-grams have been proved to be effective for style-based identification tasks including NLI. Results on the test set demonstrate that the proposed model performs very well on essay and fusion tracks by obtaining more than 0.8 on both F-macro score and accuracy.
Fewer features perform well at Native Language Identification task
Authors: Taraka Rama, {\c{C}}a{\u{g}}r{\i} {\c{C}}{\"o}ltekin
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Identification, Native Language Identification

This paper describes our results at the NLI shared task 2017. We participated in essays, speech, and fusion task that uses text, speech, and i-vectors for the task of identifying the native language of the given input. In the essay track, a linear SVM system using word bigrams and character 7-grams performed the best. In the speech track, an LDA classifier based only on i-vectors performed better than a combination system using text features from speech transcriptions and i-vectors. In the fusion task, we experimented with systems that used combination of i-vectors with higher order n-grams features, combination of i-vectors with word unigrams, a mean probability ensemble, and a stacked ensemble system. Our finding is that word unigrams in combination with i-vectors achieve higher score than systems trained with larger number of $n$-gram features. Our best-performing systems achieved F1-scores of 87.16{\%}, 83.33{\%} and 91.75{\%} on the essay track, the speech track and the fusion track respectively.
Structured Generation of Technical Reading Lists
Authors: Jonathan Gordon, Stephen Aguilar, Emily Sheng, Gully Burns
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Information Retrieval, Machine Translation

Learners need to find suitable documents to read and prioritize them in an appropriate order. We present a method of automatically generating reading lists, selecting documents based on their pedagogical value to the learner and ordering them using the structure of concepts in the domain. Resulting reading lists related to computational linguistics were evaluated by advanced learners and judged to be near the quality of those generated by domain experts. We provide an open-source implementation of our method to enable future work on reading list generation.
Effects of Lexical Properties on Viewing Time per Word in Autistic and Neurotypical Readers
Authors: Sanja {\v{S}}tajner, Victoria Yaneva, Ruslan Mitkov, Simone Paolo Ponzetto
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Lexical Simplification

Eye tracking studies from the past few decades have shaped the way we think of word complexity and cognitive load: words that are long, rare and ambiguous are more difficult to read. However, online processing techniques have been scarcely applied to investigating the reading difficulties of people with autism and what vocabulary is challenging for them. We present parallel gaze data obtained from adult readers with autism and a control group of neurotypical readers and show that the former required higher cognitive effort to comprehend the texts as evidenced by three gaze-based measures. We divide all words into four classes based on their viewing times for both groups and investigate the relationship between longer viewing times and word length, word frequency, and four cognitively-based measures (word concreteness, familiarity, age of acquisition and imagability).
Transparent text quality assessment with convolutional neural networks
Authors: Robert {\"O}stling, Gintare Grigonyte
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Feature Engineering, Multi-Task Learning, Text Classification

We present a very simple model for text quality assessment based on a deep convolutional neural network, where the only supervision required is one corpus of user-generated text of varying quality, and one contrasting text corpus of consistently high quality. Our model is able to provide local quality assessments in different parts of a text, which allows visual feedback about where potentially problematic parts of the text are located, as well as a way to evaluate which textual features are captured by our model. We evaluate our method on two corpora: a large corpus of manually graded student essays and a longitudinal corpus of language learner written production, and find that the text quality metric learned by our model is a fairly strong predictor of both essay grade and learner proficiency level.
Modelling semantic acquisition in second language learning
Authors: Ekaterina Kochmar, Ekaterina Shutova
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9

Using methods of statistical analysis, we investigate how semantic knowledge is acquired in English as a second language and evaluate the pace of development across a number of predicate types and content word combinations, as well as across the levels of language proficiency and native languages. Our exploratory study helps identify the most problematic areas for language learners with different backgrounds and at different stages of learning.
Multiple Choice Question Generation Utilizing An Ontology
Authors: Katherine Stasaski, Marti A. Hearst
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Distractor Generation, Multiple-choice, Question Generation, Question-Generation

Ontologies provide a structured representation of concepts and the relationships which connect them. This work investigates how a pre-existing educational Biology ontology can be used to generate useful practice questions for students by using the connectivity structure in a novel way. It also introduces a novel way to generate multiple-choice distractors from the ontology, and compares this to a baseline of using embedding representations of nodes. An assessment by an experienced science teacher shows a significant advantage over a baseline when using the ontology for distractor generation. A subsequent study with three science teachers on the results of a modified question generation algorithm finds significant improvements. An in-depth analysis of the teachers{'} comments yields useful insights for any researcher working on automated question generation for educational applications.
Simplifying metaphorical language for young readers: A corpus study on news text
Authors: Magdalena Wolska, Yulia Clausen
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Lexical Simplification, Reading Comprehension, Text Simplification

The paper presents first results of an ongoing project on text simplification focusing on linguistic metaphors. Based on an analysis of a parallel corpus of news text professionally simplified for different grade levels, we identify six types of simplification choices falling into two broad categories: preserving metaphors or dropping them. An annotation study on almost 300 source sentences with metaphors (grade level 12) and their simplified counterparts (grade 4) is conducted. The results show that most metaphors are preserved and when they are dropped, the semantic content tends to be preserved rather than dropped, however, it is reworded without metaphorical language. In general, some of the expected tendencies in complexity reduction, measured with psycholinguistic variables linked to metaphor comprehension, are observed, suggesting good prospect for machine learning-based metaphor simplification.
Language Based Mapping of Science Assessment Items to Skills
Authors: Farah Nadeem, Mari Ostendorf
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: General Classification, text-classification, Text Classification

Knowledge of the association between assessment questions and the skills required to solve them is necessary for analysis of student learning. This association, often represented as a Q-matrix, is either hand-labeled by domain experts or learned as latent variables given a large student response data set. As a means of automating the match to formal standards, this paper uses neural text classification methods, leveraging the language in the standards documents to identify online text for a proxy training task. Experiments involve identifying the topic and crosscutting concepts of middle school science questions leveraging multi-task training. Results show that it is possible to automatically build a Q-matrix without student response data and using a modest number of hand-labeled questions.
Connecting the Dots: Towards Human-Level Grammatical Error Correction
Authors: Shamil Chollampatt, Hwee Tou Ng
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Grammatical Error Correction, Language Modeling, Language Modelling, Machine Translation, Translation

We build a grammatical error correction (GEC) system primarily based on the state-of-the-art statistical machine translation (SMT) approach, using task-specific features and tuning, and further enhance it with the modeling power of neural network joint models. The SMT-based system is weak in generalizing beyond patterns seen during training and lacks granularity below the word level. To address this issue, we incorporate a character-level SMT component targeting the misspelled words that the original SMT-based system fails to correct. Our final system achieves 53.14{\%} F 0.5 score on the benchmark CoNLL-2014 test set, an improvement of 3.62{\%} F 0.5 over the best previous published score.
Question Generation for Language Learning: From ensuring texts are read to supporting learning
Authors: Maria Chinkina, Detmar Meurers
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Decision Making, Form, Language Acquisition, Question Generation, Question-Generation

In Foreign Language Teaching and Learning (FLTL), questions are systematically used to assess the learner{'}s understanding of a text. Computational linguistic approaches have been developed to generate such questions automatically given a text (e.g., Heilman, 2011). In this paper, we want to broaden the perspective on the different functions questions can play in FLTL and discuss how automatic question generation can support the different uses. Complementing the focus on meaning and comprehension, we want to highlight the fact that questions can also be used to make learners notice form aspects of the linguistic system and their interpretation. Automatically generating questions that target linguistic forms and grammatical categories in a text in essence supports incidental focus-on-form (Loewen, 2005) in a meaning-focused reading task. We discuss two types of questions serving this purpose, how they can be generated automatically; and we report on a crowd-sourcing evaluation comparing automatically generated to manually written questions targeting particle verbs, a challenging linguistic form for learners of English.
Systematically Adapting Machine Translation for Grammatical Error Correction
Authors: Courtney Napoles, Chris Callison-Burch
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Grammatical Error Correction, Machine Translation, Spelling Correction, Translation

n this work we adapt machine translation (MT) to grammatical error correction, identifying how components of the statistical MT pipeline can be modified for this task and analyzing how each modification impacts system performance. We evaluate the contribution of each of these components with standard evaluation metrics and automatically characterize the morphological and lexical transformations made in system output. Our model rivals the current state of the art using a fraction of the training data.
Fine-grained essay scoring of a complex writing task for native speakers
Authors: Andrea Horbach, Dirk Scholten-Akoun, Yuning Ding, Torsten Zesch
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9

Automatic essay scoring is nowadays successfully used even in high-stakes tests, but this is mainly limited to holistic scoring of learner essays. We present a new dataset of essays written by highly proficient German native speakers that is scored using a fine-grained rubric with the goal to provide detailed feedback. Our experiments with two state-of-the-art scoring systems (a neural and a SVM-based one) show a large drop in performance compared to existing datasets. This demonstrates the need for such datasets that allow to guide research on more elaborate essay scoring methods.
Exploring Optimal Voting in Native Language Identification
Authors: Cyril Goutte, Serge L{\'e}ger
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Identification, Native Language Identification

We describe the submissions entered by the National Research Council Canada in the NLI-2017 evaluation. We mainly explored the use of voting, and various ways to optimize the choice and number of voting systems. We also explored the use of features that rely on no linguistic preprocessing. Long ngrams of characters obtained from raw text turned out to yield the best performance on all textual input (written essays and speech transcripts). Voting ensembles turned out to produce small performance gains, with little difference between the various optimization strategies we tried. Our top systems achieved accuracies of 87{\%} on the essay track, 84{\%} on the speech track, and close to 92{\%} by combining essays, speech and i-vectors in the fusion track.
CIC-FBK Approach to Native Language Identification
Authors: Ilia Markov, Lingzhen Chen, Carlo Strapparava, Grigori Sidorov
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: General Classification, Language Identification, LEMMA, Native Language Identification

We present the CIC-FBK system, which took part in the Native Language Identification (NLI) Shared Task 2017. Our approach combines features commonly used in previous NLI research, i.e., word n-grams, lemma n-grams, part-of-speech n-grams, and function words, with recently introduced character n-grams from misspelled words, and features that are novel in this task, such as typed character n-grams, and syntactic n-grams of words and of syntactic relation tags. We use log-entropy weighting scheme and perform classification using the Support Vector Machines (SVM) algorithm. Our system achieved 0.8808 macro-averaged F1-score and shared the 1st rank in the NLI Shared Task 2017 scoring.