pwc-archive/papers-with-abstracts huggingface dataset:
NT2Lex: A CEFR-Graded Lexical Resource for Dutch as a Foreign Language Linked to Open Dutch WordNet
Authors: Ana{\"\i}s Tack, Thomas Fran{\c{c}}ois, Piet Desmet, C{\'e}drick Fairon
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification

In this paper, we introduce NT2Lex, a novel lexical resource for Dutch as a foreign language (NT2) which includes frequency distributions of 17,743 words and expressions attested in expert-written textbook texts and readers graded along the scale of the Common European Framework of Reference (CEFR). In essence, the lexicon informs us about what kind of vocabulary should be understood when reading Dutch as a non-native reader at a particular proficiency level. The main novelty of the resource with respect to the previously developed CEFR-graded lexicons concerns the introduction of corpus-based evidence for L2 word sense complexity through the linkage to Open Dutch WordNet (Postma et al., 2016). The resource thus contains, on top of the lemmatised and part-of-speech tagged lexical entries, a total of 11,999 unique word senses and 8,934 distinct synsets.
Chengyu Cloze Test
Authors: Zhiying Jiang, Boliang Zhang, Lifu Huang, Heng Ji
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Cloze Test, Sentence

We present a neural recommendation model for Chengyu, which is a special type of Chinese idiom. Given a query, which is a sentence with an empty slot where the Chengyu is taken out, our model will recommend the best Chengyu candidate that best fits the slot context. The main challenge lies in that the literal meaning of a Chengyu is usually very different from it{'}s figurative meaning. We propose a new neural approach to leverage the definition of each Chengyu and incorporate it as background knowledge. Experiments on both Chengyu cloze test and coherence checking in college entrance exams show that our system achieves 89.5{\%} accuracy on cloze test and outperforms human subjects who attended competitive universities in China. We will make all of our data sets and resources publicly available as a new benchmark for research purposes.
LaSTUS/TALN at Complex Word Identification (CWI) 2018 Shared Task
Authors: Ahmed Abura{'}ed, Horacio Saggion
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Lexical Simplification, Sentence, Text Simplification

This paper presents the participation of the LaSTUS/TALN team in the Complex Word Identification (CWI) Shared Task 2018 in the English monolingual track . The purpose of the task was to determine if a word in a given sentence can be judged as complex or not by a certain target audience. For the English track, task organizers provided a training and a development datasets of 27,299 and 3,328 words respectively together with the sentence in which each word occurs. The words were judged as complex or not by 20 human evaluators; ten of whom are natives. We submitted two systems: one system modeled each word to evaluate as a numeric vector populated with a set of lexical, semantic and contextual features while the other system relies on a word embedding representation and a distance metric. We trained two separate classifiers to automatically decide if each word is complex or not. We submitted six runs, two for each of the three subsets of the English monolingual CWI track.
Cross-lingual complex word identification with multitask learning
Authors: Joachim Bingel, Johannes Bjerva
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Lexical Simplification

We approach the 2018 Shared Task on Complex Word Identification by leveraging a cross-lingual multitask learning approach. Our method is highly language agnostic, as evidenced by the ability of our system to generalize across languages, including languages for which we have no training data. In the shared task, this is the case for French, for which our system achieves the best performance. We further provide a qualitative and quantitative analysis of which words pose problems for our system.
CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting
Authors: Sian Gooding, Ekaterina Kochmar
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Binary Classification, Complex Word Identification, General Classification, Lexical Simplification, Reading Comprehension, Text Simplification

This paper presents the winning systems we submitted to the Complex Word Identification Shared Task 2018. We describe our best performing systems{'} implementations and discuss our key findings from this research. Our best-performing systems achieve an F1 score of 0.8792 on the NEWS, 0.8430 on the WIKINEWS and 0.8115 on the WIKIPEDIA test sets in the monolingual English binary classification track, and a mean absolute error of 0.0558 on the NEWS, 0.0674 on the WIKINEWS and 0.0739 on the WIKIPEDIA test sets in the probabilistic track.
Complex Word Identification Based on Frequency in a Learner Corpus
Authors: Tomoyuki Kajiwara, Mamoru Komachi
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Lexical Simplification, Reading Comprehension, Text Simplification

We introduce the TMU systems for the Complex Word Identification (CWI) Shared Task 2018. TMU systems use random forest classifiers and regressors whose features are the number of characters, the number of words, and the frequency of target words in various corpora. Our simple systems performed best on 5 tracks out of 12 tracks. Our ablation analysis revealed the usefulness of a learner corpus for CWI task.
The Whole is Greater than the Sum of its Parts: Towards the Effectiveness of Voting Ensemble Classifiers for Complex Word Identification
Authors: Nikhil Wani, S Mathias, eep, Jayashree Aan Gajjam, Pushpak Bhattacharyya
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Lexical Simplification

In this paper, we present an effective system using voting ensemble classifiers to detect contextually complex words for non-native English speakers. To make the final decision, we channel a set of eight calibrated classifiers based on lexical, size and vocabulary features and train our model with annotated datasets collected from a mixture of native and non-native speakers. Thereafter, we test our system on three datasets namely News, WikiNews, and Wikipedia and report competitive results with an F1-Score ranging between 0.777 to 0.855 for each of the datasets. Our system outperforms multiple other models and falls within 0.042 to 0.026 percent of the best-performing model{'}s score in the shared task.
Grotoco@SLAM: Second Language Acquisition Modeling with Simple Features, Learners and Task-wise Models
Authors: Sigrid Klerke, H{\'e}ctor Mart{\'\i}nez Alonso, Barbara Plank
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition

We present our submission to the 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM). We focus on evaluating a range of features for the task, including user-derived measures, while examining how far we can get with a simple linear classifier. Our analysis reveals that errors differ per exercise format, which motivates our final and best-performing system: a task-wise (per exercise-format) model.
Context Based Approach for Second Language Acquisition
Authors: Nihal V. Nayak, Arjun R. Rao
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: es-en, fr-en, Language Acquisition

SLAM 2018 focuses on predicting a student{'}s mistake while using the Duolingo application. In this paper, we describe the system we developed for this shared task. Our system uses a logistic regression model to predict the likelihood of a student making a mistake while answering an exercise on Duolingo in all three language tracks - English/Spanish (en/es), Spanish/English (es/en) and French/English (fr/en). We conduct an ablation study with several features during the development of this system and discover that context based features plays a major role in language acquisition modeling. Our model beats Duolingo{'}s baseline scores in all three language tracks (AUROC scores for en/es = 0.821, es/en = 0.790 and fr/en = 0.812). Our work makes a case for providing favourable textual context for students while learning second language.
Modeling Second-Language Learning from a Psychological Perspective
Authors: Alex Rich, er, Pamela Osborn Popp, David Halpern, Anselm Rothe, Todd Gureckis
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition

Psychological research on learning and memory has tended to emphasize small-scale laboratory studies. However, large datasets of people using educational software provide opportunities to explore these issues from a new perspective. In this paper we describe our approach to the Duolingo Second Language Acquisition Modeling (SLAM) competition which was run in early 2018. We used a well-known class of algorithms (gradient boosted decision trees), with features partially informed by theories from the psychological literature. After detailing our modeling approach and a number of supplementary simulations, we reflect on the degree to which psychological theory aided the model, and the potential for cognitive science and predictive modeling competitions to gain from each other.
A Memory-Sensitive Classification Model of Errors in Early Second Language Learning
Authors: Brendan Tomoschuk, Jarrett Lovelett
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: General Classification, Language Acquisition

In this paper, we explore a variety of linguistic and cognitive features to better understand second language acquisition in early users of the language learning app Duolingo. With these features, we trained a random forest classifier to predict errors in early learners of French, Spanish, and English. Of particular note was our finding that mean and variance in error for each user and token can be a memory efficient replacement for their respective dummy-encoded categorical variables. At test, these models improved over the baseline model with AUROC values of 0.803 for English, 0.823 for French, and 0.829 for Spanish.
Language Model Based Grammatical Error Correction without Annotated Training Data
Authors: Christopher Bryant, Ted Briscoe
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Grammatical Error Correction, Language Modeling, Language Modelling, Machine Translation

Since the end of the CoNLL-2014 shared task on grammatical error correction (GEC), research into language model (LM) based approaches to GEC has largely stagnated. In this paper, we re-examine LMs in GEC and show that it is entirely possible to build a simple system that not only requires minimal annotated data (âˆ¼1000 sentences), but is also fairly competitive with several state-of-the-art systems. This approach should be of particular interest for languages where very little annotated training data exists, although we also hope to use it as a baseline to motivate future research.
A Semantic Role-based Approach to Open-Domain Automatic Question Generation
Authors: Michael Flor, Brian Riordan
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Dependency Parsing, Question Generation, Question-Generation, Reading Comprehension, Semantic Role Labeling

We present a novel rule-based system for automatic generation of factual questions from sentences, using semantic role labeling (SRL) as the main form of text analysis. The system is capable of generating both wh-questions and yes/no questions from the same semantic analysis. We present an extensive evaluation of the system and compare it to a recent neural network architecture for question generation. The SRL-based system outperforms the neural system in both average quality and variety of generated questions.
Automated Content Analysis: A Case Study of Computer Science Student Summaries
Authors: Yanjun Gao, Patricia M. Davies, Rebecca J. Passonneau
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6

Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess precis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearson{'}s correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.
Toward Data-Driven Tutorial Question Answering with Deep Learning Conversational Models
Authors: Mayank Kulkarni, Kristy Boyer
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Information Retrieval, Question Answering, Retrieval, Semantic Parsing, Transfer Learning

There has been an increase in popularity of data-driven question answering systems given their recent success. This pa-per explores the possibility of building a tutorial question answering system for Java programming from data sampled from a community-based question answering forum. This paper reports on the creation of a dataset that could support building such a tutorial question answering system and discusses the methodology to create the 106,386 question strong dataset. We investigate how retrieval-based and generative models perform on the given dataset. The work also investigates the usefulness of using hybrid approaches such as combining retrieval-based and generative models. The results indicate that building data-driven tutorial systems using community-based question answering forums holds significant promise.
Distractor Generation for Multiple Choice Questions Using Learning to Rank
Authors: Chen Liang, Xiao Yang, Neisarg Dave, Drew Wham, Bart Pursel, C. Lee Giles
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: BIG-bench Machine Learning, Distractor Generation, Ensemble Learning, Learning-To-Rank, Multiple-choice

We investigate how machine learning models, specifically ranking models, can be used to select useful distractors for multiple choice questions. Our proposed models can learn to select distractors that resemble those in actual exam questions, which is different from most existing unsupervised ontology-based and similarity-based methods. We empirically study feature-based and neural net (NN) based ranking models with experiments on the recently released SciQ dataset and our MCQL dataset. Experimental results show that feature-based ensemble learning methods (random forest and LambdaMART) outperform both the NN-based method and unsupervised baselines. These two datasets can also be used as benchmarks for distractor generation.
OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification
Authors: Sowmya Vajjala, Ivana Lu{\v{c}}i{\'c}
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Feature Engineering, Text Classification, Text Simplification

This paper describes the collection and compilation of the OneStopEnglish corpus of texts written at three reading levels, and demonstrates its usefulness for through two applications - automatic readability assessment and automatic text simplification. The corpus consists of 189 texts, each in three versions (567 in total). The corpus is now freely available under a CC by-SA 4.0 license and we hope that it would foster further research on the topics of readability assessment and text simplification.
The Effect of Adding Authorship Knowledge in Automated Text Scoring
Authors: Meng Zhang, Xie Chen, Ronan Cummins, {\O}istein E. Andersen, Ted Briscoe
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6

Some language exams have multiple writing tasks. When a learner writes multiple texts in a language exam, it is not surprising that the quality of these texts tends to be similar, and the existing automated text scoring (ATS) systems do not explicitly model this similarity. In this paper, we suggest that it could be useful to include the other texts written by this learner in the same exam as extra references in an ATS system. We propose various approaches of fusing information from multiple tasks and pass this authorship knowledge into our ATS model on six different datasets. We show that this can positively affect the model performance at a global level.
SB@GU at the Complex Word Identification 2018 Shared Task
Authors: David Alfter, Ildik{\'o} Pil{\'a}n
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, feature selection, General Classification, Language Modeling, Language Modelling, Text Simplification, Word Embeddings

In this paper, we describe our experiments for the Shared Task on Complex Word Identification (CWI) 2018 (Yimam et al., 2018), hosted by the 13th Workshop on Innovative Use of NLP for Building Educational Applications (BEA) at NAACL 2018. Our system for English builds on previous work for Swedish concerning the classification of words into proficiency levels. We investigate different features for English and compare their usefulness using feature selection methods. For the German, Spanish and French data we use simple systems based on character n-gram models and show that sometimes simple models achieve comparable results to fully feature-engineered systems.
Complex Word Identification: Convolutional Neural Network vs. Feature Engineering
Authors: Segun Taofeek Aroyehun, Jason Angel, Daniel Alej P{\'e}rez Alvarez, ro, Alex Gelbukh, er
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Feature Engineering, Text Simplification

We describe the systems of NLP-CIC team that participated in the Complex Word Identification (CWI) 2018 shared task. The shared task aimed to benchmark approaches for identifying complex words in English and other languages from the perspective of non-native speakers. Our goal is to compare two approaches: feature engineering and a deep neural network. Both approaches achieved comparable performance on the English test set. We demonstrated the flexibility of the deep-learning approach by using the same deep neural network setup in the Spanish track. Our systems achieved competitive results: all our systems were within 0.01 of the system with the best macro-F1 score on the test sets except on Wikipedia test set, on which our best system is 0.04 below the best macro-F1 score.
Deep Learning Architecture for Complex Word Identification
Authors: Dirk De Hertog, Ana{\"\i}s Tack
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: All, Binary Classification, Complex Word Identification, Deep Learning, General Classification, Lexical Simplification

We describe a system for the CWI-task that includes information on 5 aspects of the (complex) lexical item, namely distributional information of the item itself, morphological structure, psychological measures, corpus-counts and topical information. We constructed a deep learning architecture that combines those features and apply it to the probabilistic and binary classification task for all English sets and Spanish. We achieved reasonable performance on all sets with best performances seen on the probabilistic task, particularly on the English news set (MAE 0.054 and F1-score of 0.872). An analysis of the results shows that reasonable performance can be achieved with a single architecture without any domain-specific tweaking of the parameter settings and that distributional features capture almost all of the information also found in hand-crafted features.
NILC at CWI 2018: Exploring Feature Engineering and Feature Learning
Authors: Nathan Hartmann, Le dos Santos, ro Borges
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Feature Engineering, General Classification, Language Modeling, Language Modelling, Lexical Simplification, Text Simplification, Word Embeddings

This paper describes the results of NILC team at CWI 2018. We developed solutions following three approaches: (i) a feature engineering method using lexical, n-gram and psycholinguistic features, (ii) a shallow neural network method using only word embeddings, and (iii) a Long Short-Term Memory (LSTM) language model, which is pre-trained on a large text corpus to produce a contextualized word vector. The feature engineering method obtained our best results for the classification task and the LSTM model achieved the best results for the probabilistic classification task. Our results show that deep neural networks are able to perform as well as traditional machine learning methods using manually engineered features for the task of complex word identification in English.
Complex Word Identification Using Character n-grams
Authors: Maja Popovi{\'c}
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Complex Word Identification, Lexical Simplification, Machine Translation, Text Classification

This paper investigates the use of character n-gram frequencies for identifying complex words in English, German and Spanish texts. The approach is based on the assumption that complex words are likely to contain different character sequences than simple words. The multinomial Naive Bayes classifier was used with n-grams of different lengths as features, and the best results were obtained for the combination of 2-grams and 4-grams. This variant was submitted to the Complex Word Identification Shared Task 2018 for all texts and achieved F-scores between 70{\%} and 83{\%}. The system was ranked in the middle range for all English texts, as third of fourteen submissions for German, and as tenth of seventeen submissions for Spanish. The method is not very convenient for the cross-language task, achieving only 59{\%} on the French text.
Predicting Second Language Learner Successes and Mistakes by Means of Conjunctive Features
Authors: Yves Bestgen
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition

This paper describes the system developed by the Centre for English Corpus Linguistics for the 2018 Duolingo SLAM challenge. It aimed at predicting the successes and mistakes of second language learners on each of the words that compose the exercises they answered. Its main characteristic is to include conjunctive features, built by combining word ngrams with metadata about the user and the exercise. It achieved a relatively good performance, ranking fifth out of 15 systems. Complementary analyses carried out to gauge the contribution of the different sets of features to the performance confirmed the usefulness of the conjunctive features for the SLAM task.
Feature Engineering for Second Language Acquisition Modeling
Authors: Guanliang Chen, Claudia Hauff, Geert-Jan Houben
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Feature Engineering, Knowledge Tracing, Language Acquisition

Knowledge tracing serves as a keystone in delivering personalized education. However, few works attempted to model students{'} knowledge state in the setting of Second Language Acquisition. The Duolingo Shared Task on Second Language Acquisition Modeling provides students{'} trace data that we extensively analyze and engineer features from for the task of predicting whether a student will correctly solve a vocabulary exercise. Our analyses of students{'} learning traces reveal that factors like exercise format and engagement impact their exercise performance to a large extent. Overall, we extracted 23 different features as input to a Gradient Tree Boosting framework, which resulted in an AUC score of between 0.80 and 0.82 on the official test set.