pwc-archive/papers-with-abstracts huggingface dataset:
Delexicalized transfer parsing for low-resource languages using transformed and combined treebanks
Authors: Ayan Das, Affan Zaffar, Sudeshna Sarkar
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Cross-Lingual Transfer, Dependency Parsing, Machine Translation, Question Answering, Semantic Parsing

This paper describes our dependency parsing system in CoNLL-2017 shared task on Multilingual Parsing from Raw Text to Universal Dependencies. We primarily focus on the low-resource languages (surprise languages). We have developed a framework to combine multiple treebanks to train parsers for low resource languages by delexicalization method. We have applied transformation on source language treebanks based on syntactic features of the low-resource language to improve performance of the parser. In the official evaluation, our system achieves an macro-averaged LAS score of 67.61 and 37.16 on the entire blind test data and the surprise language test data respectively.
A Transition-based System for Universal Dependency Parsing
Authors: Hao Wang, Hai Zhao, Zhisong Zhang
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Dependency Parsing, Transfer Learning, Transition-Based Dependency Parsing

This paper describes the system for our participation in the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In this work, we design a system based on UDPipe1 for universal dependency parsing, where multilingual transition-based models are trained for different treebanks. Our system directly takes raw texts as input, performing several intermediate steps like tokenizing and tagging, and finally generates the corresponding dependency trees. For the special surprise languages for this task, we adopt a delexicalized strategy and predict basing on transfer learning from other related languages. In the final evaluation of the shared task, our system achieves a result of 66.53{\%} in macro-averaged LAS F1-score.
Corpus Selection Approaches for Multilingual Parsing from Raw Text to Universal Dependencies
Authors: Ryan Hornby, Clark Taylor, Jungyeul Park
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8

This paper describes UALing{'}s approach to the \textit{CoNLL 2017 UD Shared Task} using corpus selection techniques to reduce training data size. The methodology is simple: we use similarity measures to select a corpus from available training data (even from multiple corpora for surprise languages) and use the resulting corpus to complete the parsing task. The training and parsing is done with the baseline UDPipe system (Straka et al., 2016). While our approach reduces the size of training data significantly, it retains performance within 0.5{\%} of the baseline system. Due to the reduction in training data size, our system performs faster than the na{\"\i}ve, complete corpus method. Specifically, our system runs in less than 10 minutes, ranking it among the fastest entries for this task. Our system is available at \url{https://github.com/CoNLL-UD-2017/UALING}.
From Raw Text to Universal Dependencies - Look, No Tags!
Authors: Miryam de Lhoneux, Yan Shao, Ali Basirat, Eliyahu Kiperwasser, Sara Stymne, Yoav Goldberg, Joakim Nivre
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Dependency Parsing, Part-Of-Speech Tagging, Segmentation, Sentence, Sentence segmentation, TAG, Word Embeddings

We present the Uppsala submission to the CoNLL 2017 shared task on parsing from raw text to universal dependencies. Our system is a simple pipeline consisting of two components. The first performs joint word and sentence segmentation on raw text; the second predicts dependency trees from raw words. The parser bypasses the need for part-of-speech tagging, but uses word embeddings based on universal tag distributions. We achieved a macro-averaged LAS F1 of 65.11 in the official test run, which improved to 70.49 after bug fixes. We obtained the 2nd best result for sentence segmentation with a score of 89.03.
Initial Explorations of CCG Supertagging for Universal Dependency Parsing
Authors: Burak Kerim Akkus, Heval Azizoglu, Ruket Cakici
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: CCG Supertagging, Dependency Parsing, Machine Translation

In this paper we describe the system by METU team for universal dependency parsing of multilingual text. We use a neural network-based dependency parser that has a greedy transition approach to dependency parsing. CCG supertags contain rich structural information that proves useful in certain NLP tasks. We experiment with CCG supertags as additional features in our experiments. The neural network parser is trained together with dependencies and simplified CCG tags as well as other features provided.
CLCL (Geneva) DINN Parser: a Neural Network Dependency Parser Ten Years Later
Authors: Christophe Moor, Paola Merlo, James Henderson, Haozhou Wang
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Dependency Parsing, Feature Engineering, Word Embeddings

This paper describes the University of Geneva{'}s submission to the CoNLL 2017 shared task Multilingual Parsing from Raw Text to Universal Dependencies (listed as the CLCL (Geneva) entry). Our submitted parsing system is the grandchild of the first transition-based neural network dependency parser, which was the University of Geneva{'}s entry in the CoNLL 2007 multilingual dependency parsing shared task, with some improvements to speed and portability. These results provide a baseline for investigating how far we have come in the past ten years of work on neural network dependency parsing.
A Fast and Lightweight System for Multilingual Dependency Parsing
Authors: Tao Ji, Yuanbin Wu, Man Lan
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Dependency Parsing

We present a multilingual dependency parser with a bidirectional-LSTM (BiLSTM) feature extractor and a multi-layer perceptron (MLP) classifier. We trained our transition-based projective parser in UD version 2.0 datasets without any additional data. The parser is fast, lightweight and effective on big treebanks. In the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, the official results show that the macro-averaged LAS F1 score of our system Mengest is 61.33{\%}.
The ParisNLP entry at the ConLL UD Shared Task 2017: A Tale of a \#ParsingTragedy
Authors: {\'E}ric de la Clergerie, Beno{\^\i}t Sagot, Djam{\'e} Seddah
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Dependency Parsing, Sentence

We present the ParisNLP entry at the UD CoNLL 2017 parsing shared task. In addition to the UDpipe models provided, we built our own data-driven tokenization models, sentence segmenter and lexicon-based morphological analyzers. All of these were used with a range of different parsing models (neural or not, feature-rich or not, transition or graph-based, etc.) and the best combination for each language was selected. Unfortunately, a glitch in the shared task{'}s Matrix led our model selector to run generic, weakly lexicalized models, tailored for surprise languages, instead of our dataset-specific models. Because of this {\#}ParsingTragedy, we officially ranked 27th, whereas our real models finally unofficially ranked 6th.
Universal Joint Morph-Syntactic Processing: The Open University of Israel's Submission to The CoNLL 2017 Shared Task
Authors: Amir More, Reut Tsarfaty
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: MORPH, Sentence, Sentence segmentation, Word Embeddings

We present the Open University{'}s submission to the CoNLL 2017 Shared Task on multilingual parsing from raw text to Universal Dependencies. The core of our system is a joint morphological disambiguator and syntactic parser which accepts morphologically analyzed surface tokens as input and returns morphologically disambiguated dependency trees as output. Our parser requires a lattice as input, so we generate morphological analyses of surface tokens using a data-driven morphological analyzer that derives its lexicon from the UD training corpora, and we rely on UDPipe for sentence segmentation and surface-level tokenization. We report our official macro-average LAS is 56.56. Although our model is not as performant as many others, it does not make use of neural networks, therefore we do not rely on word embeddings or any other data source other than the corpora themselves. In addition, we show the utility of a lexicon-backed morphological analyzer for the MRL Modern Hebrew. We use our results on Modern Hebrew to argue that the UD community should define a UD-compatible standard for access to lexical resources, which we argue is crucial for MRLs and low resource languages in particular.
A Semi-universal Pipelined Approach to the CoNLL 2017 UD Shared Task
Authors: Hiroshi Kanayama, Masayasu Muraoka, Katsumasa Yoshikawa
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Cross-Lingual Transfer, Language Modeling, Language Modelling, Word Embeddings

This paper presents our system submitted for the CoNLL 2017 Shared Task, {``}Multilingual Parsing from Raw Text to Universal Dependencies.{''} We ran the system for all languages with our own fully pipelined components without relying on re-trained baseline systems. To train the dependency parser, we used only the universal part-of-speech tags and distance between words, and applied deterministic rules to assign dependency labels. The simple and delexicalized models are suitable for cross-lingual transfer approaches and a universal language model. Experimental results show that our model performed well in some metrics and leads discussion on topics such as contribution of each component and on syntactic similarities among languages.
A rule-based system for cross-lingual parsing of Romance languages with Universal Dependencies
Authors: Marcos Garcia, Pablo Gamallo
Date: 2017-08-01 00:00:00
Proceeding: CONLL 2017 8
Tasks: Dependency Parsing, POS, TAG

This article describes MetaRomance, a rule-based cross-lingual parser for Romance languages submitted to CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. The system is an almost delexicalized parser which does not need training data to analyze Romance languages. It contains linguistically motivated rules based on PoS-tag patterns. The rules included in MetaRomance were developed in about 12 hours by one expert with no prior knowledge in Universal Dependencies, and can be easily extended using a transparent formalism. In this paper we compare the performance of MetaRomance with other supervised systems participating in the competition, paying special attention to the parsing of different treebanks of the same language. We also compare our system with a delexicalized parser for Romance languages, and take advantage of the harmonized annotation of Universal Dependencies to propose a language ranking based on the syntactic distance each variety has from Romance languages.
Using exemplar responses for training and evaluating automated speech scoring systems
Authors: Anastassia Loukina, Klaus Zechner, James Bruno, Beata Beigman Klebanov
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6

Automated scoring engines are usually trained and evaluated against human scores and compared to the benchmark of human-human agreement. In this paper we compare the performance of an automated speech scoring engine using two corpora: a corpus of almost 700,000 randomly sampled spoken responses with scores assigned by one or two raters during operational scoring, and a corpus of 16,500 exemplar responses with scores reviewed by multiple expert raters. We show that the choice of corpus used for model evaluation has a major effect on estimates of system performance with r varying between 0.64 and 0.80. Surprisingly, this is not the case for the choice of corpus for model training: when the training corpus is sufficiently large, the systems trained on different corpora showed almost identical performance when evaluated on the same corpus. We show that this effect is consistent across several learning algorithms. We conclude that evaluating the model on a corpus of exemplar responses if one is available provides additional evidence about system validity; at the same time, investing effort into creating a corpus of exemplar responses for model training is unlikely to lead to a substantial gain in model performance.
Using Paraphrasing and Memory-Augmented Models to Combat Data Sparsity in Question Interpretation with a Virtual Patient Dialogue System
Authors: Lifeng Jin, David King, Amad Hussein, Michael White, Douglas Danforth
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Data Augmentation, General Classification, One-Shot Learning

When interpreting questions in a virtual patient dialogue system one must inevitably tackle the challenge of a long tail of relatively infrequently asked questions. To make progress on this challenge, we investigate the use of paraphrasing for data augmentation and neural memory-based classification, finding that the two methods work best in combination. In particular, we find that the neural memory-based approach not only outperforms a straight CNN classifier on low frequency questions, but also takes better advantage of the augmented data created by paraphrasing, together yielding a nearly 10{\%} absolute improvement in accuracy on the least frequently asked questions.
Predicting misreadings from gaze in children with reading difficulties
Authors: Joachim Bingel, Maria Barrett, Sigrid Klerke
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Multi-Task Learning, Reading Comprehension, Text Simplification

We present the first work on predicting reading mistakes in children with reading difficulties based on eye-tracking data from real-world reading teaching. Our approach employs several linguistic and gaze-based features to inform an ensemble of different classifiers, including multi-task learning models that let us transfer knowledge about individual readers to attain better predictions. Notably, the data we use in this work stems from noisy readings in the wild, outside of controlled lab conditions. Our experiments show that despite the noise and despite the small fraction of misreadings, gaze data improves the performance more than any other feature group and our models achieve good performance. We further show that gaze patterns for misread words do not fully generalize across readers, but that we can transfer some knowledge between readers using multitask learning at least in some cases. Applications of our models include partial automation of reading assessment as well as personalized text simplification.
Automatic Input Enrichment for Selecting Reading Material: An Online Study with English Teachers
Authors: Maria Chinkina, Ankita Oswal, Detmar Meurers
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition

Input material at the appropriate level is crucial for language acquisition. Automating the search for such material can systematically and efficiently support teachers in their pedagogical practice. This is the goal of the computational linguistic task of automatic input enrichment (Chinkina {\&} Meurers, 2016): It analyzes and re-ranks a collection of texts in order to prioritize those containing target linguistic forms. In the online study described in the paper, we collected 240 responses from English teachers in order to investigate whether they preferred automatic input enrichment over web search when selecting reading material for class. Participants demonstrated a general preference for the material provided by an automatic input enrichment system. It was also rated significantly higher than the texts retrieved by a standard web search engine with regard to the representation of linguistic forms and equivalent with regard to the relevance of the content to the topic. We discuss the implications of the results for language teaching and consider the potential strands of future research.
Estimating Linguistic Complexity for Science Texts
Authors: Farah Nadeem, Mari Ostendorf
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Feature Engineering, Reading Comprehension, Text Simplification

Evaluation of text difficulty is important both for downstream tasks like text simplification, and for supporting educators in classrooms. Existing work on automated text complexity analysis uses linear models with engineered knowledge-driven features as inputs. While this offers interpretability, these models have lower accuracy for shorter texts. Traditional readability metrics have the additional drawback of not generalizing to informational texts such as science. We propose a neural approach, training on science and other informational texts, to mitigate both problems. Our results show that neural methods outperform knowledge-based linear models for short texts, and have the capacity to generalize to genres not present in the training data.
Second Language Acquisition Modeling
Authors: Burr Settles, Chris Brust, Erin Gustafson, Masato Hagiwara, Nitin Madnani
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Knowledge Tracing, Language Acquisition

We present the task of \textit{second language acquisition (SLA) modeling}. Given a history of errors made by learners of a second language, the task is to predict errors that they are likely to make at arbitrary points in the future. We describe a large corpus of more than 7M words produced by more than 6k learners of English, Spanish, and French using Duolingo, a popular online language-learning app. Then we report on the results of a shared task challenge aimed studying the SLA task via this corpus, which attracted 15 teams and synthesized work from various fields including cognitive science, linguistics, and machine learning.
Towards Single Word Lexical Complexity Prediction
Authors: David Alfter, Elena Volodina
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: General Classification, Lexical Complexity Prediction, Prediction

In this paper we present work-in-progress where we investigate the usefulness of previously created word lists to the task of single-word lexical complexity analysis and prediction of the complexity level for learners of Swedish as a second language. The word lists used map each word to a single CEFR level, and the task consists of predicting CEFR levels for unseen words. In contrast to previous work on word-level lexical complexity, we experiment with topics as additional features and show that linking words to topics significantly increases accuracy of classification.
COAST - Customizable Online Syllable Enhancement in Texts. A flexible framework for automatically enhancing reading materials
Authors: Heiko Holz, Zarah Weiss, Oliver Brehm, Detmar Meurers
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition

This paper presents COAST, a web-based application to easily and automatically enhance syllable structure, word stress, and spacing in texts, that was designed in close collaboration with learning therapists to ensure its practical relevance. Such syllable-enhanced texts are commonly used in learning therapy or private tuition to promote the recognition of syllables in order to improve reading and writing skills. In a state of the art solutions for automatic syllable enhancement, we put special emphasis on syllable stress and support specific marking of the primary syllable stress in words. Core features of our tool are i) a highly customizable text enhancement and template functionality, and ii) a novel crowd-sourcing mechanism that we employ to address the issue of data sparsity in language resources. We successfully tested COAST with real-life practitioners in a series of user tests validating the concept of our framework.
Annotating picture description task responses for content analysis
Authors: Levi King, Markus Dickinson
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Form, Reading Comprehension

Given that all users of a language can be creative in their language usage, the overarching goal of this work is to investigate issues of variability and acceptability in written text, for both non-native speakers (NNSs) and native speakers (NSs). We control for meaning by collecting a dataset of picture description task (PDT) responses from a number of NSs and NNSs, and we define and annotate a handful of features pertaining to form and meaning, to capture the multi-dimensional ways in which responses can vary and can be acceptable. By examining the decisions made in this corpus development, we highlight the questions facing anyone working with learner language properties like variability, acceptability and native-likeness. We find reliable inter-annotator agreement, though disagreements point to difficult areas for establishing a link between form and meaning.
Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models
Authors: Vikram Ramanarayanan, Michelle LaMar
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Management

While dialog systems have been widely deployed for computer-assisted language learning (CALL) and formative assessment systems in recent years, relatively limited work has been done with respect to the psychometrics and validity of these technologies in evaluating and providing feedback regarding student learning and conversational ability. This paper formulates a Markov decision process based measurement model, and applies it to text chat data collected from crowdsourced native and non-native English language speakers interacting with an automated dialog agent. We investigate how well the model measures speaker conversational ability, and find that it effectively captures the differences in how native and non-native speakers of English accomplish the dialog task. Such models could have important implications for CALL systems of the future that effectively combine dialog management with measurement of learner conversational ability in real-time.
Generating Feedback for English Foreign Language Exercises
Authors: Bj{\"o}rn Rudzewitz, Ramon Ziai, Kordula De Kuthy, Verena M{\"o}ller, Florian Nuxoll, Detmar Meurers
Date: 2018-06-01 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition, Reading Comprehension

While immediate feedback on learner language is often discussed in the Second Language Acquisition literature (e.g., Mackey 2006), few systems used in real-life educational settings provide helpful, metalinguistic feedback to learners. In this paper, we present a novel approach leveraging task information to generate the expected range of well-formed and ill-formed variability in learner answers along with the required diagnosis and feedback. We combine this offline generation approach with an online component that matches the actual student answers against the pre-computed hypotheses. The results obtained for a set of 33 thousand answers of 7th grade German high school students learning English show that the approach successfully covers frequent answer patterns. At the same time, paraphrases and content errors require a more flexible alignment approach, for which we are planning to complement the method with the CoMiC approach successfully used for the analysis of reading comprehension answers (Meurers et al., 2011).