pwc-archive/papers-with-abstracts huggingface dataset:
On Integrating Discourse in Machine Translation
Authors: Karin Sim Smith
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Machine Translation, Translation

As the quality of Machine Translation (MT) improves, research on improving discourse in automatic translations becomes more viable. This has resulted in an increase in the amount of work on discourse in MT. However many of the existing models and metrics have yet to integrate these insights. Part of this is due to the evaluation methodology, based as it is largely on matching to a single reference. At a time when MT is increasingly being used in a pipeline for other tasks, the semantic element of the translation process needs to be properly integrated into the task. Moreover, in order to take MT to another level, it will need to judge output not based on a single reference translation, but based on notions of fluency and of adequacy {--} ideally with reference to the source text.
From Shakespeare to Twitter: What are Language Styles all about?
Authors: Wei Xu
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: All, Lexical Simplification, Position, Question Answering, Text Simplification

As natural language processing research is growing and largely driven by the availability of data, we expanded research from news and small-scale dialog corpora to web and social media. User-generated data and crowdsourcing opened the door for investigating human language of various styles with more statistical power and real-world applications. In this position/survey paper, I will review and discuss seven language styles that I believe to be important and interesting to study: influential work in the past, challenges at the present, and potential impact for the future.
Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models
Authors: Harsh Jhamtani, Varun Gangal, Eduard Hovy, Eric Nyberg
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9

Variations in writing styles are commonly used to adapt the content to a specific context, audience, or purpose. However, applying stylistic variations is still by and large a manual process, and there have been little efforts towards automating it. In this paper we explore automated methods to transform text from modern English to Shakespearean English using an end to end trainable neural model with pointers to enable copy action. To tackle limited amount of parallel data, we pre-train embeddings of words by leveraging external dictionaries mapping Shakespearean words to modern English words as well as additional text. Our methods are able to get a BLEU score of 31+, an improvement of {\mbox{$\approx$}} 6 points above the strongest baseline. We publicly release our code to foster further research in this area.
Discovering Stylistic Variations in Distributional Vector Space Models via Lexical Paraphrases
Authors: Xing Niu, Marine Carpuat
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Semantic Textual Similarity, Word Embeddings

Detecting and analyzing stylistic variation in language is relevant to diverse Natural Language Processing applications. In this work, we investigate whether salient dimensions of style variations are embedded in standard distributional vector spaces of word meaning. We hypothesizes that distances between embeddings of lexical paraphrases can help isolate style from meaning variations and help identify latent style dimensions. We conduct a qualitative analysis of latent style dimensions, and show the effectiveness of identified style subspaces on a lexical formality prediction task.
Is writing style predictive of scientific fraud?
Authors: Chlo{\'e} Braud, Anders S{\o}gaard
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Logical Reasoning

The problem of detecting scientific fraud using machine learning was recently introduced, with initial, positive results from a model taking into account various general indicators. The results seem to suggest that writing style is predictive of scientific fraud. We revisit these initial experiments, and show that the leave-one-out testing procedure they used likely leads to a slight over-estimate of the predictability, but also that simple models can outperform their proposed model by some margin. We go on to explore more abstract linguistic features, such as linguistic complexity and discourse structure, only to obtain negative results. Upon analyzing our models, we do see some interesting patterns, though: Scientific fraud, for examples, contains less comparison, as well as different types of hedging and ways of presenting logical reasoning.
``Deep'' Learning : Detecting Metaphoricity in Adjective-Noun Pairs
Authors: Yuri Bizzoni, Stergios Chatzikyriakidis, Mehdi Ghanimifard
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Deep Learning, Transfer Learning

Metaphor is one of the most studied and widespread figures of speech and an essential element of individual style. In this paper we look at metaphor identification in Adjective-Noun pairs. We show that using a single neural network combined with pre-trained vector embeddings can outperform the state of the art in terms of accuracy. In specific, the approach presented in this paper is based on two ideas: a) transfer learning via using pre-trained vectors representing adjective noun pairs, and b) a neural network as a model of composition that predicts a metaphoricity score as output. We present several different architectures for our system and evaluate their performances. Variations on dataset size and on the kinds of embeddings are also investigated. We show considerable improvement over the previous approaches both in terms of accuracy and w.r.t the size of annotated training data.
Authorship Attribution with Convolutional Neural Networks and POS-Eliding
Authors: Julian Hitschler, Esther van den Berg, Ines Rehbein
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Authorship Attribution, POS

We use a convolutional neural network to perform authorship identification on a very homogeneous dataset of scientific publications. In order to investigate the effect of domain biases, we obscure words below a certain frequency threshold, retaining only their POS-tags. This procedure improves test performance due to better generalization on unseen data. Using our method, we are able to predict the authors of scientific publications in the same discipline at levels well above chance.
Topic and audience effects on distinctively Scottish vocabulary usage in Twitter data
Authors: Philippa Shoemark, James Kirby, Sharon Goldwater
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9

Sociolinguistic research suggests that speakers modulate their language style in response to their audience. Similar effects have recently been claimed to occur in the informal written context of Twitter, with users choosing less region-specific and non-standard vocabulary when addressing larger audiences. However, these studies have not carefully controlled for the possible confound of topic: that is, tweets addressed to a broad audience might also tend towards topics that engender a more formal style. In addition, it is not clear to what extent previous results generalize to different samples of users. Using mixed-effects models, we show that audience and topic have independent effects on the rate of distinctively Scottish usage in two demographically distinct Twitter user samples. However, not all effects are consistent between the two groups, underscoring the importance of replicating studies on distinct user samples before drawing strong conclusions from social media data.
Differences in type-token ratio and part-of-speech frequencies in male and female Russian written texts
Authors: Tatiana Litvinova, Pavel Seredin, Olga Litvinova, Olga Zagorovskaya
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Diversity, POS

The differences in the frequencies of some parts of speech (POS), particularly function words, and lexical diversity in male and female speech have been pointed out in a number of papers. The classifiers using exclusively context-independent parameters have proved to be highly effective. However, there are still issues that have to be addressed as a lot of studies are performed for English and the genre and topic of texts is sometimes neglected. The aim of this paper is to investigate the association between context-independent parameters of Russian written texts and the gender of their authors and to design predictive re-gression models. A number of correlations were found. The obtained data is in good agreement with the results obtained for other languages. The model based on 5 parameters with the highest correlation coefficients was designed.
Modeling Communicative Purpose with Functional Style: Corpus and Features for German Genre and Register Analysis
Authors: Thomas Haider, Alexis Palmer
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Genre classification

While there is wide acknowledgement in NLP of the utility of document characterization by genre, it is quite difficult to determine a definitive set of features or even a comprehensive list of genres. This paper addresses both issues. First, with prototype semantics, we develop a hierarchical taxonomy of discourse functions. We implement the taxonomy by developing a new text genre corpus of contemporary German to perform a text based comparative register analysis. Second, we extract a host of style features, both deep and shallow, aiming beyond linguistically motivated features at situational correlates in texts. The feature sets are used for supervised text genre classification, on which our models achieve high accuracy. The combination of the corpus typology and feature sets allows us to characterize types of communicative purpose in a comparative setup, by qualitative interpretation of style feature loadings of a regularized discriminant analysis. Finally, to determine the dependence of genre on topics (which are arguably the distinguishing factor of sub-genre), we compare and combine our style models with Latent Dirichlet Allocation features across different corpus settings with unstable topics.
Stylistic Variation in Television Dialogue for Natural Language Generation
Authors: Grace Lin, Marilyn Walker
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Language Modeling, Language Modelling, Text Generation

Conversation is a critical component of storytelling, where key information is often revealed by what/how a character says it. We focus on the issue of character voice and build stylistic models with linguistic features related to natural language generation decisions. Using a dialogue corpus of the television series, The Big Bang Theory, we apply content analysis to extract relevant linguistic features to build character-based stylistic models, and we test the model-fit through an user perceptual experiment with Amazon{'}s Mechanical Turk. The results are encouraging in that human subjects tend to perceive the generated utterances as being more similar to the character they are modeled on, than to another random character.
Approximating Style by N-gram-based Annotation
Authors: Melanie Andresen, Heike Zinsmeister
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Authorship Attribution

The concept of style is much debated in theoretical as well as empirical terms. From an empirical perspective, the key question is how to operationalize style and thus make it accessible for annotation and quantification. In authorship attribution, many different approaches have successfully resolved this issue at the cost of linguistic interpretability: The resulting algorithms may be able to distinguish one language variety from the other, but do not give us much information on their distinctive linguistic properties. We approach the issue of interpreting stylistic features by extracting linear and syntactic n-grams that are distinctive for a language variety. We present a study that exemplifies this process by a comparison of the German academic languages of linguistics and literary studies. Overall, our findings show that distinctive n-grams can be related to linguistic categories. The results suggest that the style of German literary studies is characterized by nominal structures and the style of linguistics by verbal ones.
Assessing the Stylistic Properties of Neurally Generated Text in Authorship Attribution
Authors: Enrique Manjavacas, Jeroen De Gussem, Walter Daelemans, Mike Kestemont
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Attribute, Authorship Attribution, Data Augmentation, Language Modeling, Language Modelling, Text Generation

Recent applications of neural language models have led to an increased interest in the automatic generation of natural language. However impressive, the evaluation of neurally generated text has so far remained rather informal and anecdotal. Here, we present an attempt at the systematic assessment of one aspect of the quality of neurally generated text. We focus on a specific aspect of neural language generation: its ability to reproduce authorial writing styles. Using established models for authorship attribution, we empirically assess the stylistic qualities of neurally generated text. In comparison to conventional language models, neural models generate fuzzier text, that is relatively harder to attribute correctly. Nevertheless, our results also suggest that neurally generated text offers more valuable perspectives for the augmentation of training data.
Question Difficulty -- How to Estimate Without Norming, How to Use for Automated Grading
Authors: Ulrike Pad{\'o}
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9

Question difficulty estimates guide test creation, but are too costly for small-scale testing. We empirically verify that Bloom{'}s Taxonomy, a standard tool for difficulty estimation during question creation, reliably predicts question difficulty observed after testing in a short-answer corpus. We also find that difficulty is mirrored in the amount of variation in student answers, which can be computed before grading. We show that question difficulty and its approximations are useful for \textit{automated grading}, allowing us to identify the optimal feature set for grading each question even in an unseen-question setting.
Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System
Authors: Lifeng Jin, Michael White, Evan Jaffe, Laura Zimmerman, Douglas Danforth
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: regression

For medical students, virtual patient dialogue systems can provide useful training opportunities without the cost of employing actors to portray standardized patients. This work utilizes word- and character-based convolutional neural networks (CNNs) for question identification in a virtual patient dialogue system, outperforming a strong word- and character-based logistic regression baseline. While the CNNs perform well given sufficient training data, the best system performance is ultimately achieved by combining CNNs with a hand-crafted pattern matching system that is robust to label sparsity, providing a 10{\%} boost in system accuracy and an error reduction of 47{\%} as compared to the pattern-matching system alone.
Continuous fluency tracking and the challenges of varying text complexity
Authors: Beata Beigman Klebanov, Anastassia Loukina, John Sabatini, Tenaha O{'}Reilly
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Reading Comprehension

This paper is a preliminary report on using text complexity measurement in the service of a new educational application. We describe a reading intervention where a child takes turns reading a book aloud with a virtual reading partner. Our ultimate goal is to provide meaningful feedback to the parent or the teacher by continuously tracking the child{'}s improvement in reading fluency. We show that this would not be a simple endeavor, due to an intricate relationship between text complexity from the point of view of comprehension and reading rate.
Linked Data for Language-Learning Applications
Authors: Robyn Loughnane, Kate McCurdy, Peter Kolb, Stefan Selent
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Part-Of-Speech Tagging

The use of linked data within language-learning applications is an open research question. A research prototype is presented that applies linked-data principles to store linguistic annotation generated from language-learning content using a variety of NLP tools. The result is a database that links learning content, linguistic annotation and open-source resources, on top of which a diverse range of tools for language-learning applications can be built.
A Report on the 2017 Native Language Identification Shared Task
Authors: Shervin Malmasi, Keelan Evanini, Aoife Cahill, Joel Tetreault, Robert Pugh, Christopher Hamill, Diane Napolitano, Yao Qian
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Grammatical Error Correction, Language Acquisition, Language Identification, Native Language Identification

Native Language Identification (NLI) is the task of automatically identifying the native language (L1) of an individual based on their language production in a learned language. It is typically framed as a classification task where the set of L1s is known a priori. Two previous shared tasks on NLI have been organized where the aim was to identify the L1 of learners of English based on essays (2013) and spoken responses (2016) they provided during a standardized assessment of academic English proficiency. The 2017 shared task combines the inputs from the two prior tasks for the first time. There are three tracks: NLI on the essay only, NLI on the spoken response only (based on a transcription of the response and i-vector acoustic features), and NLI using both responses. We believe this makes for a more interesting shared task while building on the methods and results from the previous two shared tasks. In this paper, we report the results of the shared task. A total of 19 teams competed across the three different sub-tasks. The fusion track showed that combining the written and spoken responses provides a large boost in prediction accuracy. Multiple classifier systems (e.g. ensembles and meta-classifiers) were the most effective in all tasks, with most based on traditional classifiers (e.g. SVMs) with lexical/syntactic features.
Evaluation of Automatically Generated Pronoun Reference Questions
Authors: Arief Yudha Satria, Takenobu Tokunaga
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Multiple-choice, Reading Comprehension

This study provides a detailed analysis of evaluation of English pronoun reference questions which are created automatically by machine. Pronoun reference questions are multiple choice questions that ask test takers to choose an antecedent of a target pronoun in a reading passage from four options. The evaluation was performed from two perspectives: the perspective of English teachers and that of English learners. Item analysis suggests that machine-generated questions achieve comparable quality with human-made questions. Correlation analysis revealed a strong correlation between the scores of machine-generated questions and that of human-made questions.
Predicting Audience's Laughter During Presentations Using Convolutional Neural Network
Authors: Lei Chen, Chong MIn Lee
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: General Classification, text-classification, Text Classification

Public speakings play important roles in schools and work places and properly using humor contributes to effective presentations. For the purpose of automatically evaluating speakers{'} humor usage, we build a presentation corpus containing humorous utterances based on TED talks. Compared to previous data resources supporting humor recognition research, ours has several advantages, including (a) both positive and negative instances coming from a homogeneous data set, (b) containing a large number of speakers, and (c) being open. Focusing on using lexical cues for humor recognition, we systematically compare a newly emerging text classification method based on Convolutional Neural Networks (CNNs) with a well-established conventional method using linguistic knowledge. The advantages of the CNN method are both getting higher detection accuracies and being able to learn essential features automatically.
Collecting fluency corrections for spoken learner English
Authors: Andrew Caines, Emma Flint, Paula Buttery
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Grammatical Error Correction, Grammatical Error Detection, Machine Translation

We present crowdsourced collection of error annotations for transcriptions of spoken learner English. Our emphasis in data collection is on fluency corrections, a more complete correction than has traditionally been aimed for in grammatical error correction research (GEC). Fluency corrections require improvements to the text, taking discourse and utterance level semantics into account: the result is a more naturalistic, holistic version of the original. We propose that this shifted emphasis be reflected in a new name for the task: {`}holistic error correction{'} (HEC). We analyse crowdworker behaviour in HEC and conclude that the method is useful with certain amendments for future work.
Exploring Relationships Between Writing \& Broader Outcomes With Automated Writing Evaluation
Authors: Jill Burstein, Dan McCaffrey, Beata Beigman Klebanov, Guangming Ling
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Automated Writing Evaluation

Writing is a challenge, especially for at-risk students who may lack the prerequisite writing skills required to persist in U.S. 4-year postsecondary (college) institutions. Educators teaching postsecondary courses requiring writing could benefit from a better understanding of writing achievement and its role in postsecondary success. In this paper, novel exploratory work examined how automated writing evaluation (AWE) can inform our understanding of the relationship between postsecondary writing skill and broader success outcomes. An exploratory study was conducted using test-taker essays from a standardized writing assessment of postsecondary student learning outcomes. Findings showed that for the essays, AWE features were found to be predictors of broader outcomes measures: college success and learning outcomes measures. Study findings illustrate AWE{'}s potential to support educational analytics {--} i.e., relationships between writing skill and broader outcomes {--} taking a step toward moving AWE beyond writing assessment and instructional use cases.
Combining Multiple Corpora for Readability Assessment for People with Cognitive Disabilities
Authors: Victoria Yaneva, Constantin Or{\u{a}}san, Richard Evans, Omid Rohanian
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Text Simplification

Given the lack of large user-evaluated corpora in disability-related NLP research (e.g. text simplification or readability assessment for people with cognitive disabilities), the question of choosing suitable training data for NLP models is not straightforward. The use of large generic corpora may be problematic because such data may not reflect the needs of the target population. The use of the available user-evaluated corpora may be problematic because these datasets are not large enough to be used as training data. In this paper we explore a third approach, in which a large generic corpus is combined with a smaller population-specific corpus to train a classifier which is evaluated using two sets of unseen user-evaluated data. One of these sets, the ASD Comprehension corpus, is developed for the purposes of this study and made freely available. We explore the effects of the size and type of the training data used on the performance of the classifiers, and the effects of the type of the unseen test datasets on the classification performance.