pwc-archive/papers-with-abstracts huggingface dataset:
Annotating Characters in Literary Corpora: A Scheme, the CHARLES Tool, and an Annotated Novel
Authors: Hardik Vala, Stefan Dimitrov, David Jurgens, Andrew Piper, Derek Ruths
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

Characters form the focus of various studies of literary works, including social network analysis, archetype induction, and plot comparison. The recent rise in the computational modelling of literary works has produced a proportional rise in the demand for character-annotated literary corpora. However, automatically identifying characters is an open problem and there is low availability of literary texts with manually labelled characters. To address the latter problem, this work presents three contributions: (1) a comprehensive scheme for manually resolving mentions to characters in texts. (2) A novel collaborative annotation tool, CHARLES (CHAracter Resolution Label-Entry System) for character annotation and similiar cross-document tagging tasks. (3) The character annotations resulting from a pilot study on the novel Pride and Prejudice, demonstrating the scheme and tool facilitate the efficient production of high-quality annotations. We expect this work to motivate the further production of annotated literary corpora to help meet the demand of the community.
Error Typology and Remediation Strategies for Requirements Written in English by Non-Native Speakers
Authors: Marie Garnier, Patrick Saint-Dizier
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Diversity, Specificity

In most international industries, English is the main language of communication for technical documents. These documents are designed to be as unambiguous as possible for their users. For international industries based in non-English speaking countries, the professionals in charge of writing requirements are often non-native speakers of English, who rarely receive adequate training in the use of English for this task. As a result, requirements can contain a relatively large diversity of lexical and grammatical errors, which are not eliminated by the use of guidelines from controlled languages. This article investigates the distribution of errors in a corpus of requirements written in English by native speakers of French. Errors are defined on the basis of grammaticality and acceptability principles, and classified using comparable categories. Results show a high proportion of errors in the Noun Phrase, notably through modifier stacking, and errors consistent with simplification strategies. Comparisons with similar corpora in other genres reveal the specificity of the distribution of errors in requirements. This research also introduces possible applied uses, in the form of strategies for the automatic detection of errors, and in-person training provided by certification boards in requirements authoring.
Improving POS Tagging of German Learner Language in a Reading Comprehension Scenario
Authors: Lena Keiper, Andrea Horbach, Stefan Thater
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: POS, POS Tagging, Reading Comprehension

We present a novel method to automatically improve the accurracy of part-of-speech taggers on learner language. The key idea underlying our approach is to exploit the structure of a typical language learner task and automatically induce POS information for out-of-vocabulary (OOV) words. To evaluate the effectiveness of our approach, we add manual POS and normalization information to an existing language learner corpus. Our evaluation shows an increase in accurracy from 72.4{\%} to 81.5{\%} on OOV words.
SVALex: a CEFR-graded Lexical Resource for Swedish Foreign and Second Language Learners
Authors: Thomas Fran{\c{c}}ois, Elena Volodina, Ildik{\'o} Pil{\'a}n, Ana{\"\i}s Tack
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

The paper introduces SVALex, a lexical resource primarily aimed at learners and teachers of Swedish as a foreign and second language that describes the distribution of 15,681 words and expressions across the Common European Framework of Reference (CEFR). The resource is based on a corpus of coursebook texts, and thus describes receptive vocabulary learners are exposed to during reading activities, as opposed to productive vocabulary they use when speaking or writing. The paper describes the methodology applied to create the list and to estimate the frequency distribution. It also discusses some characteristics of the resulting resource and compares it to other lexical resources for Swedish. An interesting feature of this resource is the possibility to separate the wheat from the chaff, identifying the core vocabulary at each level, i.e. vocabulary shared by several coursebook writers at each level, from peripheral vocabulary which is used by the minority of the coursebook writers.
Detecting Word Usage Errors in Chinese Sentences for Learning Chinese as a Foreign Language
Authors: Yow-Ting Shiue, Hsin-Hsi Chen
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Grammatical Error Detection, Sentence

Automated grammatical error detection, which helps users improve their writing, is an important application in NLP. Recently more and more people are learning Chinese, and an automated error detection system can be helpful for the learners. This paper proposes n-gram features, dependency count features, dependency bigram features, and single-character features to determine if a Chinese sentence contains word usage errors, in which a word is written as a wrong form or the word selection is inappropriate. With marking potential errors on the level of sentence segments, typically delimited by punctuation marks, the learner can try to correct the problems without the assistant of a language teacher. Experiments on the HSK corpus show that the classifier combining all sets of features achieves an accuracy of 0.8423. By utilizing certain combination of the sets of features, we can construct a system that favors precision or recall. The best precision we achieve is 0.9536, indicating that our system is reliable and seldom produces misleading results.
LibN3L:A Lightweight Package for Neural NLP
Authors: Meishan Zhang, Jie Yang, Zhiyang Teng, Yue Zhang
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: BIG-bench Machine Learning

We present a light-weight machine learning tool for NLP research. The package supports operations on both discrete and dense vectors, facilitating implementation of linear models as well as neural models. It provides several basic layers which mainly aims for single-layer linear and non-linear transformations. By using these layers, we can conveniently implement linear models and simple neural models. Besides, this package also integrates several complex layers by composing those basic layers, such as RNN, Attention Pooling, LSTM and gated RNN. Those complex layers can be used to implement deep neural models directly.
Evaluating Lexical Simplification and Vocabulary Knowledge for Learners of French: Possibilities of Using the FLELex Resource
Authors: Ana{\"\i}s Tack, Thomas Fran{\c{c}}ois, Anne-Laure Ligozat, C{\'e}drick Fairon
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Lexical Simplification

This study examines two possibilities of using the FLELex graded lexicon for the automated assessment of text complexity in French as a foreign language learning. From the lexical frequency distributions described in FLELex, we derive a single level of difficulty for each word in a parallel corpus of original and simplified texts. We then use this data to automatically address the lexical complexity of texts in two ways. On the one hand, we evaluate the degree of lexical simplification in manually simplified texts with respect to their original version. Our results show a significant simplification effect, both in the case of French narratives simplified for non-native readers and in the case of simplified Wikipedia texts. On the other hand, we define a predictive model which identifies the number of words in a text that are expected to be known at a particular learning level. We assess the accuracy with which these predictions are able to capture actual word knowledge as reported by Dutch-speaking learners of French. Our study shows that although the predictions seem relatively accurate in general (87.4{\%} to 92.3{\%}), they do not yet seem to cover the learners{'} lack of knowledge very well.
A Shared Task for Spoken CALL?
Authors: Claudia Baur, Johanna Gerlach, Manny Rayner, Martin Russell, Helmer Strik
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

We argue that the field of spoken CALL needs a shared task in order to facilitate comparisons between different groups and methodologies, and describe a concrete example of such a task, based on data collected from a speech-enabled online tool which has been used to help young Swiss German teens practise skills in English conversation. Items are prompt-response pairs, where the prompt is a piece of German text and the response is a recorded English audio file. The task is to label pairs as {``}accept{''} or {``}reject{''}, accepting responses which are grammatically and linguistically correct to match a set of hidden gold standard answers as closely as possible. Initial resources are provided so that a scratch system can be constructed with a minimal investment of effort, and in particular without necessarily using a speech recogniser. Training data for the task will be released in June 2016, and test data in January 2017.
Joining-in-type Humanoid Robot Assisted Language Learning System
Authors: AlBara Khalifa, Tsuneo Kato, Seiichi Yamamoto
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Vocal Bursts Type Prediction

Dialogue robots are attractive to people, and in language learning systems, they motivate learners and let them practice conversational skills in more realistic environment. However, automatic speech recognition (ASR) of the second language (L2) learners is still a challenge, because their speech contains not just pronouncing, lexical, grammatical errors, but is sometimes totally disordered. Hence, we propose a novel robot assisted language learning (RALL) system using two robots, one as a teacher and the other as an advanced learner. The system is designed to simulate multiparty conversation, expecting implicit learning and enhancement of predictability of learners{'} utterance through an alignment similar to {``}interactive alignment{''}, which is observed in human-human conversation. We collected a database with the prototypes, and measured how much the alignment phenomenon observed in the database with initial analysis.
OSMAN â€• A Novel Arabic Readability Metric
Authors: Mahmoud El-Haj, Paul Rayson
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

We present OSMAN (Open Source Metric for Measuring Arabic Narratives) - a novel open source Arabic readability metric and tool. It allows researchers to calculate readability for Arabic text with and without diacritics. OSMAN is a modified version of the conventional readability formulas such as Flesch and Fog. In our work we introduce a novel approach towards counting short, long and stress syllables in Arabic which is essential for judging readability of Arabic narratives. We also introduce an additional factor called {``}Faseeh{''} which considers aspects of script usually dropped in informal Arabic writing. To evaluate our methods we used Spearman{'}s correlation metric to compare text readability for 73,000 parallel sentences from English and Arabic UN documents. The Arabic sentences were written with the absence of diacritics and in order to count the number of syllables we added the diacritics in using an open source tool called Mishkal. The results show that OSMAN readability formula correlates well with the English ones making it a useful tool for researchers and educators working with Arabic text.
Evaluating Interactive System Adaptation
Authors: Edouard Geoffrois
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

Enabling users of intelligent systems to enhance the system performance by providing feedback on their errors is an important need. However, the ability of systems to learn from user feedback is difficult to evaluate in an objective and comparative way. Indeed, the involvement of real users in the adaptation process is an impediment to objective evaluation. This issue can be solved by using an oracle approach, where users are simulated by oracles having access to the reference test data. Another difficulty is to find a meaningful metric despite the fact that system improvements depend on the feedback provided and on the system itself. A solution is to measure the minimal amount of information needed to correct all system errors. It can be shown that for any well defined non interactive task, the interactively supervised version of the task can be evaluated by combining such an oracle-based approach and a minimum supervision rate metric. This new evaluation protocol for adaptive systems is not only expected to drive progress for such systems, but also to pave the way for a specialisation of actors along the value chain of their technological development.
Complementarity, F-score, and NLP Evaluation
Authors: Leon Derczynski
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Entity Extraction using GAN, Information Retrieval, Retrieval

This paper addresses the problem of quantifying the differences between entity extraction systems, where in general only a small proportion a document should be selected. Comparing overall accuracy is not very useful in these cases, as small differences in accuracy may correspond to huge differences in selections over the target minority class. Conventionally, one may use per-token complementarity to describe these differences, but it is not very useful when the set is heavily skewed. In such situations, which are common in information retrieval and entity recognition, metrics like precision and recall are typically used to describe performance. However, precision and recall fail to describe the differences between sets of objects selected by different decision strategies, instead just describing the proportional amount of correct and incorrect objects selected. This paper presents a method for measuring complementarity for precision, recall and F-score, quantifying the difference between entity extraction approaches.
DRANZIERA: An Evaluation Protocol For Multi-Domain Opinion Mining
Authors: Mauro Dragoni, Andrea Tettamanzi, C{\'e}lia da Costa Pereira
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Opinion Mining

Opinion Mining is a topic which attracted a lot of interest in the last years. By observing the literature, it is often hard to replicate system evaluation due to the unavailability of the data used for the evaluation or to the lack of details about the protocol used in the campaign. In this paper, we propose an evaluation protocol, called DRANZIERA, composed of a multi-domain dataset and guidelines allowing both to evaluate opinion mining systems in different contexts (Closed, Semi-Open, and Open) and to compare them to each other and to a number of baselines.
Evaluating a Topic Modelling Approach to Measuring Corpus Similarity
Authors: Richard Fothergill, Paul Cook, Timothy Baldwin
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

Web corpora are often constructed automatically, and their contents are therefore often not well understood. One technique for assessing the composition of such a web corpus is to empirically measure its similarity to a reference corpus whose composition is known. In this paper we evaluate a number of measures of corpus similarity, including a method based on topic modelling which has not been previously evaluated for this task. To evaluate these methods we use known-similarity corpora that have been previously used for this purpose, as well as a number of newly-constructed known-similarity corpora targeting differences in genre, topic, time, and region. Our findings indicate that, overall, the topic modelling approach did not improve on a chi-square method that had previously been found to work well for measuring corpus similarity.
Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact
Authors: {\^A}ngela Costa, Rui Correia, Lu{\'\i}sa Coheur
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Machine Translation, Sentence, Translation

In this paper we describe a corpus of automatic translations annotated with both error type and quality. The 300 sentences that we have selected were generated by Google Translate, Systran and two in-house Machine Translation systems that use Moses technology. The errors present on the translations were annotated with an error taxonomy that divides errors in five main linguistic categories (Orthography, Lexis, Grammar, Semantics and Discourse), reflecting the language level where the error is located. After the error annotation process, we accessed the translation quality of each sentence using a four point comprehension scale from 1 to 5. Both tasks of error and quality annotation were performed by two different annotators, achieving good levels of inter-annotator agreement. The creation of this corpus allowed us to use it as training data for a translation quality classifier. We concluded on error severity by observing the outputs of two machine learning classifiers: a decision tree and a regression model.
Evaluating the Readability of Text Simplification Output for Readers with Cognitive Disabilities
Authors: Victoria Yaneva, Irina Temnikova, Ruslan Mitkov
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Reading Comprehension, Text Simplification

This paper presents an approach for automatic evaluation of the readability of text simplification output for readers with cognitive disabilities. First, we present our work towards the development of the EasyRead corpus, which contains easy-to-read documents created especially for people with cognitive disabilities. We then compare the EasyRead corpus to the simplified output contained in the LocalNews corpus (Feng, 2009), the accessibility of which has been evaluated through reading comprehension experiments including 20 adults with mild intellectual disability. This comparison is made on the basis of 13 disability-specific linguistic features. The comparison reveals that there are no major differences between the two corpora, which shows that the EasyRead corpus is to a similar reading level as the user-evaluated texts. We also discuss the role of Simple Wikipedia (Zhu et al., 2010) as a widely-used accessibility benchmark, in light of our finding that it is significantly more complex than both the EasyRead and the LocalNews corpora.
Word Embedding Evaluation and Combination
Authors: Sahar Ghannay, Benoit Favre, Yannick Est{\`e}ve, Nathalie Camelin
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Word Embeddings

Word embeddings have been successfully used in several natural language processing tasks (NLP) and speech processing. Different approaches have been introduced to calculate word embeddings through neural networks. In the literature, many studies focused on word embedding evaluation, but for our knowledge, there are still some gaps. This paper presents a study focusing on a rigorous comparison of the performances of different kinds of word embeddings. These performances are evaluated on different NLP and linguistic tasks, while all the word embeddings are estimated on the same training data using the same vocabulary, the same number of dimensions, and other similar characteristics. The evaluation results reported in this paper match those in the literature, since they point out that the improvements achieved by a word embedding in one task are not consistently observed across all tasks. For that reason, this paper investigates and evaluates approaches to combine word embeddings in order to take advantage of their complementarity, and to look for the effective word embeddings that can achieve good performances on all tasks. As a conclusion, this paper provides new perceptions of intrinsic qualities of the famous word embedding families, which can be different from the ones provided by works previously published in the scientific literature.
Benchmarking multimedia technologies with the CAMOMILE platform: the case of Multimodal Person Discovery at MediaEval 2015
Authors: Johann Poignant, Herv{\'e} Bredin, Claude Barras, Mickael Stefas, Pierrick Bruneau, Thomas Tamisier
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Benchmarking

In this paper, we claim that the CAMOMILE collaborative annotation platform (developed in the framework of the eponymous CHIST-ERA project) eases the organization of multimedia technology benchmarks, automating most of the campaign technical workflow and enabling collaborative (hence faster and cheaper) annotation of the evaluation data. This is demonstrated through the successful organization of a new multimedia task at MediaEval 2015, Multimodal Person Discovery in Broadcast TV.
Evaluating the Impact of Light Post-Editing on Usability
Authors: Sheila Castilho, Sharon O{'}Brien
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Articles

This paper discusses a methodology to measure the usability of machine translated content by end users, comparing lightly post-edited content with raw output and with the usability of source language content. The content selected consists of Online Help articles from a software company for a spreadsheet application, translated from English into German. Three groups of five users each used either the source text - the English version (EN) -, the raw MT version (DE{\_}MT), or the light PE version (DE{\_}PE), and were asked to carry out six tasks. Usability was measured using an eye tracker and cognitive, temporal and pragmatic measures of usability. Satisfaction was measured via a post-task questionnaire presented after the participants had completed the tasks.
Operational Assessment of Keyword Search on Oral History
Authors: Elizabeth Salesky, Jessica Ray, Wade Shen
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Information Retrieval, Retrieval, speech-recognition, Speech Recognition

This project assesses the resources necessary to make oral history searchable by means of automatic speech recognition (ASR). There are many inherent challenges in applying ASR to conversational speech: smaller training set sizes and varying demographics, among others. We assess the impact of dataset size, word error rate and term-weighted value on human search capability through an information retrieval task on Mechanical Turk. We use English oral history data collected by StoryCorps, a national organization that provides all people with the opportunity to record, share and preserve their stories, and control for a variety of demographics including age, gender, birthplace, and dialect on four different training set sizes. We show comparable search performance using a standard speech recognition system as with hand-transcribed data, which is promising for increased accessibility of conversational speech and oral history archives.
Odin's Runes: A Rule Language for Information Extraction
Authors: Marco A. Valenzuela-Esc{\'a}rcega, Gus Hahn-Powell, Mihai Surdeanu
Date: 2016-05-01 00:00:00
Proceeding: LREC 2016 5

Odin is an information extraction framework that applies cascades of finite state automata over both surface text and syntactic dependency graphs. Support for syntactic patterns allow us to concisely define relations that are otherwise difficult to express in languages such as Common Pattern Specification Language (CPSL), which are currently limited to shallow linguistic features. The interaction of lexical and syntactic automata provides robustness and flexibility when writing extraction rules. This paper describes Odin{'}s declarative language for writing these cascaded automata.