pwc-archive/papers-with-abstracts huggingface dataset:
Tutorial: Making Better Use of the Crowd
Authors: Jennifer Wortman Vaughan
Date: 2017-07-01 00:00:00
Proceeding: ACL 2017 7

Over the last decade, crowdsourcing has been used to harness the power of human computation to solve tasks that are notoriously difficult to solve with computers alone, such as determining whether or not an image contains a tree, rating the relevance of a website, or verifying the phone number of a business. The natural language processing community was early to embrace crowdsourcing as a tool for quickly and inexpensively obtaining annotated data to train NLP systems. Once this data is collected, it can be handed off to algorithms that learn to perform basic NLP tasks such as translation or parsing. Usually this handoff is where interaction with the crowd ends. The crowd provides the data, but the ultimate goal is to eventually take humans out of the loop. Are there better ways to make use of the crowd?In this tutorial, I will begin with a showcase of innovative uses of crowdsourcing that go beyond data collection and annotation. I will discuss applications to natural language processing and machine learning, hybrid intelligence or {``}human in the loop{''} AI systems that leverage the complementary strengths of humans and machines in order to achieve more than either could achieve alone, and large scale studies of human behavior online. I will then spend the majority of the tutorial diving into recent research aimed at understanding who crowdworkers are, how they behave, and what this should teach us about best practices for interacting with the crowd.
Classifying Illegal Activities on Tor Network Based on Web Textual Contents
Authors: Mhd Wesam Al Nabki, Eduardo Fidalgo, Enrique Alegre, Ivan de Paz
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4

The freedom of the Deep Web offers a safe place where people can express themselves anonymously but they also can conduct illegal activities. In this paper, we present and make publicly available a new dataset for Darknet active domains, which we call {''}Darknet Usage Text Addresses{''} (DUTA). We built DUTA by sampling the Tor network during two months and manually labeled each address into 26 classes. Using DUTA, we conducted a comparison between two well-known text representation techniques crossed by three different supervised classifiers to categorize the Tor hidden services. We also fixed the pipeline elements and identified the aspects that have a critical influence on the classification results. We found that the combination of TFIDF words representation with Logistic Regression classifier achieves 96.6{\%} of 10 folds cross-validation accuracy and a macro F1 score of 93.7{\%} when classifying a subset of illegal activities from DUTA. The good performance of the classifier might support potential tools to help the authorities in the detection of these activities.
Learning Compositionality Functions on Word Embeddings for Modelling Attribute Meaning in Adjective-Noun Phrases
Authors: Matthias Hartung, Fabian Kaupmann, Soufian Jebbara, Philipp Cimiano
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Attribute, Relation, Word Embeddings

Word embeddings have been shown to be highly effective in a variety of lexical semantic tasks. They tend to capture meaningful relational similarities between individual words, at the expense of lacking the capabilty of making the underlying semantic relation explicit. In this paper, we investigate the attribute relation that often holds between the constituents of adjective-noun phrases. We use CBOW word embeddings to represent word meaning and learn a compositionality function that combines the individual constituents into a phrase representation, thus capturing the compositional attribute meaning. The resulting embedding model, while being fully interpretable, outperforms count-based distributional vector space models that are tailored to attribute meaning in the two tasks of attribute selection and phrase similarity prediction. Moreover, as the model captures a generalized layer of attribute meaning, it bears the potential to be used for predictions over various attribute inventories without re-training.
Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction and Disambiguation
Authors: Alex Panchenko, er, Eugen Ruppert, Stefano Faralli, Simone Paolo Ponzetto, Chris Biemann
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Word Embeddings, Word Sense Disambiguation, Word Sense Induction

The current trend in NLP is the use of highly opaque models, e.g. neural networks and word embeddings. While these models yield state-of-the-art results on a range of tasks, their drawback is poor interpretability. On the example of word sense induction and disambiguation (WSID), we show that it is possible to develop an interpretable model that matches the state-of-the-art models in accuracy. Namely, we present an unsupervised, knowledge-free WSID approach, which is interpretable at three levels: word sense inventory, sense feature representations, and disambiguation procedure. Experiments show that our model performs on par with state-of-the-art word sense embeddings and other unsupervised systems while offering the possibility to justify its decisions in human-readable form.
Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison
Authors: Aless Raganato, ro, Jose Camacho-Collados, Roberto Navigli
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Word Sense Disambiguation

Word Sense Disambiguation is a long-standing task in Natural Language Processing, lying at the core of human language understanding. However, the evaluation of automatic systems has been problematic, mainly due to the lack of a reliable evaluation framework. In this paper we develop a unified evaluation framework and analyze the performance of various Word Sense Disambiguation systems in a fair setup. The results show that supervised systems clearly outperform knowledge-based models. Among the supervised systems, a linear classifier trained on conventional local features still proves to be a hard baseline to beat. Nonetheless, recent approaches exploiting neural networks on unlabeled corpora achieve promising results, surpassing this hard baseline in most test sets.
Which is the Effective Way for Gaokao: Information Retrieval or Neural Networks?
Authors: Shangmin Guo, Xiangrong Zeng, Shizhu He, Kang Liu, Jun Zhao
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Information Retrieval, Multiple-choice, Question Answering, Reading Comprehension, Retrieval, Sentence

As one of the most important test of China, Gaokao is designed to be difficult enough to distinguish the excellent high school students. In this work, we detailed the Gaokao History Multiple Choice Questions(GKHMC) and proposed two different approaches to address them using various resources. One approach is based on entity search technique (IR approach), the other is based on text entailment approach where we specifically employ deep neural networks(NN approach). The result of experiment on our collected real Gaokao questions showed that they are good at different categories of questions, that is IR approach performs much better at entity questions(EQs) while NN approach shows its advantage on sentence questions(SQs). We achieve state-of-the-art performance and show that it{'}s indispensable to apply hybrid method when participating in the real-world tests.
If You Can't Beat Them Join Them: Handcrafted Features Complement Neural Nets for Non-Factoid Answer Reranking
Authors: Dasha Bogdanova, Jennifer Foster, Daria Dzendzik, Qun Liu
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Answer Selection, Community Question Answering, Feature Engineering, Learning-To-Rank, Question Answering, Reranking

We show that a neural approach to the task of non-factoid answer reranking can benefit from the inclusion of tried-and-tested handcrafted features. We present a neural network architecture based on a combination of recurrent neural networks that are used to encode questions and answers, and a multilayer perceptron. We show how this approach can be combined with additional features, in particular, the discourse features used by previous research. Our neural approach achieves state-of-the-art performance on a public dataset from Yahoo! Answers and its performance is further improved by incorporating the discourse features. Additionally, we present a new dataset of Ask Ubuntu questions where the hybrid approach also achieves good results.
Recognizing Mentions of Adverse Drug Reaction in Social Media Using Knowledge-Infused Recurrent Models
Authors: Gabriel Stanovsky, Daniel Gruhl, Pablo Mendes
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Active Learning, Knowledge Graph Embeddings

Recognizing mentions of Adverse Drug Reactions (ADR) in social media is challenging: ADR mentions are context-dependent and include long, varied and unconventional descriptions as compared to more formal medical symptom terminology. We use the CADEC corpus to train a recurrent neural network (RNN) transducer, integrated with knowledge graph embeddings of DBpedia, and show the resulting model to be highly accurate (93.4 F1). Furthermore, even when lacking high quality expert annotations, we show that by employing an active learning technique and using purpose built annotation tools, we can train the RNN to perform well (83.9 F1).
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation
Authors: Ivan Vuli{\'c}, Douwe Kiela, Anna Korhonen
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Information Retrieval, Representation Learning, Retrieval, Semantic Textual Similarity

Recent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. In this work, we propose a novel evaluation framework that enables large-scale evaluation of such architectures in the free word association (WA) task, which is firmly grounded in cognitive theories of human semantic representation. This evaluation is facilitated by the existence of large manually constructed repositories of word association data. In this paper, we (1) present a detailed analysis of the new quantitative WA evaluation protocol, (2) suggest new evaluation metrics for the WA task inspired by its direct analogy with information retrieval problems, (3) evaluate various state-of-the-art representation models on this task, and (4) discuss the relationship between WA and prior evaluations of semantic representation with well-known similarity and relatedness evaluation sets. We have made the WA evaluation toolkit publicly available.
Computational Argumentation Quality Assessment in Natural Language
Authors: Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, Benno Stein
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4

Research on computational argumentation faces the problem of how to automatically assess the quality of an argument or argumentation. While different quality dimensions have been approached in natural language processing, a common understanding of argumentation quality is still missing. This paper presents the first holistic work on computational argumentation quality in natural language. We comprehensively survey the diverse existing theories and approaches to assess logical, rhetorical, and dialectical quality dimensions, and we derive a systematic taxonomy from these. In addition, we provide a corpus with 320 arguments, annotated for all 15 dimensions in the taxonomy. Our results establish a common ground for research on computational argumentation quality assessment.
A method for in-depth comparative evaluation: How (dis)similar are outputs of pos taggers, dependency parsers and coreference resolvers really?
Authors: Don Tuggener
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: coreference-resolution, Coreference Resolution, Dependency Parsing, POS, POS Tagging

This paper proposes a generic method for the comparative evaluation of system outputs. The approach is able to quantify the pairwise differences between two outputs and to unravel in detail what the differences consist of. We apply our approach to three tasks in Computational Linguistics, i.e. POS tagging, dependency parsing, and coreference resolution. We find that system outputs are more distinct than the (often) small differences in evaluation scores seem to suggest.
Integrating Meaning into Quality Evaluation of Machine Translation
Authors: Osman Ba{\c{s}}kaya, Eray Yildiz, Doruk Tunao{\u{g}}lu, Mustafa Tolga Eren, A. Seza Do{\u{g}}ru{\"o}z
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Form, Machine Translation, Translation

Machine translation (MT) quality is evaluated through comparisons between MT outputs and the human translations (HT). Traditionally, this evaluation relies on form related features (e.g. lexicon and syntax) and ignores the transfer of meaning reflected in HT outputs. Instead, we evaluate the quality of MT outputs through meaning related features (e.g. polarity, subjectivity) with two experiments. In the first experiment, the meaning related features are compared to human rankings individually. In the second experiment, combinations of meaning related features and other quality metrics are utilized to predict the same human rankings. The results of our experiments confirm the benefit of these features in predicting human evaluation of translation quality in addition to traditional metrics which focus mainly on form.
Cross-Lingual Dependency Parsing with Late Decoding for Truly Low-Resource Languages
Authors: Michael Schlichtkrull, Anders S{\o}gaard
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Dependency Parsing

In cross-lingual dependency annotation projection, information is often lost during transfer because of early decoding. We present an end-to-end graph-based neural network dependency parser that can be trained to reproduce matrices of edge scores, which can be directly projected across word alignments. We show that our approach to cross-lingual dependency parsing is not only simpler, but also achieves an absolute improvement of 2.25{\%} averaged across 10 languages compared to the previous state of the art.
Delexicalized Word Embeddings for Cross-lingual Dependency Parsing
Authors: Mathieu Dehouck, Pascal Denis
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Cross-Lingual Transfer, Dependency Parsing, Word Embeddings

This paper presents a new approach to the problem of cross-lingual dependency parsing, aiming at leveraging training data from different source languages to learn a parser in a target language. Specifically, this approach first constructs word vector representations that exploit structural (i.e., dependency-based) contexts but only considering the morpho-syntactic information associated with each word and its contexts. These delexicalized word embeddings, which can be trained on any set of languages and capture features shared across languages, are then used in combination with standard language-specific features to train a lexicalized parser in the target language. We evaluate our approach through experiments on a set of eight different languages that are part the Universal Dependencies Project. Our main results show that using such delexicalized embeddings, either trained in a monolingual or multilingual fashion, achieves significant improvements over monolingual baselines.
Stance Classification of Context-Dependent Claims
Authors: Roy Bar-Haim, Indrajit Bhattacharya, Francesco Dinuzzo, Amrita Saha, Noam Slonim
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Classification, General Classification, Sentiment Analysis, Sentiment Classification, Stance Classification

Recent work has addressed the problem of detecting relevant claims for a given controversial topic. We introduce the complementary task of Claim Stance Classification, along with the first benchmark dataset for this task. We decompose this problem into: (a) open-domain target identification for topic and claim (b) sentiment classification for each target, and (c) open-domain contrast detection between the topic and the claim targets. Manual annotation of the dataset confirms the applicability and validity of our model. We describe an implementation of our model, focusing on a novel algorithm for contrast detection. Our approach achieves promising results, and is shown to outperform several baselines, which represent the common practice of applying a single, monolithic classifier for stance classification.
Exploring the Impact of Pragmatic Phenomena on Irony Detection in Tweets: A Multilingual Corpus Study
Authors: Jihen Karoui, Farah Benamara, V{\'e}ronique Moriceau, Viviana Patti, Cristina Bosco, Nathalie Aussenac-Gilles
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Sentiment Analysis

This paper provides a linguistic and pragmatic analysis of the phenomenon of irony in order to represent how Twitter{'}s users exploit irony devices within their communication strategies for generating textual contents. We aim to measure the impact of a wide-range of pragmatic phenomena in the interpretation of irony, and to investigate how these phenomena interact with contexts local to the tweet. Informed by linguistic theories, we propose for the first time a multi-layered annotation schema for irony and its application to a corpus of French, English and Italian tweets. We detail each layer, explore their interactions, and discuss our results according to a qualitative and quantitative perspective.
A Multi-View Sentiment Corpus
Authors: Debora Nozza, Elisabetta Fersini, Enza Messina
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Emotion Recognition, General Classification, Opinion Mining, Sentiment Analysis

Sentiment Analysis is a broad task that involves the analysis of various aspect of the natural language text. However, most of the approaches in the state of the art usually investigate independently each aspect, i.e. Subjectivity Classification, Sentiment Polarity Classification, Emotion Recognition, Irony Detection. In this paper we present a Multi-View Sentiment Corpus (MVSC), which comprises 3000 English microblog posts related the movie domain. Three independent annotators manually labelled MVSC, following a broad annotation schema about different aspects that can be grasped from natural language text coming from social networks. The contribution is therefore a corpus that comprises five different views for each message, i.e. subjective/objective, sentiment polarity, implicit/explicit, irony, emotion. In order to allow a more detailed investigation on the human labelling behaviour, we provide the annotations of each human annotator involved.
A Systematic Study of Neural Discourse Models for Implicit Discourse Relation
Authors: Attapol Rutherford, Vera Demberg, Nianwen Xue
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Discourse Parsing, Relation

Inferring implicit discourse relations in natural language text is the most difficult subtask in discourse parsing. Many neural network models have been proposed to tackle this problem. However, the comparison for this task is not unified, so we could hardly draw clear conclusions about the effectiveness of various architectures. Here, we propose neural network models that are based on feedforward and long-short term memory architecture and systematically study the effects of varying structures. To our surprise, the best-configured feedforward architecture outperforms LSTM-based model in most cases despite thorough tuning. Further, we compare our best feedforward system with competitive convolutional and recurrent networks and find that feedforward can actually be more effective. For the first time for this task, we compile and publish outputs from previous neural and non-neural systems to establish the standard for further comparison.
Joint, Incremental Disfluency Detection and Utterance Segmentation from Speech
Authors: Julian Hough, David Schlangen
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Speech Recognition

We present the joint task of incremental disfluency detection and utterance segmentation and a simple deep learning system which performs it on transcripts and ASR results. We show how the constraints of the two tasks interact. Our joint-task system outperforms the equivalent individual task systems, provides competitive results and is suitable for future use in conversation agents in the psychiatric domain.
From Segmentation to Analyses: a Probabilistic Model for Unsupervised Morphology Induction
Authors: Toms Bergmanis, Sharon Goldwater
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Morphological Analysis, Segmentation

A major motivation for unsupervised morphological analysis is to reduce the sparse data problem in under-resourced languages. Most previous work focus on segmenting surface forms into their constituent morphs (taking: tak +ing), but surface form segmentation does not solve the sparse data problem as the analyses of take and taking are not connected to each other. We present a system that adapts the MorphoChains system (Narasimhan et al., 2015) to provide morphological analyses that aim to abstract over spelling differences in functionally similar morphs. This results in analyses that are not compelled to use all the orthographic material of a word (stopping: stop +ing) or limited to only that material (acidified: acid +ify +ed). On average across six typologically varied languages our system has a similar or better F-score on EMMA (a measure of underlying morpheme accuracy) than three strong baselines; moreover, the total number of distinct morphemes identified by our system is on average 12.8{\%} lower than for Morfessor (Virpioja et al., 2013), a state-of-the-art surface segmentation system.
Creating POS Tagging and Dependency Parsing Experts via Topic Modeling
Authors: Atreyee Mukherjee, S K{\"u}bler, ra, Matthias Scheutz
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Dependency Parsing, Domain Adaptation, POS, POS Tagging, Topic Models

Part of speech (POS) taggers and dependency parsers tend to work well on homogeneous datasets but their performance suffers on datasets containing data from different genres. In our current work, we investigate how to create POS tagging and dependency parsing experts for heterogeneous data by employing topic modeling. We create topic models (using Latent Dirichlet Allocation) to determine genres from a heterogeneous dataset and then train an expert for each of the genres. Our results show that the topic modeling experts reach substantial improvements when compared to the general versions. For dependency parsing, the improvement reaches 2 percent points over the full training baseline when we use two topics.
Universal Dependencies and Morphology for Hungarian - and on the Price of Universality
Authors: Veronika Vincze, Katalin Simk{\'o}, Zsolt Sz{\'a}nt{\'o}, Rich{\'a}rd Farkas
Date: 2017-04-01 00:00:00
Proceeding: EACL 2017 4
Tasks: Morphological Tagging

In this paper, we present how the principles of universal dependencies and morphology have been adapted to Hungarian. We report the most challenging grammatical phenomena and our solutions to those. On the basis of the adapted guidelines, we have converted and manually corrected 1,800 sentences from the Szeged Treebank to universal dependency format. We also introduce experiments on this manually annotated corpus for evaluating automatic conversion and the added value of language-specific, i.e. non-universal, annotations. Our results reveal that converting to universal dependencies is not necessarily trivial, moreover, using language-specific morphological features may have an impact on overall performance.