pwc-archive/papers-with-abstracts huggingface dataset:
Finding Good Conversations Online: The Yahoo News Annotated Comments Corpus
Authors: Courtney Napoles, Joel Tetreault, Aasish Pappu, Enrica Rosato, Brian Provenzale
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4

This work presents a dataset and annotation scheme for the new task of identifying {``}good{''} conversations that occur online, which we call ERICs: Engaging, Respectful, and/or Informative Conversations. We develop a taxonomy to reflect features of entire threads and individual comments which we believe contribute to identifying ERICs; code a novel dataset of Yahoo News comment threads (2.4k threads and 10k comments) and 1k threads from the Internet Argument Corpus; and analyze the features characteristic of ERICs. This is one of the largest annotated corpora of online human dialogues, with the most detailed set of annotations. It will be valuable for identifying ERICs and other aspects of argumentation, dialogue, and discourse.
Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task
Authors: Merel Scholman, Vera Demberg
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Relation

Traditional discourse annotation tasks are considered costly and time-consuming, and the reliability and validity of these tasks is in question. In this paper, we investigate whether crowdsourcing can be used to obtain reliable discourse relation annotations. We also examine the influence of context on the reliability of the data. The results of a crowdsourced connective insertion task showed that the method can be used to obtain reliable annotations: The majority of the inserted connectives converged with the original label. Further, the method is sensitive to the fact that multiple senses can often be inferred for a single relation. Regarding the presence of context, the results show no significant difference in distributions of insertions between conditions overall. However, a by-item comparison revealed several characteristics of segments that determine whether the presence of context makes a difference in annotations. The findings discussed in this paper can be taken as evidence that crowdsourcing can be used as a valuable method to obtain insights into the sense(s) of relations.
A Code-Switching Corpus of Turkish-German Conversations
Authors: {\"O}zlem {\c{C}}etino{\u{g}}lu
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Automatic Speech Recognition (ASR), Language Identification, Language Modelling, Part-Of-Speech Tagging, Sentence, Sentiment Analysis, Speech Recognition

We present a code-switching corpus of Turkish-German that is collected by recording conversations of bilinguals. The recordings are then transcribed in two layers following speech and orthography conventions, and annotated with sentence boundaries and intersentential, intrasentential, and intra-word switch points. The total amount of data is 5 hours of speech which corresponds to 3614 sentences. The corpus aims at serving as a resource for speech or text analysis, as well as a collection for linguistic inquiries.
Annotating omission in statement pairs
Authors: H{\'e}ctor Mart{\'\i}nez Alonso, Amaury Delamaire, Beno{\^\i}t Sagot
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Natural Language Inference

We focus on the identification of omission in statement pairs. We compare three annotation schemes, namely two different crowdsourcing schemes and manual expert annotation. We show that the simplest of the two crowdsourcing approaches yields a better annotation quality than the more complex one. We use a dedicated classifier to assess whether the annotators{'} behavior can be explained by straightforward linguistic features. The classifier benefits from a modeling that uses lexical information beyond length and overlap measures. However, for our task, we argue that expert and not crowdsourcing-based annotation is the best compromise between annotation cost and quality.
Annotating Speech, Attitude and Perception Reports
Authors: Corien Bary, Leopold Hess, Kees Thijs, Peter Berck, Iris Hendrickx
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: RAG

We present REPORTS, an annotation scheme for the annotation of speech, attitude and perception reports. Such a scheme makes it possible to annotate the various text elements involved in such reports (e.g. embedding entity, complement, complement head) and their relations in a uniform way, which in turn facilitates the automatic extraction of information on, for example, complementation and vocabulary distribution. We also present the Ancient Greek corpus RAG (Thucydides{'} History of the Peloponnesian War), to which we have applied this scheme using the annotation tool BRAT. We discuss some of the issues, both theoretical and practical, that we encountered, show how the corpus helps in answering specific questions, and conclude that REPORTS fitted in well with our needs.
Breaking NLP: Using Morphosyntax, Semantics, Pragmatics and World Knowledge to Fool Sentiment Analysis Systems
Authors: Taylor Mahler, Willy Cheung, Micha Elsner, David King, Marie-Catherine de Marneffe, Cory Shain, Symon Stevens-Guille, Michael White
Date: 2017-09-01 00:00:00
Proceeding: WS 2017 9
Tasks: Sentiment Analysis, World Knowledge

This paper describes our {``}breaker{''} submission to the 2017 EMNLP {``}Build It Break It{''} shared task on sentiment analysis. In order to cause the {``}builder{''} systems to make incorrect predictions, we edited items in the blind test data according to linguistically interpretable strategies that allow us to assess the ease with which the builder systems learn various components of linguistic structure. On the whole, our submitted pairs break all systems at a high rate (72.6{\%}), indicating that sentiment analysis as an NLP task may still have a lot of ground to cover. Of the breaker strategies that we consider, we find our semantic and pragmatic manipulations to pose the most substantial difficulties for the builder systems.
Representation and Interchange of Linguistic Annotation. An In-Depth, Side-by-Side Comparison of Three Designs
Authors: Richard Eckart de Castilho, Nancy Ide, Emanuele Lapponi, Stephan Oepen, Keith Suderman, Erik Velldal, Marc Verhagen
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4

For decades, most self-respecting linguistic engineering initiatives have designed and implemented custom representations for various layers of, for example, morphological, syntactic, and semantic analysis. Despite occasional efforts at harmonization or even standardization, our field today is blessed with a multitude of ways of encoding and exchanging linguistic annotations of these types, both at the levels of {`}abstract syntax{'}, naming choices, and of course file formats. To a large degree, it is possible to work within and across design plurality by conversion, and often there may be good reasons for divergent design reflecting differences in use. However, it is likely that some abstract commonalities across choices of representation are obscured by more superficial differences, and conversely there is no obvious procedure to tease apart what actually constitute contentful vs. mere technical divergences. In this study, we seek to conceptually align three representations for common types of morpho-syntactic analysis, pinpoint what in our view constitute contentful differences, and reflect on the underlying principles and specific requirements that led to individual choices. We expect that a more in-depth understanding of these choices across designs may led to increased harmonization, or at least to more informed design of future representations.
TDB 1.1: Extensions on Turkish Discourse Bank
Authors: Deniz Zeyrek, Murathan Kurfal{\i}
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Implicit Relations, Relation

This paper presents the recent developments on Turkish Discourse Bank (TDB). First, the resource is summarized and an evaluation is presented. Then, TDB 1.1, i.e. enrichments on 10{\%} of the corpus are described (namely, senses for explicit discourse connectives, and new annotations for three discourse relation types - implicit relations, entity relations and alternative lexicalizations). The method of annotation is explained and the data are evaluated.
Two Layers of Annotation for Representing Event Mentions in News Stories
Authors: Maria Pia di Buono, Martin Tutek, Jan {\v{S}}najder, Goran Glava{\v{s}}, Bojana Dalbelo Ba{\v{s}}i{\'c}, Nata{\v{s}}a Mili{\'c}-Frayling
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Event Extraction, Vocal Bursts Valence Prediction

In this paper, we describe our preliminary study on annotating event mention as a part of our research on high-precision news event extraction models. To this end, we propose a two-layer annotation scheme, designed to separately capture the functional and conceptual aspects of event mentions. We hypothesize that the precision of models can be improved by modeling and extracting separately the different aspects of news events, and then combining the extracted information by leveraging the complementarities of the models. In addition, we carry out a preliminary annotation using the proposed scheme and analyze the annotation quality in terms of inter-annotator agreement.
Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems
Authors: Syed Sarfaraz Akhtar, Arihant Gupta, Avijit Vajpayee, Arjit Srivastava, Manish Shrivastava
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Dependency Parsing, Machine Translation, Named Entity Recognition (NER), Question Answering, Semantic Textual Similarity, Translation, Word Sense Disambiguation, Word Similarity

With the advent of word representations, word similarity tasks are becoming increasing popular as an evaluation metric for the quality of the representations. In this paper, we present manually annotated monolingual word similarity datasets of six Indian languages - Urdu, Telugu, Marathi, Punjabi, Tamil and Gujarati. These languages are most spoken Indian languages worldwide after Hindi and Bengali. For the construction of these datasets, our approach relies on translation and re-annotation of word similarity datasets of English. We also present baseline scores for word representation models using state-of-the-art techniques for Urdu, Telugu and Marathi by evaluating them on newly created word similarity datasets.
The BECauSE Corpus 2.0: Annotating Causality and Overlapping Relations
Authors: Jesse Dunietz, Lori Levin, Jaime Carbonell
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Decision Making

Language of cause and effect captures an essential component of the semantics of a text. However, causal language is also intertwined with other semantic relations, such as temporal precedence and correlation. This makes it difficult to determine when causation is the primary intended meaning. This paper presents BECauSE 2.0, a new version of the BECauSE corpus with exhaustively annotated expressions of causal language, but also seven semantic relations that are frequently co-present with causation. The new corpus shows high inter-annotator agreement, and yields insights both about the linguistic expressions of causation and about the process of annotating co-present semantic relations.
Catching the Common Cause: Extraction and Annotation of Causal Relations and their Participants
Authors: Ines Rehbein, Josef Ruppenhofer
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Language Modeling, Language Modelling

In this paper, we present a simple, yet effective method for the automatic identification and extraction of causal relations from text, based on a large English-German parallel corpus. The goal of this effort is to create a lexical resource for German causal relations. The resource will consist of a lexicon that describes constructions that trigger causality as well as the participants of the causal event, and will be augmented by a corpus with annotated instances for each entry, that can be used as training data to develop a system for automatic classification of causal relations. Focusing on verbs, our method harvested a set of 100 different lexical triggers of causality, including support verb constructions. At the moment, our corpus includes over 1,000 annotated instances. The lexicon and the annotated data will be made available to the research community.
Assessing SRL Frameworks with Automatic Training Data Expansion
Authors: Silvana Hartmann, {\'E}va M{\'u}jdricza-Maydt, Ilia Kuznetsov, Iryna Gurevych, Anette Frank
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Question Answering, Semantic Role Labeling

We present the first experiment-based study that explicitly contrasts the three major semantic role labeling frameworks. As a prerequisite, we create a dataset labeled with parallel FrameNet-, PropBank-, and VerbNet-style labels for German. We train a state-of-the-art SRL tool for German for the different annotation styles and provide a comparative analysis across frameworks. We further explore the behavior of the frameworks with automatic training data generation. VerbNet provides larger semantic expressivity than PropBank, and we find that its generalization capacity approaches PropBank in SRL training, but it benefits less from training data expansion than the sparse-data affected FrameNet.
Inducing Script Structure from Crowdsourced Event Descriptions via Semi-Supervised Clustering
Authors: Lilian Wanzare, Aless Zarcone, ra, Stefan Thater, Manfred Pinkal
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Clustering, Question Answering, Semantic Role Labeling

We present a semi-supervised clustering approach to induce script structure from crowdsourced descriptions of event sequences by grouping event descriptions into paraphrase sets (representing event types) and inducing their temporal order. Our approach exploits semantic and positional similarity and allows for flexible event order, thus overcoming the rigidity of previous approaches. We incorporate crowdsourced alignments as prior knowledge and show that exploiting a small number of alignments results in a substantial improvement in cluster quality over state-of-the-art models and provides an appropriate basis for the induction of temporal order. We also show a coverage study to demonstrate the scalability of our approach.
A Consolidated Open Knowledge Representation for Multiple Texts
Authors: Rachel Wities, Vered Shwartz, Gabriel Stanovsky, Meni Adler, Ori Shapira, Shyam Upadhyay, Dan Roth, Eugenio Martinez Camara, Iryna Gurevych, Ido Dagan
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Lexical Entailment, Open Information Extraction

We propose to move from Open Information Extraction (OIE) ahead to Open Knowledge Representation (OKR), aiming to represent information conveyed jointly in a set of texts in an open text-based manner. We do so by consolidating OIE extractions using entity and predicate coreference, while modeling information containment between coreferring elements via lexical entailment. We suggest that generating OKR structures can be a useful step in the NLP pipeline, to give semantic applications an easy handle on consolidated information across multiple texts.
Event-Related Features in Feedforward Neural Networks Contribute to Identifying Causal Relations in Discourse
Authors: Edoardo Maria Ponti, Anna Korhonen
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Position, Question Answering, Text Summarization

Causal relations play a key role in information extraction and reasoning. Most of the times, their expression is ambiguous or implicit, i.e. without signals in the text. This makes their identification challenging. We aim to improve their identification by implementing a Feedforward Neural Network with a novel set of features for this task. In particular, these are based on the position of event mentions and the semantics of events and participants. The resulting classifier outperforms strong baselines on two datasets (the Penn Discourse Treebank and the CSTNews corpus) annotated with different schemes and containing examples in two languages, English and Portuguese. This result demonstrates the importance of events for identifying discourse relations.
Stance Detection in Facebook Posts of a German Right-wing Party
Authors: Manfred Klenner, Don Tuggener, Simon Clematide
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Relation Extraction, Stance Detection

We argue that in order to detect stance, not only the explicit attitudes of the stance holder towards the targets are crucial. It is the whole narrative the writer drafts that counts, including the way he hypostasizes the discourse referents: as benefactors or villains, as victims or beneficiaries. We exemplify the ability of our system to identify targets and detect the writer{'}s stance towards them on the basis of about 100 000 Facebook posts of a German right-wing party. A reader and writer model on top of our verb-based attitude extraction directly reveal stance conflicts.
Behind the Scenes of an Evolving Event Cloze Test
Authors: Nathanael Chambers
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Cloze Test, Common Sense Reasoning, Language Modeling, Language Modelling

This paper analyzes the narrative event cloze test and its recent evolution. The test removes one event from a document{'}s chain of events, and systems predict the missing event. Originally proposed to evaluate learned knowledge of event scenarios (e.g., scripts and frames), most recent work now builds ngram-like language models (LM) to beat the test. This paper argues that the test has slowly/unknowingly been altered to accommodate LMs.5 Most notably, tests are auto-generated rather than by hand, and no effort is taken to include core script events. Recent work is not clear on evaluation goals and contains contradictory results. We implement several models, and show that the test{'}s bias to high-frequency events explains the inconsistencies. We conclude with recommendations on how to return to the test{'}s original intent, and offer brief suggestions on a path forward.
LSDSem 2017 Shared Task: The Story Cloze Test
Authors: Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, James Allen
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Cloze Test, Reading Comprehension, Sentence

The LSDSem{'}17 shared task is the Story Cloze Test, a new evaluation for story understanding and script learning. This test provides a system with a four-sentence story and two possible endings, and the system must choose the correct ending to the story. Successful narrative understanding (getting closer to human performance of 100{\%}) requires systems to link various levels of semantics to commonsense knowledge. A total of eight systems participated in the shared task, with a variety of approaches including.
Story Cloze Task: UW NLP System
Authors: Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, Noah A. Smith
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Language Modeling, Language Modelling

This paper describes University of Washington NLP{'}s submission for the Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem 2017) shared task{---}the Story Cloze Task. Our system is a linear classifier with a variety of features, including both the scores of a neural language model and style features. We report 75.2{\%} accuracy on the task. A further discussion of our results can be found in Schwartz et al. (2017).
LSDSem 2017: Exploring Data Generation Methods for the Story Cloze Test
Authors: Michael Bugert, Yevgeniy Puzikov, Andreas R{\"u}ckl{\'e}, Judith Eckle-Kohler, Teresa Martin, Eugenio Mart{\'\i}nez-C{\'a}mara, Daniil Sorokin, Maxime Peyrard, Iryna Gurevych
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Cloze Test

The Story Cloze test is a recent effort in providing a common test scenario for text understanding systems. As part of the LSDSem 2017 shared task, we present a system based on a deep learning architecture combined with a rich set of manually-crafted linguistic features. The system outperforms all known baselines for the task, suggesting that the chosen approach is promising. We additionally present two methods for generating further training data based on stories from the ROCStories corpus.
Sentiment Analysis and Lexical Cohesion for the Story Cloze Task
Authors: Michael Flor, Swapna Somasundaran
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Sentiment Analysis

We present two NLP components for the Story Cloze Task {--} dictionary-based sentiment analysis and lexical cohesion. While previous research found no contribution from sentiment analysis to the accuracy on this task, we demonstrate that sentiment is an important aspect. We describe a new approach, using a rule that estimates sentiment congruence in a story. Our sentiment-based system achieves strong results on this task. Our lexical cohesion system achieves accuracy comparable to previously published baseline results. A combination of the two systems achieves better accuracy than published baselines. We argue that sentiment analysis should be considered an integral part of narrative comprehension.
Resource-Lean Modeling of Coherence in Commonsense Stories
Authors: Niko Schenk, Christian Chiarcos
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Cloze Test, Reading Comprehension, Semantic Role Labeling

We present a resource-lean neural recognizer for modeling coherence in commonsense stories. Our lightweight system is inspired by successful attempts to modeling discourse relations and stands out due to its simplicity and easy optimization compared to prior approaches to narrative script learning. We evaluate our approach in the Story Cloze Test demonstrating an absolute improvement in accuracy of 4.7{\%} over state-of-the-art implementations.
An RNN-based Binary Classifier for the Story Cloze Test
Authors: Melissa Roemmele, Sosuke Kobayashi, Naoya Inoue, Andrew Gordon
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Cloze Test, Sentence, Word Embeddings

The Story Cloze Test consists of choosing a sentence that best completes a story given two choices. In this paper we present a system that performs this task using a supervised binary classifier on top of a recurrent neural network to predict the probability that a given story ending is correct. The classifier is trained to distinguish correct story endings given in the training data from incorrect ones that we artificially generate. Our experiments evaluate different methods for generating these negative examples, as well as different embedding-based representations of the stories. Our best result obtains 67.2{\%} accuracy on the test set, outperforming the existing top baseline of 58.5{\%}.
IIT (BHU): System Description for LSDSem'17 Shared Task
Authors: Pranav Goel, Anil Kumar Singh
Date: 2017-04-01 00:00:00
Proceeding: WS 2017 4
Tasks: Cloze Test, Common Sense Reasoning, regression, Semantic Similarity, Semantic Textual Similarity

This paper describes an ensemble system submitted as part of the LSDSem Shared Task 2017 - the Story Cloze Test. The main conclusion from our results is that an approach based on semantic similarity alone may not be enough for this task. We test various approaches and compare them with two ensemble systems. One is based on voting and the other on logistic regression based classifier. Our final system is able to outperform the previous state of the art for the Story Cloze test. Another very interesting observation is the performance of sentiment based approach which works almost as well on its own as our final ensemble system.