pwc-archive/papers-with-abstracts huggingface dataset:
Debugging Sequence-to-Sequence Models with Seq2Seq-Vis
Authors: Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, Alex Rush, er
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Attribute, Translation

Neural attention-based sequence-to-sequence models (seq2seq) (Sutskever et al., 2014; Bahdanau et al., 2014) have proven to be accurate and robust for many sequence prediction tasks. They have become the standard approach for automatic translation of text, at the cost of increased model complexity and uncertainty. End-to-end trained neural models act as a black box, which makes it difficult to examine model decisions and attribute errors to a specific part of a model. The highly connected and high-dimensional internal representations pose a challenge for analysis and visualization tools. The development of methods to understand seq2seq predictions is crucial for systems in production settings, as mistakes involving language are often very apparent to human readers. For instance, a widely publicized incident resulted from a translation system mistakenly translating {``}good morning{''} into {``}attack them{''} leading to a wrongful arrest (Hern, 2017).
Does Syntactic Knowledge in Multilingual Language Models Transfer Across Languages?
Authors: Prajit Dhar, Arianna Bisazza
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Language Acquisition, Language Modeling, Language Modelling

Recent work has shown that neural models can be successfully trained on multiple languages simultaneously. We investigate whether such models learn to share and exploit common syntactic knowledge among the languages on which they are trained. This extended abstract presents our preliminary results.
Exploiting Attention to Reveal Shortcomings in Memory Models
Authors: Kaylee Burns, Aida Nematzadeh, Erin Grant, Alison Gopnik, Tom Griffiths
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: BIG-bench Machine Learning, Decision Making, General Classification, Question Answering

The decision making processes of deep networks are difficult to understand and while their accuracy often improves with increased architectural complexity, so too does their opacity. Practical use of machine learning models, especially for question and answering applications, demands a system that is interpretable. We analyze the attention of a memory network model to reconcile contradictory performance on a challenging question-answering dataset that is inspired by theory-of-mind experiments. We equate success on questions to task classification, which explains not only test-time failures but also how well the model generalizes to new training conditions.
End-to-end Image Captioning Exploits Distributional Similarity in Multimodal Space
Authors: Pranava Swaroop Madhyastha, Josiah Wang, Lucia Specia
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Image Captioning, Text Generation

We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn {`}distributional similarity{'} in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space. To validate our hypothesis, we focus on the {`}image{'} side of image captioning, and vary the input image representation but keep the RNN text generation model of a CNN-RNN constant. Our analysis indicates that image captioning models (i) are capable of separating structure from noisy input representations; (ii) experience virtually no significant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together. Our experiments all point to one fact: that our distributional similarity hypothesis holds. We conclude that, regardless of the image representation, image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace.
The Data Challenge in Misinformation Detection: Source Reputation vs. Content Veracity
Authors: Fatemeh Torabi Asr, Maite Taboada
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Articles, Domain Generalization, Fact Checking, Fake News Detection, General Classification, Misinformation, text-classification, Text Classification

Misinformation detection at the level of full news articles is a text classification problem. Reliably labeled data in this domain is rare. Previous work relied on news articles collected from so-called {``}reputable{''} and {``}suspicious{''} websites and labeled accordingly. We leverage fact-checking websites to collect individually-labeled news articles with regard to the veracity of their content and use this data to test the cross-domain generalization of a classifier trained on bigger text collections but labeled according to source reputation. Our results suggest that reputation-based classification is not sufficient for predicting the veracity level of the majority of news articles, and that the system performance on different test datasets depends on topic distribution. Therefore collecting well-balanced and carefully-assessed training data is a priority for developing robust misinformation detection systems.
Retrieve and Re-rank: A Simple and Effective IR Approach to Simple Question Answering over Knowledge Graphs
Authors: Vishal Gupta, Manoj Chinnakotla, Manish Shrivastava
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Entity Linking, Information Retrieval, Knowledge Graphs, Learning-To-Rank, Question Answering, Relation Prediction, Re-Ranking, Retrieval, Triplet

SimpleQuestions is a commonly used benchmark for single-factoid question answering (QA) over Knowledge Graphs (KG). Existing QA systems rely on various components to solve different sub-tasks of the problem (such as entity detection, entity linking, relation prediction and evidence integration). In this work, we propose a different approach to the problem and present an information retrieval style solution for it. We adopt a two-phase approach: candidate generation and candidate re-ranking to answer questions. We propose a Triplet-Siamese-Hybrid CNN (TSHCNN) to re-rank candidate answers. Our approach achieves an accuracy of 80{\%} which sets a new state-of-the-art on the SimpleQuestions dataset.
Information Nutrition Labels: A Plugin for Online News Evaluation
Authors: Vincentius Kevin, Birte H{\"o}gden, Claudia Schwenger, Ali {\c{S}}ahan, Neelu Madan, Piush Aggarwal, Anusha Bangaru, Farid Muradov, Ahmet Aker
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Articles, Decision Making, Nutrition, TAG

In this paper we present a browser plugin \textit{NewsScan} that assists online news readers in evaluating the quality of online content they read by providing \textit{information nutrition labels} for online news articles. In analogy to groceries, where nutrition labels help consumers make choices that they consider best for themselves, information nutrition labels tag online news articles with data that help readers judge the articles they engage with. This paper discusses the choice of the labels, their implementation and visualization.
Joint Modeling for Query Expansion and Information Extraction with Reinforcement Learning
Authors: Motoki Taniguchi, Yasuhide Miura, Tomoko Ohkuma
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Decision Making, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Retrieval

Information extraction about an event can be improved by incorporating external evidence. In this study, we propose a joint model for pseudo-relevance feedback based query expansion and information extraction with reinforcement learning. Our model generates an event-specific query to effectively retrieve documents relevant to the event. We demonstrate that our model is comparable or has better performance than the previous model in two publicly available datasets. Furthermore, we analyzed the influences of the retrieval effectiveness in our model on the extraction performance.
Towards Automatic Fake News Detection: Cross-Level Stance Detection in News Articles
Authors: Costanza Conforti, Mohammad Taher Pilehvar, Nigel Collier
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Articles, Fake News Detection, Stance Detection

In this paper, we propose to adapt the four-staged pipeline proposed by Zubiaga et al. (2018) for the Rumor Verification task to the problem of Fake News Detection. We show that the recently released FNC-1 corpus covers two of its steps, namely the \textit{Tracking} and the \textit{Stance Detection} task. We identify asymmetry in length in the input to be a key characteristic of the latter step, when adapted to the framework of Fake News Detection, and propose to handle it as a specific type of \textit{Cross-Level Stance Detection}. Inspired by theories from the field of Journalism Studies, we implement and test two architectures to successfully model the internal structure of an article and its interactions with a claim.
Automated Fact-Checking of Claims in Argumentative Parliamentary Debates
Authors: Nona Naderi, Graeme Hirst
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Binary Classification, Classification, Fact Checking, General Classification, Multi-class Classification, Semantic Textual Similarity

We present an automated approach to distinguish true, false, stretch, and dodge statements in questions and answers in the Canadian Parliament. We leverage the truthfulness annotations of a U.S. fact-checking corpus by training a neural net model and incorporating the prediction probabilities into our models. We find that in concert with other linguistic features, these probabilities can improve the multi-class classification results. We further show that dodge statements can be detected with an F1 measure as high as 82.57{\%} in binary classification settings.
Stance Detection in Fake News A Combined Feature Representation
Authors: Bilal Ghanem, Paolo Rosso, Francisco Rangel
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: General Classification, Stance Detection, Word Embeddings

With the uncontrolled increasing of fake news and rumors over the Web, different approaches have been proposed to address the problem. In this paper, we present an approach that combines lexical, word embeddings and n-gram features to detect the stance in fake news. Our approach has been tested on the Fake News Challenge (FNC-1) dataset. Given a news title-article pair, the FNC-1 task aims at determining the relevance of the article and the title. Our proposed approach has achieved an accurate result (59.6 {\%} Macro F1) that is close to the state-of-the-art result with 0.013 difference using a simple feature representation. Furthermore, we have investigated the importance of different lexicons in the detection of the classification labels.
Zero-shot Relation Classification as Textual Entailment
Authors: Abiola Obamuyide, Andreas Vlachos
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Classification, General Classification, Knowledge Base Population, Natural Language Inference, Question Answering, Relation, Relation Classification, Relation Extraction, zero-shot-classification, Zero-Shot Learning, Zero-shot Relation Classification

We consider the task of relation classification, and pose this task as one of textual entailment. We show that this formulation leads to several advantages, including the ability to (i) perform zero-shot relation classification by exploiting relation descriptions, (ii) utilize existing textual entailment models, and (iii) leverage readily available textual entailment datasets, to enhance the performance of relation classification systems. Our experiments show that the proposed approach achieves 20.16{\%} and 61.32{\%} in F1 zero-shot classification performance on two datasets, which further improved to 22.80{\%} and 64.78{\%} respectively with the use of conditional encoding.
Where is Your Evidence: Improving Fact-checking by Justification Modeling
Authors: Tariq Alhindi, Savvas Petridis, Smar Muresan, a
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Argument Mining, BIG-bench Machine Learning, Binary Classification, Emotion Recognition, Fact Checking, General Classification

Fact-checking is a journalistic practice that compares a claim made publicly against trusted sources of facts. Wang (2017) introduced a large dataset of validated claims from the POLITIFACT.com website (LIAR dataset), enabling the development of machine learning approaches for fact-checking. However, approaches based on this dataset have focused primarily on modeling the claim and speaker-related metadata, without considering the evidence used by humans in labeling the claims. We extend the LIAR dataset by automatically extracting the justification from the fact-checking article used by humans to label a given claim. We show that modeling the extracted justification in conjunction with the claim (and metadata) provides a significant improvement regardless of the machine learning model used (feature-based or deep learning) both in a binary classification task (true, false) and in a six-way classification task (pants on fire, false, mostly false, half true, mostly true, true).
UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF)
Authors: Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, Sebastian Riedel
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Information Retrieval, Natural Language Inference, Reading Comprehension, Retrieval, Sentence, Sentence Retrieval

In this paper we describe our 2nd place FEVER shared-task system that achieved a FEVER score of 62.52{\%} on the provisional test set (without additional human evaluation), and 65.41{\%} on the development set. Our system is a four stage model consisting of document retrieval, sentence retrieval, natural language inference and aggregation. Retrieval is performed leveraging task-specific features, and then a natural language inference model takes each of the retrieved sentences paired with the claimed fact. The resulting predictions are aggregated across retrieved sentences with a Multi-Layer Perceptron, and re-ranked corresponding to the final prediction.
SIRIUS-LTG: An Entity Linking Approach to Fact Extraction and Verification
Authors: Farhad Nooralahzadeh, Lilja {\O}vrelid
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Entity Linking, Information Retrieval, Natural Language Inference, Question Answering, Retrieval, Sentence

This article presents the SIRIUS-LTG system for the Fact Extraction and VERification (FEVER) Shared Task. It consists of three components: 1) \textit{Wikipedia Page Retrieval}: First we extract the entities in the claim, then we find potential Wikipedia URI candidates for each of the entities using a SPARQL query over DBpedia 2) \textit{Sentence selection}: We investigate various techniques i.e. Smooth Inverse Frequency (SIF), Word Mover{'}s Distance (WMD), Soft-Cosine Similarity, Cosine similarity with unigram Term Frequency Inverse Document Frequency (TF-IDF) to rank sentences by their similarity to the claim. 3) \textit{Textual Entailment}: We compare three models for the task of claim classification. We apply a Decomposable Attention (DA) model (Parikh et al., 2016), a Decomposed Graph Entailment (DGE) model (Khot et al., 2018) and a Gradient-Boosted Decision Trees (TalosTree) model (Sean et al., 2017) for this task. The experiments show that the pipeline with simple Cosine Similarity using TFIDF in sentence selection along with DA model as labelling model achieves the best results on the development set (F1 evidence: 32.17, label accuracy: 59.61 and FEVER score: 0.3778). Furthermore, it obtains 30.19, 48.87 and 36.55 in terms of F1 evidence, label accuracy and FEVER score, respectively, on the test set. Our system ranks 15th among 23 participants in the shared task prior to any human-evaluation of the evidence.
Integrating Entity Linking and Evidence Ranking for Fact Extraction and Verification
Authors: Motoki Taniguchi, Tomoki Taniguchi, Takumi Takahashi, Yasuhide Miura, Tomoko Ohkuma
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Entity Linking, Natural Language Inference, Retrieval, RTE, Sentence, Sentence Retrieval

We describe here our system and results on the FEVER shared task. We prepared a pipeline system which composes of a document selection, a sentence retrieval, and a recognizing textual entailment (RTE) components. A simple entity linking approach with text match is used as the document selection component, this component identifies relevant documents for a given claim by using mentioned entities as clues. The sentence retrieval component selects relevant sentences as candidate evidence from the documents based on TF-IDF. Finally, the RTE component selects evidence sentences by ranking the sentences and classifies the claim simultaneously. The experimental results show that our system achieved the FEVER score of 0.4016 and outperformed the official baseline system.
Robust Document Retrieval and Individual Evidence Modeling for Fact Extraction and Verification.
Authors: Tuhin Chakrabarty, Tariq Alhindi, Smar Muresan, a
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Natural Language Inference, Retrieval, Sentence

This paper presents the ColumbiaNLP submission for the FEVER Workshop Shared Task. Our system is an end-to-end pipeline that extracts factual evidence from Wikipedia and infers a decision about the truthfulness of the claim based on the extracted evidence. Our pipeline achieves significant improvement over the baseline for all the components (Document Retrieval, Sentence Selection and Textual Entailment) both on the development set and the test set. Our team finished 6th out of 24 teams on the leader-board based on the preliminary results with a FEVER score of 49.06 on the blind test set compared to 27.45 of the baseline system.
An End-to-End Multi-task Learning Model for Fact Checking
Authors: Sizhen Li, Shuai Zhao, Bo Cheng, Hao Yang
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Common Sense Reasoning, Entity Linking, Fact Checking, Fake News Detection, Multi-Task Learning, Natural Language Inference

With huge amount of information generated every day on the web, fact checking is an important and challenging task which can help people identify the authenticity of most claims as well as providing evidences selected from knowledge source like Wikipedia. Here we decompose this problem into two parts: an entity linking task (retrieving relative Wikipedia pages) and recognizing textual entailment between the claim and selected pages. In this paper, we present an end-to-end multi-task learning with bi-direction attention (EMBA) model to classify the claim as {``}supports{''}, {``}refutes{''} or {``}not enough info{''} with respect to the pages retrieved and detect sentences as evidence at the same time. We conduct experiments on the FEVER (Fact Extraction and VERification) paper test dataset and shared task test dataset, a new public dataset for verification against textual sources. Experimental results show that our method achieves comparable performance compared with the baseline system.
Team GESIS Cologne: An all in all sentence-based approach for FEVER
Authors: Wolfgang Otto
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: All, Coreference Resolution, Sentence

In this system description of our pipeline to participate at the Fever Shared Task, we describe our sentence-based approach. Throughout all steps of our pipeline, we regarded single sentences as our processing unit. In our IR-Component, we searched in the set of all possible Wikipedia introduction sentences without limiting sentences to a fixed number of relevant documents. In the entailment module, we judged every sentence separately and combined the result of the classifier for the top 5 sentences with the help of an ensemble classifier to make a judgment whether the truth of a statement can be derived from the given claim.
Team SWEEPer: Joint Sentence Extraction and Fact Checking with Pointer Networks
Authors: Christopher Hidey, Mona Diab
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Fact Checking, Information Retrieval, Multi-Task Learning, Natural Language Inference, Question Answering, Reading Comprehension, Sentence

Many tasks such as question answering and reading comprehension rely on information extracted from unreliable sources. These systems would thus benefit from knowing whether a statement from an unreliable source is correct. We present experiments on the FEVER (Fact Extraction and VERification) task, a shared task that involves selecting sentences from Wikipedia and predicting whether a claim is supported by those sentences, refuted, or there is not enough information. Fact checking is a task that benefits from not only asserting or disputing the veracity of a claim but also finding evidence for that position. As these tasks are dependent on each other, an ideal model would consider the veracity of the claim when finding evidence and also find only the evidence that is relevant. We thus jointly model sentence extraction and verification on the FEVER shared task. Among all participants, we ranked 5th on the blind test set (prior to any additional human evaluation of the evidence).
QED: A fact verification system for the FEVER shared task
Authors: Jackson Luken, Nanjiang Jiang, Marie-Catherine de Marneffe
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Fact Verification, General Classification, Retrieval

This paper describes our system submission to the 2018 Fact Extraction and VERification (FEVER) shared task. The system uses a heuristics-based approach for evidence extraction and a modified version of the inference model by Parikh et al. (2016) for classification. Our process is broken down into three modules: potentially relevant documents are gathered based on key phrases in the claim, then any possible evidence sentences inside those documents are extracted, and finally our classifier discards any evidence deemed irrelevant and uses the remaining to classify the claim{'}s veracity. Our system beats the shared task baseline by 12{\%} and is successful at finding correct evidence (evidence retrieval F1 of 62.5{\%} on the development set).
Team UMBC-FEVER : Claim verification using Semantic Lexical Resources
Authors: Ankur Padia, Francis Ferraro, Tim Finin
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Claim Verification, Information Retrieval, Retrieval

We describe our system used in the 2018 FEVER shared task. The system employed a frame-based information retrieval approach to select Wikipedia sentences providing evidence and used a two-layer multilayer perceptron to classify a claim as correct or not. Our submission achieved a score of 0.3966 on the Evidence F1 metric with accuracy of 44.79{\%}, and FEVER score of 0.2628 F1 points.