pwc-archive/papers-with-abstracts huggingface dataset:
Personalized Response Generation with Tensor Factorization
Authors: Zhenghui Wang, Lingxiao Luo, Diyi Yang
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Decoder, Language Modeling, Language Modelling, Response Generation

Personalized response generation is essential for more human-like conversations. However, how to model user personalization information with no explicit user persona descriptions or demographics still remains under-investigated. To tackle the data sparsity problem and the huge number of users, we utilize tensor factorization to model users’ personalization information with their posting histories. Specifically, we introduce the personalized response embedding for all question-user pairs and form them into a three-mode tensor, decomposed by Tucker decomposition. The personalized response embedding is fed to either the decoder of an LSTM-based Seq2Seq model or a transformer language model to help generate more personalized responses. To evaluate how personalized the generated responses are, we further propose a novel ranking-based metric called Per-Hits@k which measures how likely are the generated responses come from the corresponding users. Results on a large-scale conversation dataset show that our proposed tensor factorization based models generate more personalized and higher quality responses compared to baselines.
GOT: Testing for Originality in Natural Language Generation
Authors: Jennifer Brooks, Abdou Youssef
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Text Generation

We propose an approach to automatically test for originality in generation tasks where no standard automatic measures exist. Our proposal addresses original uses of language, not necessarily original ideas. We provide an algorithm for our approach and a run-time analysis. The algorithm, which finds all of the original fragments in a ground-truth corpus and can reveal whether a generated fragment copies an original without attribution, has a run-time complexity of theta(nlogn) where n is the number of sentences in the ground truth.
Evaluating Text Generation from Discourse Representation Structures
Authors: Chunliu Wang, Rik van Noord, Arianna Bisazza, Johan Bos
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Text Generation

We present an end-to-end neural approach to generate English sentences from formal meaning representations, Discourse Representation Structures (DRSs). We use a rather standard bi-LSTM sequence-to-sequence model, work with a linearized DRS input representation, and evaluate character-level and word-level decoders. We obtain very encouraging results in terms of reference-based automatic metrics such as BLEU. But because such metrics only evaluate the surface level of generated output, we develop a new metric, ROSE, that targets specific semantic phenomena. We do this with five DRS generation challenge sets focusing on tense, grammatical number, polarity, named entities and quantities. The aim of these challenge sets is to assess the neural generator’s systematicity and generalization to unseen inputs.
Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task
Authors: Shreyan Bakshi, Soumya Batra, Peyman Heidari, Ankit Arun, Shashank Jain, Michael White
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Text Generation

We explore the use of self-training and acceptability classifiers with pre-trained models for natural language generation in structure-to-text settings using three GEM datasets (E2E, WebNLG-en, Schema-Guided Dialog). With the Schema-Guided Dialog dataset, we also experiment with including multiple turns of context in the input. We find that self-training with reconstruction matching along with acceptability classifier filtering can improve semantic correctness, though gains are limited in the full-data setting. With context-conditioning, we find that including multiple turns in the context encourages the model to align with the user’s word and phrasing choices as well as to generate more self-consistent responses. In future versions of the GEM challenge, we encourage the inclusion of few-shot tracks to encourage research on data efficiency.
NUIG-DSI’s submission to The GEM Benchmark 2021
Authors: Nivranshu Pasricha, Mihael Arcan, Paul Buitelaar
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Data-to-Text Generation, Text Generation

This paper describes the submission by NUIG-DSI to the GEM benchmark 2021. We participate in the modeling shared task where we submit outputs on four datasets for data-to-text generation, namely, DART, WebNLG (en), E2E and CommonGen. We follow an approach similar to the one described in the GEM benchmark paper where we use the pre-trained T5-base model for our submission. We train this model on additional monolingual data where we experiment with different masking strategies specifically focused on masking entities, predicates and concepts as well as a random masking strategy for pre-training. In our results we find that random masking performs the best in terms of automatic evaluation metrics, though the results are not statistically significantly different compared to other masking strategies.
SimpleNER Sentence Simplification System for GEM 2021
Authors: K V Aditya Srivatsa, Monil Gokani, Manish Shrivastava
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: NER, Sentence

This paper describes SimpleNER, a model developed for the sentence simplification task at GEM-2021. Our system is a monolingual Seq2Seq Transformer architecture that uses control tokens pre-pended to the data, allowing the model to shape the generated simplifications according to user desired attributes. Additionally, we show that NER-tagging the training data before use helps stabilize the effect of the control tokens and significantly improves the overall performance of the system. We also employ pretrained embeddings to reduce data sparsity and allow the model to produce more generalizable outputs.
System Description for the CommonGen task with the POINTER model
Authors: Anna Shvets
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Diversity, Position, Semantic Similarity, Semantic Textual Similarity

In a current experiment we were testing CommonGen dataset for structure-to-text task from GEM living benchmark with the constraint based POINTER model. POINTER represents a hybrid architecture, combining insertion-based and transformer paradigms, predicting the token and the insertion position at the same time. The text is therefore generated gradually in a parallel non-autoregressive manner, given the set of keywords. The pretrained model was fine-tuned on a training split of the CommonGen dataset and the generation result was compared to the validation and challenge splits. The received metrics outputs, which measure lexical equivalence, semantic similarity and diversity, are discussed in details in a present system description.
The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task
Authors: Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Retrieval

The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are Supported or Refuted based on evidence retrieved from Wikipedia (or NotEnoughInfo if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018, FEVEROUS requires partial evidence to be returned for NotEnoughInfo claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.
Evidence Selection as a Token-Level Prediction Task
Authors: Dominik Stammbach
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Claim Verification, Evidence Selection, Prediction, Retrieval, Sentence, Sentence Classification

In Automated Claim Verification, we retrieve evidence from a knowledge base to determine the veracity of a claim. Intuitively, the retrieval of the correct evidence plays a crucial role in this process. Often, evidence selection is tackled as a pairwise sentence classification task, i.e., we train a model to predict for each sentence individually whether it is evidence for a claim. In this work, we fine-tune document level transformers to extract all evidence from a Wikipedia document at once. We show that this approach performs better than a comparable model classifying sentences individually on all relevant evidence selection metrics in FEVER. Our complete pipeline building on this evidence selection procedure produces a new state-of-the-art result on FEVER, a popular claim verification benchmark.
FaBULOUS: Fact-checking Based on Understanding of Language Over Unstructured and Structured information
Authors: Mostafa Bouziane, Hugo Perrin, Amine Sadeq, Thanh Nguyen, Aurélien Cluzeau, Julien Mardas
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Fact Checking, Retrieval

As part of the FEVEROUS shared task, we developed a robust and finely tuned architecture to handle the joint retrieval and entailment on text data as well as structured data like tables. We proposed two training schemes to tackle the hurdles inherent to multi-hop multi-modal datasets. The first one allows having a robust retrieval of full evidence sets, while the second one enables entailment to take full advantage of noisy evidence inputs. In addition, our work has revealed important insights and potential avenue of research for future improvement on this kind of dataset. In preliminary evaluation on the FEVEROUS shared task test set, our system achieves 0.271 FEVEROUS score, with 0.4258 evidence recall and 0.5607 entailment accuracy.
Team Papelo at FEVEROUS: Multi-hop Evidence Pursuit
Authors: Christopher Malon
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11

We develop a system for the FEVEROUS fact extraction and verification task that ranks an initial set of potential evidence and then pursues missing evidence in subsequent hops by trying to generate it, with a “next hop prediction module” whose output is matched against page elements in a predicted article. Seeking evidence with the next hop prediction module continues to improve FEVEROUS score for up to seven hops. Label classification is trained on possibly incomplete extracted evidence chains, utilizing hints that facilitate numerical comparison. The system achieves .281 FEVEROUS score and .658 label accuracy on the development set, and finishes in second place with .259 FEVEROUS score and .576 label accuracy on the test set.
Modeling Entity Knowledge for Fact Verification
Authors: Yang Liu, Chenguang Zhu, Michael Zeng
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Descriptive, Fact Verification, Retrieval

Fact verification is a challenging task of identifying the truthfulness of given claims based on the retrieval of relevant evidence texts. Many claims require understanding and reasoning over external entity information for precise verification. In this paper, we propose a novel fact verification model using entity knowledge to enhance its performance. We retrieve descriptive text from Wikipedia for each entity, and then encode these descriptions by a smaller lightweight network to be fed into the main verification model. Furthermore, we boost model performance by adopting and predicting the relatedness between the claim and each evidence as additional signals. We demonstrate experimentally on a large-scale benchmark dataset FEVER that our framework achieves competitive results with a FEVER score of 72.89% on the test set.
Verdict Inference with Claim and Retrieved Elements Using RoBERTa
Authors: In-Zu Gi, Ting-Yu Fang, Richard Tzong-Han Tsai
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Fact Verification, Retrieval

Automatic fact verification has attracted recent research attention as the increasing dissemination of disinformation on social media platforms. The FEVEROUS shared task introduces a benchmark for fact verification, in which a system is challenged to verify the given claim using the extracted evidential elements from Wikipedia documents. In this paper, we propose our 3rd place three-stage system consisting of document retrieval, element retrieval, and verdict inference for the FEVEROUS shared task. By considering the context relevance in the fact extraction and verification task, our system achieves 0.29 FEVEROUS score on the development set and 0.25 FEVEROUS score on the blind test set, both outperforming the FEVEROUS baseline.
Stance Detection in German News Articles
Authors: Laura Mascarell, Tatyana Ruzsics, Christian Schneebeli, Philippe Schlattner, Luca Campanella, Severin Klingler, Cristina Kadar
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Articles, Fake News Detection, Stance Detection

The widespread use of the Internet and the rapid dissemination of information poses the challenge of identifying the veracity of its content. Stance detection, which is the task of predicting the position of a text in regard to a specific target (e.g. claim or debate question), has been used to determine the veracity of information in tasks such as rumor classification and fake news detection. While most of the work and available datasets for stance detection address short texts snippets extracted from textual dialogues, social media platforms, or news headlines with a strong focus on the English language, there is a lack of resources targeting long texts in other languages. Our contribution in this paper is twofold. First, we present a German dataset of debate questions and news articles that is manually annotated for stance and emotion detection. Second, we leverage the dataset to tackle the supervised task of classifying the stance of a news article with regards to a debate question and provide baseline models as a reference for future work on stance detection in German news articles.
FANG-COVID: A New Large-Scale Benchmark Dataset for Fake News Detection in German
Authors: Justus Mattern, Yu Qiao, Elma Kerz, Daniel Wiechmann, Markus Strohmaier
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Articles, Fake News Detection

As the world continues to fight the COVID-19 pandemic, it is simultaneously fighting an ‘infodemic’ – a flood of disinformation and spread of conspiracy theories leading to health threats and the division of society. To combat this infodemic, there is an urgent need for benchmark datasets that can help researchers develop and evaluate models geared towards automatic detection of disinformation. While there are increasing efforts to create adequate, open-source benchmark datasets for English, comparable resources are virtually unavailable for German, leaving research for the German language lagging significantly behind. In this paper, we introduce the new benchmark dataset FANG-COVID consisting of 28,056 real and 13,186 fake German news articles related to the COVID-19 pandemic as well as data on their propagation on Twitter. Furthermore, we propose an explainable textual- and social context-based model for fake news detection, compare its performance to “black-box” models and perform feature ablation to assess the relative importance of human-interpretable features in distinguishing fake news from authentic news.
Combining sentence and table evidence to predict veracity of factual claims using TaPaS and RoBERTa
Authors: Martin Funkquist
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Retrieval, Sentence

This paper describes a method for retrieving evidence and predicting the veracity of factual claims, on the FEVEROUS dataset. The evidence consists of both sentences and table cells. The proposed method is part of the FEVER shared task. It uses similarity scores between TF-IDF vectors to retrieve the textual evidence and similarity scores between dense vectors created by fine-tuned TaPaS models for tabular evidence retrieval. The evidence is passed through a dense neural network to produce a veracity label. The FEVEROUS score for the proposed system is 0.126.
Automatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning
Authors: Aalok Sathe, Joonsuk Park
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Fact Checking, Misinformation, Multiple Instance Learning, Natural Language Inference, Sentence

Automatic fact-checking is crucial for recognizing misinformation spreading on the internet. Most existing fact-checkers break down the process into several subtasks, one of which determines candidate evidence sentences that can potentially support or refute the claim to be verified; typically, evidence sentences with gold-standard labels are needed for this. In a more realistic setting, however, such sentence-level annotations are not available. In this paper, we tackle the natural language inference (NLI) subtask—given a document and a (sentence) claim, determine whether the document supports or refutes the claim—only using document-level annotations. Using fine-tuned BERT and multiple instance learning, we achieve 81.9% accuracy, significantly outperforming the existing results on the WikiFactCheck-English dataset.
Neural Re-rankers for Evidence Retrieval in the FEVEROUS Task
Authors: Mohammed Saeed, Giulio Alfarano, Khai Nguyen, Duc Pham, Raphael Troncy, Paolo Papotti
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Fact Checking, Misinformation, Retrieval

Computational fact-checking has gained a lot of traction in the machine learning and natural language processing communities. A plethora of solutions have been developed, but methods which leverage both structured and unstructured information to detect misinformation are of particular relevance. In this paper, we tackle the FEVEROUS (Fact Extraction and VERification Over Unstructured and Structured information) challenge which consists of an open source baseline system together with a benchmark dataset containing 87,026 verified claims. We extend this baseline model by improving the evidence retrieval module yielding the best evidence F1 score among the competitors in the challenge leaderboard while obtaining an overall FEVEROUS score of 0.20 (5th best ranked system).
A Fact Checking and Verification System for FEVEROUS Using a Zero-Shot Learning Approach
Authors: Orkun Temiz, Özgün Ozan Kılıç, Arif Ozan Kızıldağ, Tuğba Taşkaya Temizel
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (FEVER) 2021 11
Tasks: Fact Checking, Natural Language Inference, Question Answering, Zero-Shot Learning

In this paper, we propose a novel fact checking and verification system to check claims against Wikipedia content. Our system retrieves relevant Wikipedia pages using Anserini, uses BERT-large-cased question answering model to select correct evidence, and verifies claims using XLNET natural language inference model by comparing it with the evidence. Table cell evidence is obtained through looking for entity-matching cell values and TAPAS table question answering model. The pipeline utilizes zero-shot capabilities of existing models and all the models used in the pipeline requires no additional training. Our system got a FEVEROUS score of 0.06 and a label accuracy of 0.39 in FEVEROUS challenge.
Measuring Alignment to Authoritarian State Media as Framing Bias
Authors: Timothy Niven, Hung-Yu Kao
Date: 2020-12-01 00:00:00
Proceeding: NLP4IF (COLING) 2020 12
Tasks: Selection bias

We introduce what is to the best of our knowledge a new task in natural language processing: measuring alignment to authoritarian state media. We operationalize alignment in terms of sociological definitions of media bias. We take as a case study the alignment of four Taiwanese media outlets to the Chinese Communist Party state media. We present the results of an initial investigation using the frequency of words in psychologically meaningful categories. Our findings suggest that the chosen word categories correlate with framing choices. We develop a calculation method that yields reasonable results for measuring alignment, agreeing well with the known labels. We confirm that our method does capture event selection bias, but whether it captures framing bias requires further investigation.
A Risk Communication Event Detection Model via Contrastive Learning
Authors: Mingi Shin, Sungwon Han, Sungkyu Park, Meeyoung Cha
Date: 2020-12-01 00:00:00
Proceeding: NLP4IF (COLING) 2020 12
Tasks: Contrastive Learning, Event Detection

This paper presents a time-topic cohesive model describing the communication patterns on the coronavirus pandemic from three Asian countries. The strength of our model is two-fold. First, it detects contextualized events based on topical and temporal information via contrastive learning. Second, it can be applied to multiple languages, enabling a comparison of risk communication across cultures. We present a case study and discuss future implications of the proposed model.
Using BERT for Qualitative Content Analysis in Psychosocial Online Counseling
Authors: Philipp Grandeit, Carolyn Haberkern, Maximiliane Lang, Jens Albrecht, Robert Lehmann
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (NLP+CSS) 2020 11
Tasks: BIG-bench Machine Learning, text-classification, Text Classification

Qualitative content analysis is a systematic method commonly used in the social sciences to analyze textual data from interviews or online discussions. However, this method usually requires high expertise and manual effort because human coders need to read, interpret, and manually annotate text passages. This is especially true if the system of categories used for annotation is complex and semantically rich. Therefore, qualitative content analysis could benefit greatly from automated coding. In this work, we investigate the usage of machine learning-based text classification models for automatic coding in the area of psycho-social online counseling. We developed a system of over 50 categories to analyze counseling conversations, labeled over 10.000 text passages manually, and evaluated the performance of different machine learning-based classifiers against human coders.
Swimming with the Tide? Positional Claim Detection across Political Text Types
Authors: Nico Blokker, Erenay Dayanik, Gabriella Lapesa, Sebastian Padó
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (NLP+CSS) 2020 11
Tasks: Articles

Manifestos are official documents of political parties, providing a comprehensive topical overview of the electoral programs. Voters, however, seldom read them and often prefer other channels, such as newspaper articles, to understand the party positions on various policy issues. The natural question to ask is how compatible these two formats (manifesto and newspaper reports) are in their representation of party positioning. We address this question with an approach that combines political science (manual annotation and analysis) and natural language processing (supervised claim identification) in a cross-text type setting: we train a classifier on annotated newspaper data and test its performance on manifestos. Our findings show a) strong performance for supervised classification even across text types and b) a substantive overlap between the two formats in terms of party positioning, with differences regarding the salience of specific issues.