pwc-archive/papers-with-abstracts huggingface dataset:
Word-Level Uncertainty Estimation for Black-Box Text Classifiers using RNNs
Authors: Jakob Smedegaard Andersen, Tom Sch{\"o}ner, Walid Maalej
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Decision Making, Sentiment Analysis

Estimating uncertainties of Neural Network predictions paves the way towards more reliable and trustful text classifications. However, common uncertainty estimation approaches remain as black-boxes without explaining which features have led to the uncertainty of a prediction. This hinders users from understanding the cause of unreliable model behaviour. We introduce an approach to decompose and visualize the uncertainty of text classifiers at the level of words. Our approach builds on top of Recurrent Neural Networks and Bayesian modelling in order to provide detailed explanations of uncertainties, enabling a deeper reasoning about unreliable model behaviours. We conduct a preliminary experiment to check the impact and correctness of our approach. By explaining and investigating the predictive uncertainties of a sentiment analysis task, we argue that our approach is able to provide a more profound understanding of artificial decision making.
Few-Shot Text Classification with Edge-Labeling Graph Neural Network-Based Prototypical Network
Authors: Chen Lyu, Weijie Liu, Ping Wang
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Few-Shot Text Classification, Graph Neural Network, Language Modeling, Language Modelling, Relation Classification, Sentiment Analysis, text-classification, Text Classification

In this paper, we propose a new few-shot text classification method. Compared with supervised learning methods which require a large corpus of labeled documents, our method aims to make it possible to classify unlabeled text with few labeled data. To achieve this goal, we take advantage of advanced pre-trained language model to extract the semantic features of each document. Furthermore, we utilize an edge-labeling graph neural network to implicitly models the intra-cluster similarity and the inter-cluster dissimilarity of the documents. Finally, we take the results of the graph neural network as the input of a prototypical network to classify the unlabeled texts. We verify the effectiveness of our method on a sentiment analysis dataset and a relation classification dataset and achieve the state-of-the-art performance on both tasks.
ManyEnt: A Dataset for Few-shot Entity Typing
Authors: Markus Eberts, Kevin Pech, Adrian Ulges
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Entity Typing

We introduce ManyEnt, a benchmark for entity typing models in few-shot scenarios. ManyEnt offers a rich typeset, with a fine-grain variant featuring 256 entity types and a coarse-grain one with 53 entity types. Both versions have been derived from the Wikidata knowledge graph in a semi-automatic fashion. We also report results for two baselines using BERT, reaching up to 70.68{\%} accuracy (10-way 1-shot).
Knowledge Base Embedding By Cooperative Knowledge Distillation
Authors: Rapha{\"e}l Sourty, Jose G. Moreno, Fran{\c{c}}ois-Paul Servant, Lynda Tamine-Lechani
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Knowledge Distillation, Representation Learning

Knowledge bases are increasingly exploited as gold standard data sources which benefit various knowledge-driven NLP tasks. In this paper, we explore a new research direction to perform knowledge base (KB) representation learning grounded with the recent theoretical framework of knowledge distillation over neural networks. Given a set of KBs, our proposed approach KD-MKB, learns KB embeddings by mutually and jointly distilling knowledge within a dynamic teacher-student setting. Experimental results on two standard datasets show that knowledge distillation between KBs through entity and relation inference is actually observed. We also show that cooperative learning significantly outperforms the two proposed baselines, namely traditional and sequential distillation.
IntKB: A Verifiable Interactive Framework for Knowledge Base Completion
Authors: Bernhard Kratzwald, Guo Kunpeng, Stefan Feuerriegel, Dennis Diefenbach
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Knowledge Base Completion, Question Answering

Knowledge bases (KBs) are essential for many downstream NLP tasks, yet their prime shortcoming is that they are often incomplete. State-of-the-art frameworks for KB completion often lack sufficient accuracy to work fully automated without human supervision. As a remedy, we propose : a novel interactive framework for KB completion from text based on a question answering pipeline. Our framework is tailored to the specific needs of a human-in-the-loop paradigm: (i) We generate facts that are aligned with text snippets and are thus immediately verifiable by humans. (ii) Our system is designed such that it continuously learns during the KB completion task and, therefore, significantly improves its performance upon initial zero- and few-shot relations over time. (iii) We only trigger human interactions when there is enough information for a correct prediction. Therefore, we train our system with negative examples and a fold-option if there is no answer. Our framework yields a favorable performance: it achieves a hit@1 ratio of 29.7{\%} for initially unseen relations, upon which it gradually improves to 46.2{\%}.
Flight of the PEGASUS? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization
Authors: Travis Goodwin, Max Savery, Dina Demner-Fushman
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Abstractive Text Summarization, Document Summarization, Few-Shot Learning, Multi-Document Summarization

Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks including automatic summarization. However, most work has focused on (relatively) data-rich single-document summarization settings. In this paper, we explore highly-abstractive multi-document summarization where the summary is explicitly conditioned on a user-given topic statement or question. We compare the summarization quality produced by three state-of-the-art transformer-based models: BART, T5, and PEGASUS. We report the performance on four challenging summarization datasets: three from the general domain and one from consumer health in both zero-shot and few-shot learning settings. While prior work has shown significant differences in performance for these models on standard summarization tasks, our results indicate that with as few as 10 labeled examples there is no statistically significant difference in summary quality, suggesting the need for more abstractive benchmark collections when determining state-of-the-art.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Authors: Haoran Li, Junnan Zhu, Jiajun Zhang, Xiaodong He, Chengqing Zong
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Decoder, Sentence, Sentence Summarization

This paper studies the problem of generating a summary for a given sentence-image pair. Existing multimodal sequence-to-sequence approaches mainly focus on enhancing the decoder by visual signals, while ignoring that the image can improve the ability of the encoder to identify highlights of a news event or a document. Thus, we propose a multimodal selective gate network that considers reciprocal relationships between textual and multi-level visual features, including global image descriptor, activation grids, and object proposals, to select highlights of the event when encoding the source sentence. In addition, we introduce a modality regularization to encourage the summary to capture the highlights embedded in the image more accurately. To verify the generalization of our model, we adopt the multimodal selective gate to the text-based decoder and multimodal-based decoder. Experimental results on a public multimodal sentence summarization dataset demonstrate the advantage of our models over baselines. Further analysis suggests that our proposed multimodal selective gate network can effectively select important information in the input sentence.
Controllable Abstractive Sentence Summarization with Guiding Entities
Authors: Changmeng Zheng, Yi Cai, Guanjie Zhang, Qing Li
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Informativeness, Sentence, Sentence Summarization, Text Summarization

Entities are the major proportion and build up the topic of text summaries. Although existing text summarization models can produce promising results of automatic metrics, for example, ROUGE, it is difficult to guarantee that an entity is contained in generated summaries. In this paper, we propose a controllable abstractive sentence summarization model which generates summaries with guiding entities. Instead of generating summaries from left to right, we start with a selected entity, generate the left part first, then the right part of a complete summary. Compared to previous entity-based text summarization models, our method can ensure that entities appear in final output summaries rather than generating the complete sentence with implicit entity and article representations. Our model can also generate more novel entities with them incorporated into outputs directly. To evaluate the informativeness of the proposed model, we develop a fine-grained informativeness metrics in the relevance, extraness and omission perspectives. We conduct experiments in two widely-used sentence summarization datasets and experimental results show that our model outperforms the state-of-the-art methods in both automatic evaluation scores and informativeness metrics.
HOLMS: Alternative Summary Evaluation with Large Language Models
Authors: Yassine Mrabet, Dina Demner-Fushman
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Document Summarization, Extractive Summarization

Efficient document summarization requires evaluation measures that can not only rank a set of systems based on an average score, but also highlight which individual summary is better than another. However, despite the very active research on summarization approaches, few works have proposed new evaluation measures in the recent years. The standard measures relied upon for the development of summarization systems are most often ROUGE and BLEU which, despite being efficient in overall system ranking, remain lexical in nature and have a limited potential when it comes to training neural networks. In this paper, we present a new hybrid evaluation measure for summarization, called HOLMS, that combines both language models pre-trained on large corpora and lexical similarity measures. Through several experiments, we show that HOLMS outperforms ROUGE and BLEU substantially in its correlation with human judgments on several extractive summarization datasets for both linguistic quality and pyramid scores.
An Anchor-Based Automatic Evaluation Metric for Document Summarization
Authors: Kexiang Wang, Tianyu Liu, Baobao Chang, Zhifang Sui
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Document Summarization

The widespread adoption of reference-based automatic evaluation metrics such as ROUGE has promoted the development of document summarization. In this paper, we consider a new protocol for designing reference-based metrics that require the endorsement of source document(s). Following protocol, we propose an anchored ROUGE metric fixing each summary particle on source document, which bases the computation on more solid ground. Empirical results on benchmark datasets validate that source document helps to induce a higher correlation with human judgments for ROUGE metric. Being self-explanatory and easy-to-implement, the protocol can naturally foster various effective designs of reference-based metrics besides the anchored ROUGE introduced here.
On the Faithfulness for E-commerce Product Summarization
Authors: Peng Yuan, Haoran Li, Song Xu, Youzheng Wu, Xiaodong He, BoWen Zhou
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Attribute

In this work, we present a model to generate e-commerce product summaries. The consistency between the generated summary and the product attributes is an essential criterion for the ecommerce product summarization task. To enhance the consistency, first, we encode the product attribute table to guide the process of summary generation. Second, we identify the attribute words from the vocabulary, and we constrain these attribute words can be presented in the summaries only through copying from the source, i.e., the attribute words not in the source cannot be generated. We construct a Chinese e-commerce product summarization dataset, and the experimental results on this dataset demonstrate that our models significantly improve the faithfulness.
SumTitles: a Summarization Dataset with Low Extractiveness
Authors: Valentin Malykh, Konstantin Chernis, Ekaterina Artemova, Irina Piontkovskaya
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Abstractive Text Summarization, Text Summarization

The existing dialogue summarization corpora are significantly extractive. We introduce a methodology for dataset extractiveness evaluation and present a new low-extractive corpus of movie dialogues for abstractive text summarization along with baseline evaluation. The corpus contains 153k dialogues and consists of three parts: 1) automatically aligned subtitles, 2) automatically aligned scenes from scripts, and 3) manually aligned scenes from scripts. We also present an alignment algorithm which we use to construct the corpus.
TWEETSUM: Event oriented Social Summarization Dataset
Authors: Ruifang He, Liangliang Zhao, Huanyu Liu
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Extractive Summarization, Relation Network

With social media becoming popular, a vast of short and noisy messages are produced by millions of users when a hot event happens. Developing social summarization systems becomes more and more critical for people to quickly grasp core and essential information. However, the publicly available and high-quality large scale social summarization dataset is rare. Constructing such corpus is not easy and very expensive since short texts have very complex social characteristics. In this paper, we construct TWEETSUM, a new event-oriented dataset for social summarization. The original data is collected from twitter and contains 12 real world hot events with a total of 44,034 tweets and 11,240 users. Each event has four expert summaries, and we also have the annotation quality evaluation. In addition, we collect additional social signals (i.e. user relations, hashtags and user profiles) and further establish user relation network for each event. Besides the detailed dataset description, we show the performance of several typical extractive summarization methods on TWEETSUM to establish baselines. For further researches, we will release this dataset to the public.
Exploiting a lexical resource for discourse connective disambiguation in German
Authors: Peter Bourgonje, Manfred Stede
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Classification, Discourse Parsing

In this paper we focus on connective identification and sense classification for explicit discourse relations in German, as two individual sub-tasks of the overarching Shallow Discourse Parsing task. We successively augment a purely-empirical approach based on contextualised embeddings with linguistic knowledge encoded in a connective lexicon. In this way, we improve over published results for connective identification, achieving a final F1-score of 87.93; and we introduce, to the best of our knowledge, first results for German sense classification, achieving an F1-score of 87.13. Our approach demonstrates that a connective lexicon can be a valuable resource for those languages that do not have a large PDTB-style-annotated coprus available.
Chinese Paragraph-level Discourse Parsing with Global Backward and Local Reverse Reading
Authors: Feng Jiang, Xiaomin Chu, Peifeng Li, Fang Kong, Qiaoming Zhu
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Discourse Parsing, Position

Discourse structure tree construction is the fundamental task of discourse parsing and most previous work focused on English. Due to the cultural and linguistic differences, existing successful methods on English discourse parsing cannot be transformed into Chinese directly, especially in paragraph level suffering from longer discourse units and fewer explicit connectives. To alleviate the above issues, we propose two reading modes, i.e., the global backward reading and the local reverse reading, to construct Chinese paragraph level discourse trees. The former processes discourse units from the end to the beginning in a document to utilize the left-branching bias of discourse structure in Chinese, while the latter reverses the position of paragraphs in a discourse unit to enhance the differentiation of coherence between adjacent discourse units. The experimental results on Chinese MCDTB demonstrate that our model outperforms all strong baselines.
A Neural Model for Aggregating Coreference Annotation in Crowdsourcing
Authors: Maolin Li, Hiroya Takamura, Sophia Ananiadou
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: coreference-resolution, Coreference Resolution

Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. Collecting sufficient labelled data from expert annotators to train a high-performance coreference resolution system is time-consuming and expensive. Crowdsourcing makes it possible to obtain the required amounts of data rapidly and cost-effectively. However, crowd-sourced labels can be noisy. To ensure high-quality data, it is crucial to infer the correct labels by aggregating the noisy labels. In this paper, we split the aggregation into two subtasks, i.e, mention classification and coreference chain inference. Firstly, we predict the general class of each mention using an autoencoder, which incorporates contextual information about each mention, while at the same time taking into account the mention{'}s annotation complexity and annotators{'} reliability at different levels. Secondly, to determine the coreference chain of each mention, we use weighted voting which takes into account the learned reliability in the first subtask. Experimental results demonstrate the effectiveness of our method in predicting the correct labels. We also illustrate our model{'}s interpretability through a comprehensive analysis of experimental results.
Variation in Coreference Strategies across Genres and Production Media
Authors: Berfin Akta{\c{s}}, Manfred Stede
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: coreference-resolution, Coreference Resolution

In response to (i) inconclusive results in the literature as to the properties of coreference chains in written versus spoken language, and (ii) a general lack of work on automatic coreference resolution on both spoken language and social media, we undertake a corpus study involving the various genre sections of Ontonotes, the Switchboard corpus, and a corpus of Twitter conversations. Using a set of measures that previously have been applied individually to different data sets, we find fairly clear patterns of {``}behavior{''} for the different genres/media. Besides their role for psycholinguistic investigation (why do we employ different coreference strategies when we write or speak) and for the placement of Twitter in the spoken{--}written continuum, we see our results as a contribution to approaching genre-/media-specific coreference resolution.
Towards automatically generating Questions under Discussion to link information and discourse structure
Authors: Kordula De Kuthy, Madeeswaran Kannan, Haemanth Santhi Ponnusamy, Detmar Meurers
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Question Generation, Question-Generation, Sentence

Questions under Discussion (QUD; Roberts, 2012) are emerging as a conceptually fruitful approach to spelling out the connection between the information structure of a sentence and the nature of the discourse in which the sentence can function. To make this approach useful for analyzing authentic data, Riester, Brunetti {\&} De Kuthy (2018) presented a discourse annotation framework based on explicit pragmatic principles for determining a QUD for every assertion in a text. De Kuthy et al. (2018) demonstrate that this supports more reliable discourse structure annotation, and Ziai and Meurers (2018) show that based on explicit questions, automatic focus annotation becomes feasible. But both approaches are based on manually specified questions. In this paper, we present an automatic question generation approach to partially automate QUD annotation by generating all potentially relevant questions for a given sentence. While transformation rules can concisely capture the typical question formation process, a rule-based approach is not sufficiently robust for authentic data. We therefore employ the transformation rules to generate a large set of sentence-question-answer triples and train a neural question generation model on them to obtain both systematic question type coverage and robustness.
Semi-supervised Multi-task Learning for Multi-label Fine-grained Sexism Classification
Authors: Harika Abburi, Pulkit Parikh, Niyati Chhaya, Vasudeva Varma
Date: 2020-12-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Multi-Task Learning

Sexism, a form of oppression based on one{'}s sex, manifests itself in numerous ways and causes enormous suffering. In view of the growing number of experiences of sexism reported online, categorizing these recollections automatically can assist the fight against sexism, as it can facilitate effective analyses by gender studies researchers and government officials involved in policy making. In this paper, we investigate the fine-grained, multi-label classification of accounts (reports) of sexism. To the best of our knowledge, we work with considerably more categories of sexism than any published work through our 23-class problem formulation. Moreover, we propose a multi-task approach for fine-grained multi-label sexism classification that leverages several supporting tasks without incurring any manual labeling cost. Unlabeled accounts of sexism are utilized through unsupervised learning to help construct our multi-task setup. We also devise objective functions that exploit label correlations in the training data explicitly. Multiple proposed methods outperform the state-of-the-art for multi-label sexism classification on a recently released dataset across five standard metrics.