pwc-archive/papers-with-abstracts huggingface dataset:
Hey AI, Can You Solve Complex Tasks by Talking to Agents?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Training giant models from scratch for each complex task is resource- and data-inefficient. To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language. We design a synthetic benchmark, CommaQA, with three complex reasoning tasks (explicit, implicit, numeric) designed to be solved by communicating with existing QA agents. For instance, using text and table QA agents to answer questions such as "Who had the longest javelin throw from USA?". We show that black-box models struggle to learn this task from scratch (accuracy under 50\%) even with access to each agent's knowledge and gold facts supervision. In contrast, models that learn to communicate with agents outperform black-box models, reaching scores of 100\% when given gold decomposition supervision. However, we show that the challenge of learning to solve complex tasks by communicating with existing agents \emph{without relying on any auxiliary supervision or data} still remains highly elusive. We will release CommaQA, along with a compositional generalization test split, to advance research in this direction.
Delving Deep into Extractive Question Answering Data
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Extractive Question-Answering, Question Answering, Vocal Bursts Type Prediction

The impact of large-scale pre-trained language models on Question Answering in recent times is undeniably positive.
Few prior works have attempted however to provide detailed insight into  how such models learn from QA dataset component parts.
For example, what specific kinds of examples are most important for models to learn from? In this paper, we examine two English QA datasets, namely SQuAD1.1 and NewsQA, and report findings on the internal characteristics of these widely employed extractive QA datasets. Experiment results reveal: (i) Models learn relatively independently of examples from outside a given question type (the performance on each question type mainly comes from that data belonging to that same question type); (ii) Increased difficulty in the training data results in better performance; (iii) Learning from QA data approximates to the process of learning question-answer matches.
Maximum Proxy-Likelihood Estimation for Non-autoregressive Machine Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Translation

Maximum Likelihood Estimation (MLE) is commonly used in machine translation, where models with higher likelihood are assumed to perform better in translation. However, this assumption does not hold in the non-autoregressive Transformers (NATs), a new family of translation models. In this paper, we present both theoretical and empirical analysis on why simply maximizing the likelihood does not produce a good NAT model. Based on the theoretical analysis, we propose Maximum Proxy-Likelihood Estimation (MPLE), a novel method to address the training issue in MLE. Additionally, MPLE provides a novel perspective to understand existing success in training NATs, namely much previous work can be regarded as implicitly optimizing our objective.
Towards Job-Transition-Tag Graph for a Better Job Title Representation Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Representation Learning, TAG

Works on learning job title representation are mainly based on Job-Transition Graph, built from the working history of talents. However, since the records are usually messy, this graph is very sparse, which affects the quality of the learned representation and hinders further analysis. To address this specific issue, we propose to enrich the graph with additional nodes that improve the quality of job title representation. Specifically, we construct Job-Transition-Tag Graph, a heterogeneous graph containing two types of nodes, i.e., job titles and tags (i.e., words related to job responsibilities or functions). Along this line, we reformulate job title representation learning as the task of learning node embedding on the Job-Transition-Tag Graph. Experiments on a public CareerBuilder12 dataset and a private Randstad dataset show interest of our approach.
How does the pre-training objective affect what large language models learn about linguistic properties?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our knowledge, no previous work so far has investigated how different pre-training objectives affect what BERT learns about linguistics properties. We hypothesize that linguistically motivated objectives (e.g. MLM) should help BERT to acquire better linguistic knowledge compared to using non-linguistically motivated objectives, i.e. hard for humans to guess the association between the input and the label to be predicted. To this end, we pre-train BERT with two linguistically motivated objectives and three non-linguistically motivated ones. We then probe for linguistic characteristics encoded in the representation of the resulting models. We find strong evidence that there is no actual differences in probing performance between the representations learned by the two different types of objectives. These surprising results question the dominant narrative of linguistically informed pre-training.
A Weak Self-supervision with Transition-Based Modeling for Reference Resolution
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Entity Resolution

The reference resolution is a task to find the link between an entity and its source action in the same recipe. In this study, we introduce a weak self-supervision method with a transition-based model for reference resolution tasks for recipes, where the aim of the task is to make the syntax of the instructions used for reference resolution with self annotation. The results show that our approach to the problem outperforms the previous unsupervised methods with %8 F1. Especially, our models show %82 accuracies of pronoun, and %85 accuracies for null entity resolution.
REDTABS: A Collection of Report Document Datasets for Long Text and Multi-Table Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Document Summarization, Informativeness, Text Generation

Automatic document summarization aims to produce a concise summary covering the input document's salient content. 
Within a report document, both the textual and non-textual content (e.g., tables and figures) can be important information sources for the summary. However, most available document summarization datasets focus on the text and filter out the non-textual content. Missing tabular data can limit the informativeness of produced summaries, especially when target summaries require to cover quantitative descriptions of critical metrics, whose numerical information is usually kept in tables. In this paper, we address this issue by introducing REDTABS, the first collection of large-scale datasets for long text and multi-table summarization. Built on companies' annual reports, it includes three large-scale datasets for summarizing these companies' business, results of operations, and overall conditions, respectively. We also present the Segment-Alignment-based long Text and multi-Table summarization (SATT) method incorporating textual and tabular data into the summarization process. Besides, we propose a set of automatic evaluation metrics to assess the numerical information in summaries produced by summarization models. Dataset analyses and experimental results reveal the importance of incorporating textual and tabular data into the report document summarization. We will release our data and code to facilitate advances in summarization and text generation research.
Learning Functional Distributional Semantics with Visual Data
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Language Acquisition

Functional Distributional Semantics is a recently proposed framework for learning distributional semantics that provides linguistic interpretability. It models the meaning of a word as a binary classifier rather than a numerical vector. In this work, we propose a method to train a Functional Distributional Semantics model with grounded visual data. We train it on the Visual Genome dataset, which is closer to the kind of data encountered in human language acquisition than a large text corpus. On four external evaluation datasets, our model outperforms previous work on learning semantics from Visual Genome.
Models can use keywords to answer questions that human cannot
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Reading Comprehension, Reading Comprehension

Recent studies raised that reading comprehension (RC) models learn to exploit biases and annotation artifacts in current Machine Reading Comprehension (MRC) datasets to achieve impressive performance. This hinders the community from measuring sophisticated understanding of RC systems. MRC questions whose answers can be rightly predicted without understanding their contexts are defined as biased ones. Previous researches aimed to split unintended biases and determine their influence have some limitations. Some methods using partial test data to extract biases lack holistic consideration with question-context-option tuple. Others relied on artificial statistical features are limited by question types. 
In this paper, we employ two simple heuristics to identify biased questions in current MRC datasets through human-annotated keywords. We implement three neural networks on the biased data and find that they have outstanding abilities to capture the biases, and further study the superficial features of the biased data exploited by models as shortcuts in views of lexical choice and paragraphs. Experiments show that (i) models can answer some questions merely using several keywords which are unanswerable or difficulty for human. (ii) lexical choice preference in options creates biases utilized by models. (iii) fewer paragraphs are more likely to introduce biases in MRC datasets.
Generative Prompt Tuning for Relation Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Language Modeling, Language Modelling, Masked Language Modeling, Relation, Relation Classification, Text Infilling

Prompt tuning is proposed to better tune pre-trained language models by filling the objective gap between the pre-training process and the downstream tasks. Current methods mainly convert the downstream tasks into masked language modeling (MLM) problems, which have proven effective for tasks with simple label sets. However, when applied to relation classification tasks which often exhibit a complex label space, vanilla prompt tuning methods designed for MLM may struggle with handling complex label verbalizations with variable length as in such methods, the locations and number of masked tokens are typically fixed. Inspired by the text infilling task for pre-training generative models that can flexibly predict missing spans, we propose a novel generative prompt tuning method to reformulate relation classification as an infilling problem to eliminate the rigid prompt restrictions, which allows our method to process label verbalizations of varying lengths at multiple predicted positions and thus be able to fully leverage rich semantics of entity and relation labels. In addition, we design entity-guided decoding and discriminative relation scoring to predict relations effectively and efficiently in the inference process. Extensive experiments under low-resource settings and fully supervised settings demonstrate the effectiveness of our approach.
Generating Diverse and High-Quality Abstractive Summaries with Variational Transformers
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Abstractive Text Summarization, Decoder, Diversity, Informativeness, Vocal Bursts Intensity Prediction

Existing works on abstractive summarization mainly focus on boosting summarization's quality (informativeness, contextual similarity). To generate summaries of both high diversity and quality, we proposes the Transformer+CVAE model, which integrates the CVAE framework into the Transformer by introducing the prior/recognition networks that bridges the Transformer encoder and decoder. We utilize the latent variables generated in the global receptive field of the transformer by fusing them to the starting-of-sequence ([SOS]) of the decoder inputs. To better tune the weights of the latent variables in the sequence, we designed a gated unit to blend the latent representation and the [SOS] token.  Evaluated on the Gigaword dataset, our model outperforms the state-of-the-art seq-to-seq models and the base Transformer in diversity and quality metrics. After scrutinizing the pre-training and the gating mechanism we apply, we discover that both schemes help improve the quality of generated summaries in the CVAE framework.
Empirical Analysis of Training Strategies of Transformer-based Japanese Chit-chat Systems
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder

In recent years, several high-performance conversational systems have been proposed based on the Transformer encoder-decoder model.
Although previous studies analyzed the effects of the model parameters and the decoding method on subjective dialogue evaluations with overall metrics, they did not analyze how the differences of fine-tuning datasets affect on user's detailed impression. 
In addition, the Transformer-based approach has only been verified for English, not for such languages with large inter-language distances as Japanese.
In this study, we develop large-scale Transformer-based Japanese dialogue models and Japanese chit-chat datasets to examine the effectiveness of the Transformer-based approach for building chit-chat dialogue systems.
We evaluated and analyzed the impressions of human dialogues in different fine-tuning datasets, model parameters, and the use of additional information.
Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, Response Generation, Sentence

Targeted-guided response generation enables dialogue systems to smoothly guide a conversation from a dialogue context toward a target sentence. Such control is useful for designing dialogue systems that direct a conversation toward specific goals, such as providing counselling and creating non-obtrusive  recommendations. In this paper, we introduce a new technique for target-guided response generation, which first finds a bridging path of commonsense knowledge concepts between the source and target, and then uses the identified bridging path to generate transition responses. Additionally, we propose techniques to re-purpose existing dialogue datasets for target-guided generation. Finally, we demonstrate the shortcomings of existing automated metrics for this task, and propose a novel evaluation metric that we show is more effective for target-guided response evaluation. Our experiments show that our proposed evaluation metric is reliable and our techniques outperform baselines on the generation task. Our work generally enables dialog system designers to exercise more control over the conversations that their systems produce.
SlotGAN: Detecting Mentions in Text via Adversarial Distant Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence, valid

We present SlotGAN, a framework for training a mention detection model that only requires unlabeled text and a gazetteer. It consists of a generator trained to extract spans from an input sentence, and a discriminator trained to determine whether a span comes from the generator, or from the gazetteer.
We evaluate the method on English newswire data and compare it against supervised, weakly-supervised, and unsupervised methods. We find that the performance of the method is lower than these baselines, because it tends to generate more and longer spans, and in some cases it relies only on capitalization. In other cases, it generates spans that are valid but differ from the benchmark. When evaluated with metrics based on overlap, we find that SlotGAN performs within 95% of the precision of a supervised method, and 84% of its recall. Our results suggest that the model can generate spans that overlap well, but an additional filtering mechanism is required.
ReadE: Learning Relation-Dependent Entity Representation for Knowledge Graph Completion
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Graph Embedding, Graph Neural Network, Knowledge Graph Completion, Knowledge Graph Embedding, Relation

Conventional knowledge graph embedding methods learn semantic representations for entities considering their intrinsic interactions through powerful graph neural networks. However, previous methods represent each node solely with a coarse-grained unique representation, regardless of the variance of emphasis of entity semantics by different relations. To tackle this problem, we propose ReadE, a method to learn relation-dependent entity representations of which the semantic information is emphasized by varied relations types. First, we propose a relation-controlled gating mechanism targeting on utilizing the relation to control the information flow in the aggregation step of the graph neural network. Second, we propose a contrastive learning method with mixing both relation-level and entity-level negative samples to enhance semantics preserved in relation-dependent entity representations. Experiments on three benchmarks show that our proposed model outperforms all strong baselines. The code will be made open-sourced on Github.
Evidence Decomposition Graph Network for Fact Verification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Fact Verification, Sentence

Fact verification is the task to verify a given claim according to extracted evidence sentences. 
Most existing works use whole evidence sentences or break them into phrases to perform evidence interaction, where evidence is treated either too coarsely or over fragmented. We also find that many models suffer from exposure bias, which finally leads to them only paying attention to the evidence ranked higher by previous steps while failing to recognize crucial pieces from all candidates. In this paper, we propose an Evidence Decomposition Graph Network (EDGN), which decomposes each evidence sentence, especially the complex ones, into several simple sentences, highlighting the required key information without losing sentence structure and meaning. EDGN also absorbs a simple but effective evidence shuffling method to mitigate exposure bias. Experiments on the FEVER benchmark show our model can take all evidence candidates into account, distill necessary key information from complex evidence, and outperform existing methods in the literature. We will release our code to the community for further exploration.
Temporal Language Modeling for Short Text Document Classification with Transformers
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Document Classification, Language Modeling, Language Modelling, Masked Language Modeling

Language models are typically trained on solely text data, not utilizing documents timestamp, which is available in most internet corpora. In this paper, we examine the impact of incorporating timestamp into transformer language model in terms of downstream classification task and masked language modeling on 2 short texts corpora. We examine different timestamp components: day of the month, month, year, weekday. We test different methods of incorporating date into the model: prefixing date components into text input and adding trained date embeddings. Our study shows, that such a temporal language model performs better than a regular language model for both documents from training data time span and unseen time span. That holds true for classification and language modeling. Prefixing date components into text performs not worse than training special date components embeddings.
Domain-aware Self-supervised Pre-training for Weakly-supervised Meme Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Representation Learning, Self-Supervised Learning

Existing self-supervised learning strategies are constrained to a limited set of trivial and generic downstream tasks that predominantly target uni-modal applications. This has isolated progress for imperative multi-modal applications that are diverse in terms of complexity and domain-affinity such as meme analysis.  Here, we introduce two self-supervised pre-training strategies, namely Ext-PIE-Net and MM-SimCLR that (i) employ multi-modal hate-speech data during pre-training,  and (ii) extend existing self-supervision learning approaches by incorporating multiple specialized pretext tasks; effectively catering to the required complex multi-modal representation learning for meme analysis. 
We experiment with different self-supervision strategies, including potential variants that could help learn rich cross-modality representations and evaluate using popular linear probing on the Hateful Memes task. The proposed solutions strongly compete with the fully-supervised baseline in a weakly-supervised setting, while distinctly outperforming them on all three tasks of the Memotion challenge with 0.18\%, 23.64\%, and 0.93\% performance gain, respectively. Further we demonstrate generalizability of the proposed solutions by reporting competitive performance on the HarMeme task. Finally, we empirically establish efficient convergence of the proposed solutions during fine-tuning, in a weak-supervision setup by arguing that the complexity of the self-supervision strategy and downstream task at hand are correlated. Our efforts highlight the requirement of better self-supervision strategies involving specialised pretext tasks for efficient fine-tuning and generalizable performance.
On the Robustness of Reading Comprehension Models to Entity Renaming
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Continual Pretraining, Machine Reading Comprehension, Reading Comprehension

We study the robustness of machine reading comprehension (MRC) models to entity renaming---do models make more wrong predictions when answer entities have different names? Such failures imply that models overly rely on entity information to answer questions, and thus may generalize poorly when facts about the world change or questions are asked about novel entities. To systematically audit this issue, we present a general and scalable pipeline to replace entity names with names from a variety of sources, ranging from common English names to names from other languages to arbitrary strings. Across five datasets and three pretrained model architectures, MRC models consistently perform worse when entities are renamed, with particularly large accuracy drops on datasets constructed via distant supervision. We also find large differences between models: SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa, despite having similar accuracy on unperturbed test data. We further experiment with different masking strategies as the continual pretraining objective and find that entity-based masking can improve the robustness of MRC models.