pwc-archive/papers-with-abstracts huggingface dataset:
Hierarchical Attention Decoder for Solving Math Word Problems
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Math

To answer math word problems (MWPs), models need to formalize equations from the source text of math problems. Recently, the tree-structured decoder has significantly improved model performance on this task by generating the target equation in a tree format. However, current decoders usually ignore the hierarchical relationships between tree nodes and their parents, which hinders further improvement. Thus, we propose a structure called hierarchical attention tree to aid the generation procedure of the decoder. As our decoder follows a graph-based encoder, our full model is therefore named as Graph to Hierarchical Attention Tree (G2HAT). We show a tree-structured decoder with hierarchical accumulative multi-head attention leads to significant performance improvement and reaches new state-of-the-art (SOTA) on both English MAWPS and Chinese Math23k MWP benchmarks. For further study, we also apply pre-trained language models for G2HAT, which even results in new higher performance.
Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Generation, Question-Generation, Reading Comprehension

Reading is integral to everyday life, and yet learning to read is a struggle for many young learners.  During lessons, teachers can use comprehension questions to increase engagement, test reading skills, and improve retention. Historically such questions were written by skilled teachers, but recently language models have been used to generate comprehension questions. However, many existing Question Generation (QG) systems focus on generating extractive questions from the text, and have no way to control the type of the generated question. In this paper, we study QG for reading comprehension where inferential questions are critical and extractive techniques cannot be used. We propose a two-step model (HTA-WTA) that takes advantage of previous datasets, and can generate questions for a specific targeted comprehension skill. We propose a new reading comprehension dataset that contains questions annotated with story-based reading comprehension skills (SBRCS), allowing for a more complete reader assessment. Across several experiments, our results show that HTA-WTA outperforms multiple strong baselines on this new dataset. We show that the HTA-WTA model tests for strong SCRS by asking deep inferential questions.
Evaluating Extreme Hierarchical Multi-label Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Hierarchical Multi-label Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

Several natural language processing (NLP) tasks are defined as a classification problem in its most complex form: Multi-label Hierarchical Extreme classification, in which items may be associated with multiple classes from a set of thousands of possible classes organized in a hierarchy and with a highly unbalanced distribution both in terms of class frequency and the number of labels per item. We analyze the state of the art of evaluation metrics based on a set of formal properties and we define an information theoretic based metric inspired by the Information Contrast Model (ICM). Experiments on synthetic data and a case study on real data show the suitability of the ICM for such  scenarios.
Rethinking Negative Sampling for Handling Missing Entity Annotations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Sentence

Negative sampling is highly effective in handling missing annotations for named entity recognition (NER). One of our contributions is an analysis on how it makes sense through introducing two insightful concepts: missampling and uncertainty. Empirical studies show low missampling rate and high uncertainty are both essential for achieving promising performances with negative sampling. Based on the sparsity of named entities, we also theoretically derive a lower bound for the probability of zero missampling rate, which is only relevant to sentence length. The other contribution is an adaptive and weighted sampling distribution that further improves negative sampling via our former analysis. Experiments on synthetic datasets and well-annotated datasets (e.g., CoNLL-2003) show that our proposed approach benefits negative sampling in terms of F1 score and loss convergence. Besides, models with improved negative sampling have achieved new state-of-the-art results on real-world datasets (e.g., EC).
Inducing Global and Local Knowledge Attention in Multi-turn Dialog Understanding
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Response Generation, Spoken Language Understanding

In multi-turn dialog understanding, semantic frames are constructed by detecting intents and slots within each user utterance. However, recent works lack the capability of modeling multi-turn dynamics within a dialog where the contexts are mostly adopted for updating dialog states instead of capturing overall intent semantic flows in spoken language understanding (SLU). Moreover, external knowledge related to dialogs may be beneficial in exploring deep semantic information across dialog turns, which many works only considered for end-to-end response generation. In this paper, we propose to equip a BERT-based joint framework with a context attention module and a knowledge attention module to introduce knowledge attention with contexts between two SLU tasks. We propose three attention mechanisms to induce both global and local attention on knowledge triples. Experimental results in two complicated multi-turn dialog datasets have demonstrated significant improvements of our proposed framework by mutually modeling two SLU tasks with filtered knowledge and dialog contexts. Attention visualization also provides nice interpretability of how our modules leverage knowledge across the utterance.
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Computational Efficiency, Fairness

The first NLP experiment many researchers performed in their careers likely involved training a standard architecture on labeled English data and optimizing for accuracy, without accounting for other dimensions such as fairness, interpretability, or computational efficiency. We show through surveys that this is indeed the case and refer to it as the square one experimental setup. NLP research often goes beyond the square one setup, e.g, focusing not only on accuracy, but also on fairness or interpretability, but typically along a single dimension. Most work focused on multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on. We show this through manual classification of recent NLP research papers and ACL Test-of-Time award recipients. Such one-dimensionality of most research means we are only exploring a fraction of the NLP research search space. We provide historical and recent examples of how the square one bias has led researchers to draw false conclusions or make unwise choices, point to promising yet unexplored directions on the research manifold, and make practical recommendations to enable more multi-dimensional research.
RepAL: A Simple and Plug-and-play Method for Improving Unsupervised Sentence Representations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Representation Learning, Sentence, Sentence Embedding, Sentence-Embedding

Unsupervised sentence representation learning is a fundamental problem in natural language processing and has been studied extensively in recent years. This paper presents Representation ALchemy (RepAL), an extremely simple post-processing method that enhances unsupervised sentence representations. The basic idea in RepAL is to extract redundant information from the representation of a sentence generated by the existing models and then refine the representation through an embedding refinement operation to filter such redundant information. In this paper, we analyze the redundant information from two levels: sentence-level and corpus-level, and the theoretical analysis for the latter is also conducted. We point out that RepAL is free of training and is a plug-and-play method that can be combined with most existing unsupervised sentence learning models. Extensive experiments demonstrate RepAL’s effectiveness and show that RepAL is a model-agnostic method for unsupervised sentence embedding enhancement. Besides, we also designed detailed ablation studies to understand why RepAL works and provided in-depth analysis and understanding of the redundant information.
Increasing Entity Linking upper bound through a more effective Candidate Generation System
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Entity Linking

Entity Linking (EL) aligns entity mentions in text to entries in a knowledge base. It usually comprises of two phases: candidate generation and candidate ranking. While most methods focus on the latter phase, it is candidate generation that sets the upper bound for both time and accuracy of an EL system. We propose a simple approach for improving candidate generation by efficiently embedding mention-entity pairs in dense space through a BERT-based bi-encoder. Specifically, we introduce a new pooling function and incorporate entity type side-information. We achieve a new state-of-the-art 84.28% recall of the gold entity in the Zero-shot EL dataset with just 50 candidates, compared to the previous 82.06% with 64 candidates. We report the results from extensive experimentation using our proposed model on both seen and unseen entity datasets. Our results suggest that our approach could be a useful complement to existing EL methods.
Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Clustering, Sentence, Sentence Embeddings, Topic Models, Word Embeddings

Recent work incorporates pre-trained word embeddings such as BERT embeddings into Neural Topic Models (NTMs), generating highly coherent topics. However, with high-quality contextualized document representations, do we really need sophisticated neural models to obtain coherent and interpretable topics? In this paper, we conduct thorough experiments showing that directly clustering high-quality sentence embeddings with an appropriate word selecting method can generate more coherent and diverse topics than NTMs, achieving also higher efficiency and simplicity.
Adversarial multi-task learning to solve holistic semantic polishing problem of Chinese text
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Multi-Task Learning, Sentence

For the core obstacles faced by Chinese proofreading, we think that the effective holistic semantic feature representation of a sentence and its characteristics is a key. Here, we propose a novel adversarial multi-task learning framework to realize above point. Wherein, sentence scoring and word prediction tasks are considered as not only a group of multi-tasks, but also a couple of generative adversarial relationship. Moreover, a policy network is introduced to achieve text polishing based on above adversarial multi-task model and Monte Carlo search training strategy. The ablation experiment results on Xuexi and CLUE corpus show that, for text scoring and prediction tasks, adversarial multi-task learning achieves a 10-20% improvement in accuracy and F1-score compared to the baseline. And, in text correction task, our method is significantly better than the baselines.
A Simple Overlapping Relation Extraction Method Based on Dropout
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Relation, Relation Extraction

The relation extraction (RE) task of knowledge acquisition aims to identify all specific relations of two entities. Prior works had proved the effectiveness of tagging methods, especially on the overlapping triple extraction problem. In this work, we introduce a simple relation extraction method (DropRel) based on dropout. We propose a Dropout-Normalization layer to generate different vectors for one token pair. Our model also provides a unified perspective for entity boundary and relation tagging tasks. We define a novel dependence relation for relation triple to integrate boundary into relation. The empirical experiments show that the DropRel model outperforms previous state-of-the-art methods on NYT and WEBNLG datasets. And our ablation study shows the DropRel without the dependence relation is better on the WEBNLG dataset.
Low rank softmax can have unargmaxable classes in theory but rarely in practice
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Translation

Classifiers in natural language processing (NLP) often have a large number of output classes. For example, neural language models (LMs) and machine translation (MT) models both predict tokens from a vocabulary of thousands. The softmax output layer of these models typically receives as input a dense feature representation, which has much lower dimensionality than the output. In theory, the result is some words may be impossible to predict via argmax, irrespective of input features, and empirically, this has been shown to happen in small language models (Demeter et al., 2020). In this paper we ask whether it can happen in practical large language models and translation models. To do so, we develop algorithms to detect such unargmaxable tokens in public models. We find that that 13 out of 150 models do indeed have such tokens; however, they are very infrequent and unlikely to impact model quality. We release our algorithms and code to the public.
Few-shot Named Entity Recognition with Joint Token and Sentence Awareness
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Few-shot NER, Named Entity Recognition, Named Entity Recognition (NER), Semantic Similarity, Semantic Textual Similarity, Sentence

Few-shot learning has been proposed and rapidly emerging as a viable means for completing various tasks. Recently, few-shot models have been used for Named Entity Recognition (NER). Prototypical network shows high efficiency on few-shot NER. However, existing prototypical methods only consider the similarity of tokens in query sets and support sets and ignore the semantic similarity among the sentences which contain these entities. We present a novel model, Few-shot Named Entity Recognition with  \textbf{J}oint \textbf{T}oken and \textbf{S}entence \textbf{A}wareness (\textbf{JTSA}), to address the issue.  The sentence awareness is introduced to probe the semantic similarity among the sentences. The Token awareness is used to explore the similarity of the tokens. To further improve the robustness and results of the model, we adopt the joint learning scheme on the few-shot NER. Experimental results demonstrate that our model outperforms state-of-the-art models on two standard Few-shot NER datasets.
ConTFV: A Contrastive Learning Framework for Table-based Fact Verification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Binary Classification, Contrastive Learning, Fact Verification, Semantic Parsing, Semantic Similarity, Semantic Textual Similarity, Table-based Fact Verification

Table-based fact verification is a binary classification task where the challenging part lies in the table's structural parsing and symbolic reasoning. Jointly pre-training on abundant textual and tabular data has been conducted for table semantic parsing recently. However, these models are designed for the table's general understanding, directly fine-tuning for table-based fact verification cannot exploit the advantages to the full. In this paper, we propose ConTFV, a Contrastive learning framework for Table-based Fact Verification, to make the pre-trained model more task-relevant and tap its potential for better representations. By transforming it into a semantic similarity task, our method can outperform baselines with 1.2% on TabFact.
BufferSearch: Generating Black-Box Adversarial Texts With Lower Queries
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Adversarial Attack, text-classification, Text Classification

Machine learning security has recently become a prominent topic in the natural language processing (NLP) area. The existing black-box adversarial attack suffers prohibitively from the high model querying complexity, resulting in easily being captured by anti-attack monitors. Meanwhile, how to eliminate redundant model queries is rarely explored. In this paper, we propose a query-efficient approach BufferSearch to effectively attack general intelligent NLP systems with the minimal number of querying requests. In general, BufferSearch makes use of historical information and conducts statistical test to avoid incurring model queries frequently. Numerically, we demonstrate the effectiveness of BufferSearch on various benchmark text-classification experiments by achieving the competitive attacking performance but with a significant reduction of query quantity. Furthermore,BufferSearch performs multiple times better than competitors within restricted query budget. Our work establishes a strong benchmark for the future study of query-efficiency in NLP  adversarial attacks. The source code is available at \url{https://tinyurl.com/buffersearch}.
Event Detection via Derangement Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Event Detection, Question Answering

Event detection (ED), aiming to detect events from texts and categorize them, is vital to understanding the messages.  Recently, ED without triggers has been proposed and gained benefits since it relieves the tedious effort of data labeling.  However, it still suffers from several formidable challenges: multi-label, insufficient clues, and imbalanced event types.  We, therefore, propose a novel Derangement Question-Answering (DQA) framework on top of BERT to tackle the above challenges.  More specially, we treat the input text as a {\em question} and directly concatenate it with all event types, who are deemed as {\em answers}. Thus, by utilizing the original information, we can facilitate the power of self-attention in BERT to absorb the semantic relation between the original input text and the event types.  Moreover, we design a simple yet effective {\em derangement} mechanism to relieve the issue of imbalanced event types.  By including such perturbation, we can train a more robust model to promote the semantic information in the major events while recording the position of the minor events than the vanilla QA framework.  The empirical results show that: (1) our proposed DQA framework attains state-of-the-art performance over previous competitive models. (2) Our model can automatically link the triggers with the event types while signifying the corresponding arguments.
Comprehensive Multi-Modal Interactions for Referring Image Segmentation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Image Segmentation, Segmentation, Semantic Segmentation

We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intra-modal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach's performance on four benchmark datasets, showing considerable performance gains over the existing state-of-the-art (SOTA) methods.
Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Chinese Word Segmentation, Segmentation

Word Segmentation is a fundamental step for understanding Chinese language. Previous neural approaches for unsupervised Chinese Word Segmentation (CWS) only exploits shallow semantic information, which can miss important context. Large scale Pre-trained language models (PLM) have achieved great success in many areas because of its ability to capture the deep contextual semantic relation. In this paper, we propose to take advantage of the deep semantic information embedded in PLM (e.g., BERT) with a self-training manner,  which iteratively probes and transforms the semantic information in PLM into explicit word segmentation ability. Extensive experiment results show that our proposed approach achieves state-of-the-art F1 score on  two CWS benchmark datasets.
Gradient Sparsification For \emph{Masked Fine-Tuning} of Transformers
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Transfer Learning, XLM-R

Fine-tuning masked language models is widely adopted for transfer learning to downstream tasks and can be achieved by (1) freezing gradients of the pretrained network or only updating gradients of a newly added classification layer or (2) performing gradient updates on all parameters. Gradual unfreezing trades off between the two by gradually unfreezing gradients of whole layers during training. We propose to extend this to {\em stochastic gradient masking} to regularize pretrained language models for improved fine-tuning performance. We introduce \emph{GradDrop} and variants thereof, a class of gradient sparsification methods that mask gradients prior to gradient descent. Unlike gradual unfreezing which is non-sparse and deterministic, GradDrop is sparse and stochastic. Experiments on the multilingual XGLUE benchmark with XLM-R$_{\text{Large}}$ show that \emph{GradDrop} outperforms standard fine-tuning and gradual unfreezing, while being competitive against methods that use additional translated data and intermediate pretraining. Lastly, we identify cases where largest zero-shot performance gains are on less resourced languages.