pwc-archive/papers-with-abstracts huggingface dataset:
HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Entity Alignment, Question Answering, Text Generation

Tables are often created with hierarchies, but existing works on table reasoning mainly focus on flat tables and neglect hierarchical tables. Hierarchical tables challenge numerical reasoning by complex hierarchical indexing, as well as implicit relationships of calculation and semantics. We present a new dataset, HiTab, to study question answering (QA) and natural language generation (NLG) over hierarchical tables. HiTab is a cross-domain dataset constructed from a wealth of statistical reports and Wikipedia pages, and has unique characteristics: (1) nearly all tables are hierarchical, and (2) QA pairs are not proposed by annotators from scratch, but are revised from real and meaningful sentences authored by analysts. (3) to reveal complex numerical reasoning in statistical reports, we provide fine-grained annotations of quantity and entity alignment. 
Experiments suggest that this HiTab presents a strong challenge for existing baselines and a valuable benchmark for future research. Targeting hierarchical structure, we devise a hierarchy-aware logical form for symbolic reasoning over tables, which shows high effectiveness. Targeting table reasoning, we leverage entity and quantity alignment to explore partially supervised training in QA and conditional generation in NLG, and largely reduce spurious predictions in QA and produce better descriptions in NLG.
Echo-Attention: Attend Once and Get $N$ Attentions for Free
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Language Modeling, Language Modelling, Machine Translation, Question Answering, Translation

This paper proposes echo-attention layers, an efficient method for improving the expressiveness of the self-attention layers without incurring significant parameter or training time costs. The key idea is to iteratively refine the attentional activations via stateful repeated computation, i.e., we compute the activations once and get $N$ refinements (echo-attentions) at a relatively cheap cost. To this end, we introduce an update and state transition function that operates over these attentional activations. Via a set of extensive experiments, we show that this the proposed Echoformer model demonstrates widespread benefits across 21 datasets including language modeling, machine translation, language understanding and question answering.
On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Domain Adaptation, Domain Generalization, Language Modeling, Language Modelling, Machine Translation, Translation

Domain adaptation of neural networks commonly relies on three training phases: pretraining, selected data training and then fine tuning. Data selection improves target domain generalization by training further on pretraining data identified by relying on a small sample of target domain data. This work examines the benefit of data selection for language modeling and machine translation. Our experiments assess the  complementarity of selection with fine tuning and result in practical recommendations: 
(i) selected data must be similar to the fine-tuning domain but not so much as to erode the complementary effect of fine-tuning;
(ii) there is a trade-off between selecting little data for fast but limited progress or much data for slow but long lasting progress;
(iii) data selection can be applied early during pretraining, with performance gains comparable to long pretraining session; 
(iv) data selection from domain classifiers is often more effective than the popular contrastive data selection method.
Learning-based Memetic Algorithm for Hard-label Textual Attack
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Combinatorial Optimization, Hard-label Attack, Natural Language Inference, text-classification, Text Classification

Deep neural networks are widely known to be vulnerable to adversarial examples in Natural Language Processing. However, existing textual adversarial attacks usually utilize the gradient or prediction confidence to generate adversarial examples, making it hard to be applied in real-world applications. To this end, we consider a more rigorous setting, namely hard-label attack, in which the attacker could only access the prediction labels. There are only a few hard-label attacks proposed currently, among which the one based on genetic algorithm exhibits high attack performance. It inspires us to design a new hard-label attack for better performance based on a combinatorial optimization approach. In this work, we propose a novel hard-label attack, named Learning-based Memetic Algorithm (LMA), which integrates the word importance learned from the attack history into the search of memetic algorithm to optimize the adversary perturbation. Extensive evaluations for text classification and textual entailment using various datasets and models demonstrate that the proposed LMA significantly outperforms existing hard-label attack regarding attack performance and adversary quality.
Fully Hyperbolic Neural Networks
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

Hyperbolic neural networks have shown great potential for modeling complex data. However, existing hyperbolic networks are not completely hyperbolic, as they encode features in the hyperbolic space yet formalize most of their operations in the tangent space (a Euclidean subspace) at the origin of the hyperbolic model. This hybrid method greatly limits the modeling ability of networks. In this paper, we propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model by adapting the Lorentz transformations (including boost and rotation) to formalize essential operations of neural networks. Moreover, we also prove that linear transformation in tangent spaces used by existing hyperbolic networks is a relaxation of the Lorentz rotation and does not include the boost, implicitly limiting the capabilities of existing hyperbolic networks. The experimental results on four NLP tasks show that our method has better performance for building both shallow and deep networks. Our code will be released to facilitate follow-up research.
A Novel Metric for Evaluating Semantics Preservation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Language Modeling, Language Modelling, Predicate Detection, Sentence, Sentence Compression, text similarity

In this paper, we leverage pre-trained language models (PLMs) to precisely evaluate the semantics preservation of edition process on sentences. Our metric, Neighboring Distribution Divergence (NDD), evaluates the disturbance on predicted distribution of neighboring words from mask language model (MLM). NDD is capable of detecting precise changes in semantics which are easily ignored by text similarity. By exploiting the property of NDD, we implement a unsupervised and even training-free algorithm for extractive sentence compression. We show that NDD-based algorithm outperforms previous perplexity-based unsupervised algorithm by a large margin. For further exploration on interpretability, we evaluate NDD by pruning on syntactic dependency treebanks and apply NDD for predicate detection as well.
3M:Multi-document Summarization Considering Main and Minor Relationship
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Document Summarization, Multi-Document Summarization

The multi-document summary task is an important branch of the information aggregation task. Compared with the single-document summary, the input of multi-document summary is much longer and the logic is more complicated. This article proposes a hypothesis: taking the content of a document as the main body and the content of other documents as auxiliary information, a summary that combines all the information in the document collection can be generated. Based on this assumption, the multi-document summarization task can select one main document, and then combine the information of other documents for summary generation. This paper combines CopyTransformer and the Maximal Marginal Relevance (MMR) to design Multi-document summarization considering Main and Minor relationship model(3M). Empirical results on the Multi-News and DUC 2004 dataset show that the 3M brings substantial improvements over several strong baselines, manual evaluation shows that the generated abstract is fluent and can better express the content of the main document. In addition, by selecting different main documents, 3M can generate multiple abstracts with different styles for a set of documents.
Safety Bench: Identifying Safety-Sensitive Situations for Open-domain Conversational Systems
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

The social impact of natural language processing and its applications has received increasing attention. Here, we focus on the problem of safety for end-to-end conversational AI.  We survey the problem landscape therein, introducing a taxonomy of three observed phenomena: the Instigator, Yea-Sayer, and Impostor effects. To help researchers better understand the impact of their conversational models with respect to these scenarios, we present Safety Bench, a set of open-source tooling for quickly assessing safety issues. Finally, we provide extensive analysis of these tools using five popular models and make recommendations for future use.
Evaluation of Transfer Learning for Polish with a text-to-text model
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Decoder, Denoising, Machine Translation, Question Answering, Transfer Learning, Translation

We present polT - a general purpose text-to-text model for Polish that can be fine-tuned on a variety on Natural Language Processing (NLP) tasks with a single training objective. Unsupervised denoising pre-training is performed efficiently by initializing the model weights with multi-lingual T5 (mT5) counterpart. We evaluate performance of polT, mT5, Polish BART (plBART) and Polish GPT-2 (papuGaPT2) on diverse downstream tasks such as: text-to-text KLEJ benchmark, en-pl machine translation, question answering and summarization. The polT scores top on all of this tasks except summarization where plBART is best. In general (except summarization), the larger the model the better the results. The encoder-decoder architectures prove to be better than decoder-only equivalent. Additionally, since summarization and question answering lack benchmark datasets for Polish language we describe in detail their construction and will make them publicly available.
CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Emotion Recognition, Emotion Recognition in Conversation

As the use of interactive machines grow, the task of Emotion Recognition in Conversation (ERC) became more important. If the machine generated sentences reflect emotion, more human-like sympathetic conversations are possible. Since emotion recognition in conversation is inaccurate if the previous utterances are not taken into account, many studies reflect the dialogue context to improve the performances. We introduce CoMPM, a context embedding module (CoM) combined with a pre-trained memory module (PM) that tracks memory of the speaker's previous utterances within the context, and show that the pre-trained memory significantly improves the final accuracy of emotion recognition. We achieve competitive performance with previous methods on English datasets (MELD, EmoryNLP, IEMOCAP, DailyDailog), and achieve good performance with small data sets. In addition, our method shows that it can be extended to other languages because structured knowledge is not required unlike existing methods.
Subword-based Cross-lingual Transfer of Embeddings from Hindi to Marathi
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Cross-Lingual Transfer, Word Embeddings, Word Similarity

Word embeddings are growing to be a crucial resource in the field of NLP for any language. This work focuses on static subword embeddings transfer for Indian languages from a relatively higher resource language to a genealogically related low resource language. We work with Hindi-Marathi as our language pair, simulating a low-resource scenario for Marathi. We demonstrate the consistent benefits of unsupervised morphemic segmentation on both source and target sides over the treatment performed by FastText. We show that a trivial "copy-and-paste'' embeddings transfer based on even perfect bilingual lexicons is inadequate in capturing language-specific relationships. Our best-performing approach uses an EM-style approach to learning bilingual subword embeddings; the resulting embeddings are evaluated using the publicly available Marathi Word Similarity task as well as WordNet-Based Synonymy Tests. We find that our approach significantly outperforms the FastText baseline on both tasks; on the former task, its performance is close to that of pretrained FastText Marathi embeddings that use two orders of magnitude more Marathi data.
Cross-lingual Constituency Parsing with Linguistic Typology Knowledge
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Constituency Parsing, Cross-Lingual Transfer, Dependency Parsing, Transfer Learning

Cross-lingual Transfer learning (CLT) has successfully been applied to the dependency parsing task. This is the first work that evaluates a CLT based approach to the Constituency parsing task. Furthermore, we utilized the linguistic typology knowledge in WALS database to improve the cross-lingual transferring ability of our proposed parser.
Challenges in Generalization in Open Domain Question Answering
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Natural Questions, Open-Domain Question Answering, Question Answering, Retrieval, Systematic Generalization, TriviaQA

Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with training questions. However, it is as of yet unclear which aspects of novel questions that make them challenging. Drawing upon studies on systematic generalization, we introduce and annotate questions according to three categories that measure different levels and kinds of generalization: training set overlap, compositional generalization (comp-gen), and novel entity generalization (novel-entity). When evaluating six popular parametric and non-parametric models, we find that for the established Natural Questions and TriviaQA datasets, even the strongest model performance for comp-gen/novel-entity is 13.1/5.4% and 9.6/1.5% lower compared to that for the full test set – indicating the challenge posed by these types of questions. Furthermore, we show that whilst non-parametric models can handle questions containing novel entities, they struggle with those requiring compositional generalization. Through thorough analysis we find that key question difficulty factors are: cascading errors from the retrieval component, frequency of question pattern, and frequency of the entity.
Zero-shot Cross-lingual Transfer is Under-specified Optimization
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Cross-Lingual Transfer, Zero-Shot Cross-Lingual Transfer

Pretrained multilingual encoders enable zero-shot cross-lingual transfer performance, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that high variance results from zero-shot cross-lingual transfer solving an under-specified optimization problem. We show that the source language monolingual model and source + target bilingual model are linearly connected using a model interpolation, suggesting that the model struggles to identify good solutions for both source and target languages using the source language alone.
Multi-Task End-to-End Training Improves Conversational Recommendation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Conversational Recommendation, Dialogue Generation, Movie Recommendation

In this paper, we analyze the performance of a multitask end-to-end transformer model on the task of conversational recommendations, which aim to provide recommendations based on a user’s explicit preferences expressed in dialogue. While previous works in this area adopt complex multi-component approaches where the dialogue generation and entity recommendation tasks are handled by separate components, we show that a unified transformer model, based on the T5 text-to-text transformer model, can perform competitively in both recommending relevant items and generating conversation dialogue. We fine-tune our model on the ReDIAL conversational movie recommendation dataset, and create additional training tasks derived from MovieLens (such as the prediction of movie attributes and related movies based on an input movie), in a multitask learning setting. Using a series of probe studies, we demonstrate that the learned knowledge in the additional tasks is transferred to the conversational setting, where each task leads to a $9\% - 52\%$ increase in its related probe score.
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Natural Language Understanding

A central question in natural language understanding (NLU) research is whether high performance demonstrates the models' strong reasoning capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. The transformations involve removing instances of specific word classes and often lead to non-sensical sentences. Our results show that performance remains high for most GLUE tasks when the models are fine-tuned or tested on corrupted data, suggesting that the models leverage other cues for prediction even in non-sensical contexts.
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Language Modeling, Language Modelling

Standard pretrained language models operate
on sequences of subword tokens without direct access to the characters that compose each
token’s string representation. We probe the
embedding layer of pretrained language models and show that models learn the internal
character composition of whole word and subword tokens to a surprising extent, without
ever seeing the characters coupled with the tokens. Our results show that the embedding
layer of RoBERTa holds enough information
to accurately spell up to a third of the vocabulary and reach high average character ngram
overlap on all token types. We further test
whether enriching subword models with additional character information can improve language modeling, and observe that this method
has a near-identical learning curve as training without spelling-based enrichment. Overall, our results suggest that language modeling
objectives incentivize the model to implicitly
learn some notion of spelling, and that explicitly teaching the model how to spell does not
enhance its performance on such tasks.
Modeling Context With Linear Attention for Scalable Document-Level Translation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Document Translation, Machine Translation, Sentence, Translation

Document-level neural machine translation allows models to leverage dependencies beyond sentence-internal context to produce more coherent and consistent translations. However, these models, predominantly based on transformers, are difficult to scale to long documents due to the quadratic time and space complexity of their self-attention layers. Recent efforts on efficient attention variants improve scalability, but it is yet unclear if and to what extent their inductive biases are suitable for document translation. In this paper, we explore the efficacy of a recent linear attention model by Peng et al. (2021) on document-level translation and augment it with a sentential gating mechanism. We evaluate the model on the IWSLT 2015 and OpenSubtitles 2018 datasets against a strong transformer baseline and achieve up to 40% decoding speedup with similar or improved BLEU scores. We show that the sentential gate further improves translation quality on IWSLT, a dataset with long sequences.
Unmasking the Trade-off: Measuring Gender Bias Mitigation and Over-debiasing Effects in Pretrained Language Models
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

Pretrained language models (PLMs) have demonstrated success across many natural language processing tasks. However, evidence suggests that they encode gender bias present in the corpora they are trained on. Existing bias mitigation methods are usually devised to remove all associations related to gender. This can hurt the performance of PLMs, because it can lead to a loss of genuine and factual associations (e.g., not associating the word "mother" with females over males). To measure the extent of undesirable loss of gender associations (i.e. over-debiasing), we introduce the Desirable Associations evaluation corpus for Gender (DA-Gender). We find that three popular debiasing methods result in substantial undesirable loss of gender associations. Our results highlight the importance of mitigating bias without removing genuine gender association, and our dataset constitutes the first benchmark to evaluate over-debiasing.
Multimodal Entity Tagging with Multimodal Knowledge Base
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Information Retrieval, Retrieval

To enhance research on multimodal knowledge base and multimodal information processing, we propose a new task called multimodal entity tagging (MET) with a multimodal knowledge base (MKB). We also develop a dataset for the problem using an existing MKB. In an MKB, there are entities and their associated texts and images. In MET, given a text-image pair, one uses the information in the MKB to automatically identify the related entity in the text-image pair. We solve the task by using the information retrieval paradigm and implement several baselines using state-of-the-art methods in NLP and CV.  We conduct extensive experiments and make analyses on the experimental results. The results show that the task is challenging, but current technologies can achieve relatively high performance. We will release the dataset, code, and models for future research.
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Language Modeling, Language Modelling, parameter-efficient fine-tuning

We show that with small-to-medium training data, fine-tuning only the bias terms (or a subset of the bias terms) of pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, bias-only fine-tuning is competitive with other sparse fine-tuning methods.
Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.