pwc-archive/papers-with-abstracts huggingface dataset:
All You May Need for VQA are Image Captions
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: All, Image Captioning, Question Answering, Question Generation, Question-Generation, Visual Question Answering, Visual Question Answering (VQA)

Visual Question Answering (VQA) is a challenge that has benefited tremendously from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. We propose here a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annotations combined with neural models for question generation. We show that the resulting data is powerful enough to boost the state-of-the-art zero-shot results on VQA by double digits, and exhibits a level of robustness that lacks in models with the same architecture trained on human-annotated data.
Discourse-Aware Prompt Design for Text Generation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Blocking, Conditional Text Generation, Language Modelling, Text Generation

Current efficient fine-tuning methods (e.g., adapters, prefix-tuning, etc.) have optimized conditional text generation via training a small set of extra parameters of the neural language model, while freezing the rest for efficiency. While showing strong performance on some generation tasks, they don't generalize across all generation tasks. In this work, we show that prompt based conditional text generation can be improved with simple and efficient methods that simulate modeling the discourse structure of human written text.We introduce two key design choices: First, we show that a higher-level discourse structure of human written text can be modelled with hierarchical blocking on prefix parameters. It enables spanning different parts of the input and output text and yields more coherent output generations.  Second, we propose sparse prefix tuning by introducing attention sparsity on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function, respectively. We find that sparse attention enables the prefix-tuning to better control of the input contents (salient facts) yielding more efficient tuning of the prefix-parameters. Our experiments show that structured design of prefix parameters can yield more coherent, faithful and relevant generations than baseline prefix-tuning on all generation tasks and perform at par with fine-tuning while being more efficient.
Multiplicative Position-aware Transformer Models for Language Understanding
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Position

In order to utilize positional ordering information in transformer models, various flavors of absolute and relative position embeddings have been proposed. However, there is no comprehensive comparison of position embedding methods in the literature. In this paper, we review existing position embedding methods and compare their accuracy on downstream NLP tasks, using our own implementations. We also propose a novel multiplicative embedding method which leads to superior accuracy when compared to existing methods. Finally, we show that our proposed embedding method, served as a drop-in replacement of the default absolute position embedding, can improve the RoBERTa-base and RoBERTa-large models on SQuAD1.1 and SQuAD2.0 datasets.
Document Classification with Word Sense Knowledge
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Classification, Document Classification, Word Sense Disambiguation

The performance of Word Sense Disambiguation (WSD) on a standard evaluation framework has reached an estimated upper bound. However, there is limited research on the application of WSD to relevant NLP tasks due to the high computational cost of supervised systems. In this paper, we propose a partial WSD method with sense category information and incorporate the sense knowledge into a supervised document classification framework. Experimental results show that the proposed method can constantly boost the system’s performance on document classification datasets against strong baselines.
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Common Sense Reasoning, World Knowledge

Recent work have shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization.To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out "easy" instances (Sakaguchi et al., 2020) culminating in a recent proposal to eliminate single-word correlations altogether (Gardner et al., 2021).In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to "throw the baby out with the bathwater" and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.
Semantics is Actually 82% Distributional, but Neural Networks Aren't.
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Distributional semantics is often proposed as the linguistic theory underpinning many of the most efficient current NLP systems. In the present paper, we question the linguistic well-foundedness of these models, addressing it from the perspective of distributional substitution. To that end, we provide a dataset of human judgments on the distributional hypothesis, and highlight how humans cannot systematically distinguish pairs of words solely from contextual information. We stress that earlier static embedding architectures are competitive with more modern contextual embeddings on the distributional substitution task, and that neither serve as good models of human linguistic behavior.
Probing the Role of Positional Information in Vision-Language Models
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Contrastive Learning, Image-text matching, Object Localization, Position, Question Answering, Text Matching, Visual Question Answering, Visual Question Answering (VQA)

In most Vision-Language models (VL) the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we probe the use of the PI in the representation and study its effect on Visual Question Answering. We show that the model is not capable of leveraging the PI for image-text matching task on a challenge set where only position differs. Yet, our experiments with probing confirm that the PI is indeed present in the representation. We introduce two strategies (i) Positional Information Pre-training and (ii) Contrastive Learning on PI using Cross-Modality Matching. Doing so, the model can correctly classify if image with detailed PI statements matches. Additionally to the 2D information from bounding boxes, we introduce the object's depth as a new feature for a better object localization in the space. Even though we were able to improve the model properties as defined by our probes, it only has a negligible effect on the downstream performance. Our results thus highlight an important issue of multimodal modeling: the mere presence of information detectable by a probing classifier is not a guarantee that the information is available in a cross-modal setup.
Prompt-based Zero-shot Relation Classification with Semantic Knowledge Augmentation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Relation, Relation Classification, Sentence, Zero-shot Relation Classification

In relation classification, recognizing unseen (new) relations for which there are no training instances is a challenging task. We propose a prompt-based model with semantic knowledge augmentation (ZS-SKA) to recognize unseen relations under the zero-shot setting. We present a new word-level sentence translation rule and generate augmented instances with unseen relations from instances with seen relations using that new rule. We design prompts based on an external knowledge graph to integrate semantic knowledge information learned from seen relations. Instead of using the actual label sets in the prompt template, we construct weighted virtual label words. We learn the representations of both seen and unseen relations with augmented instances and prompts. We then calculate the distance between the generated representations using prototypical networks to predict unseen relations. Extensive experiments conducted on three public datasets show that ZS-SKA outperforms state-of-the-art methods under the zero-shot scenarios. Our experimental results also demonstrate the effectiveness and robustness of ZS-SKA.
Sentence-Level Resampling for Named Entity Recognition
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Data Augmentation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Sentence

As a fundamental task in natural language processing, named entity recognition (NER) aims to locate and classify named entities in unstructured text. However, named entities are always the minority among all tokens in the text. This data imbalance problem presents a challenge to machine learning models as their learning objective is usually dominated by the majority of non-entity tokens. To alleviate data imbalance, we propose a set of sentence-level resampling methods  where the importance of each training sentence is computed based on its tokens and entities. We study the generalizability of these resampling methods on a wide variety of NER models (CRF, Bi-LSTM, and BERT) across corpora from diverse domains (general, social, and medical texts). Extensive experiments show that the proposed methods improve span-level macro F1-scores of the evaluated NER models on multiple corpora, frequently outperforming sub-sentence-level resampling, data augmentation, and special loss functions such as focal and Dice loss.
Distant Supervision for Relation Extraction with Hierarchical Attention-Based Networks
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Relation, Relation Extraction, Sentence

Distant supervision employs external knowledge bases to automatically label corpora. The labeled sentences in a corpus are usually packaged and trained for relation extraction using a multi-instance learning paradigm. The automated distant supervision inevitably introduces label noises. Previous studies that used sentence-level attention mechanisms to de-noise neither considered correlation among sentences in a bag nor correlation among bags. This paper proposes hierarchical attention-based networks that can de-noise at both sentence and bag levels. In the calculation of bag representation, we provide weights to sentence representations using sentence-level attention that considers correlations among sentences in each bag. Then, we employ bag-level attention to merge the similar bags by considering their correlations and to provide properer weights in the calculation of bag group representation. Experimental results on the New York Times datasets show that the proposed method outperforms the state-of-the-art ones.
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Continual Learning, Few-Shot Learning, Question Answering, Transfer Learning

Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.
Analytical Reasoning of Text
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions. However, current neural models with implicit reasoning ability struggle to solve this task. In this paper, we study the challenge of analytical reasoning of text and collect a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task, and present an approach dubbed ARM. It extracts knowledge such as participants and facts from the context. Such knowledge are applied to an inference engine to deduce legitimate solutions for drawing conclusions. In our experiments, we find that ubiquitous pre-trained models struggle to deal with this task as their performance is close to random guess. Results show that ARM outperforms pre-trained models significantly. Moreover, we demonstrate that ARM has better explicit interpretable reasoning ability.
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Code Generation, Knowledge Distillation, Language Modeling, Language Modelling, Semantic Parsing, Sentence

Semantic parsing considers the task of mapping a natural language sentence into a target formal representation, where various sophisticated sequence-to-sequence (seq2seq) models have been applied with promising results. Generally, these target representations follow a syntax formalism that limits permitted forms. However, it is neither easy nor flexible to explicitly integrate this syntax formalism into a neural seq2seq model. In this paper, we present a structure-aware self-attention language model to capture structural information of target representations and propose a knowledge distillation based approach to incorporating the target language model into a seq2seq model, where grammar rules, sketches or extra corpus are not required in the training process. An ablation study shows that the proposed language model can notably improve the performance of the baseline model. The experiments show that our method achieves new state-of-the-art performance among neural approaches on four semantic parsing (ATIS, GEO) and Python code generation (Django, CoNaLa) tasks.
A Two-stage Attention-based Model for Customer Satisfaction Prediction in E-commerce Customer Service
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Nowadays, customer satisfaction prediction (CSP) on e-commerce platforms has become a hot research topic for both intelligent customer service and artificial customer service. CSP aims to discover customer satisfaction according to the dialogue content of customer and customer service, for the purpose of improving service quality and customer experience. In this paper, we focus on CSP for intelligent customer service chatbots. Although previous works have made some progress in many aspects, they mostly ignore the huge differences of expressions between customer and customer service, and fail to adequately consider the internal relations of those two kinds of personalized expressions. Thus, for emphasizing the importance of modeling customer part and  service part separately, in this work we propose a two-stage dialogue-level classification model, which contains an intra-stage and an inter-stage to handle the issues above. In the intra-stage, we model customer part and service part separately by using attention mechanism combined with personalized context to obtain {\it customer state} and  {\it service state}. Then we interact those two states with each other in the inter-stage to capture the final satisfaction representation of the whole dialogue. Experiment results demonstrate that our model achieves better performance than several competitive baselines on our in-house dataset and four public datasets.
A Graph Fusion Approach for Cross-Lingual Machine Reading Comprehension
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: graph construction, Machine Reading Comprehension, Machine Translation, Reading Comprehension

Although great progress has been made for Machine Reading Comprehension (MRC) in English, scaling out to a large number of languages remains a huge challenge due to the lack of large amounts of annotated training data in non-English languages. To address this challenge, some recent efforts of cross-lingual MRC employ machine translation to transfer knowledge from English to other languages, through either explicit alignment or implicit attention. For effective knowledge transition, it is beneficial to leverage both semantic and syntactic information. However, the existing methods fail to explicitly incorporate syntax information in model learning. Consequently, the models are not robust to errors in alignment and noises in attention. In this work, we propose a novel approach, named GraFusionMRC, which jointly models the cross-lingual alignment information and the mono-lingual syntax information using a graph. We develop a series of algorithms including graph construction, learning, and pre-training. The experiments on two benchmark datasets for cross-lingual MRC show that our approach outperforms all strong baselines, which verifies the effectiveness of syntax information for cross-lingual MRC. The code will be made open-sourced on Github.
Towards Equal Opportunity Fairness through Adversarial Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Fairness

Adversarial training is a common approach for bias mitigation in natural language processing. Although most work on debiasing is based around the equal opportunity criterion, it is not explicitly captured in standard adversarial training. In this paper, we propose an augmented discriminator for adversarial training, which takes the target class as input to create richer features and more explicitly model equal opportunity. Experimental results over two datasets show that our method substantially improves over standard adversarial debiasing methods, in terms of the performance--fairness trade-off.
Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, NMT, Translation

We introduce Bi-SimCut: a simple but effective strategy to boost neural machine translation (NMT) performance. It consists of two training procedures: bidirectional pretraining and unidirectional finetuning. Both procedures utilize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff samples. Without utilizing extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.1M): BLEU scores of $31.16$ for $\texttt{en}\rightarrow\texttt{de}$ and $38.37$ for $\texttt{de}\rightarrow\texttt{en}$ on the IWSLT14 dataset, $30.78$ for $\texttt{en}\rightarrow\texttt{de}$ and $35.15$ for $\texttt{de}\rightarrow\texttt{en}$ on the WMT14 dataset, and $27.17$ for $\texttt{zh}\rightarrow\texttt{en}$ on the WMT17 dataset. SimCut is not a new method, but a version of Cutoff (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method. Given the universality and simplicity of Bi-SimCut and SimCut, we believe they can serve as strong baselines for future NMT research.
Semantic Parsing for Planning Goals as Constrained Combinatorial Contextual Bandits
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Multi-Armed Bandits, Semantic Parsing

We are working towards AI planning systems with natural language interfaces. In this paper, we tackle the semantic parsing problem of learning to set the logical goals of the planning system based on a natural language description of the task. The current state of the art in semantic parsing is to use supervised learning with deep neural networks but this needs a lot of labelled data made by domain experts. To reduce this need, we additionally use a reward signal that comes from completing the AI planning task. We formalize this as a constrained combinatorial contextual bandit problem. The context is created by using a deep neural network for feature extraction and the constrained combinatorial nature of the task can be used to increase the efficiency of learning. We show this result theoretically with our lower regret bound and then experimentally in our extension of the TextWorld problem.
Graph Recurrent Neural Network for Text Classification
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Classification, text-classification, Text Classification

Graph Neural Networks(GNNs) application to text classification is currently one of the most popular fields. Most GNNs-based models only focus on the interaction of words in the document, whereas the word order is ignored, and the related semantic information is lost. In addition, when the graph density increases, the word nodes become over-smooth. As a result, the semantic information of the document is destroyed. In this paper, TextGRNN, a text classification method based on GNN is proposed to solve the above problems. First, our proposed model constructs the document-level graph via Visibility Graph, in which the graph density is restrained, and updates the word representations by GNN. Then the TextGRNN model utilizes Bi-LSTM that can recognize word order to learn the semantic information of the document. Finally, the attention mechanism is used to highlight the essential words. Numerous experiments on three benchmark datasets demonstrate that our model is preferable to state-of-the-art text classification methods.