pwc-archive/papers-with-abstracts huggingface dataset:
On Measuring Social Biases in Prompt-Based Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Form, Language Modelling, Natural Language Inference, Prompt Engineering, Question Answering

Large language models trained on a mixture of NLP tasks that are converted into a text-to-text format using prompts, can generalize into novel forms of language and handle novel tasks. A large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. We consider an alternative measure and inquire whether the way in which an input is encoded affects \textit{social biases} promoted in outputs. In this paper, we study T0, a large-scale multi-task text-to-text language model trained using prompt-based learning. We consider two different forms of semantically equivalent inputs: \textit{question-answer} format and \textit{premise-hypothesis} format. We use an existing bias benchmark for the former BBQ and create the first bias benchmark in natural language inference BBNLI with hand-written hypotheses while also converting each benchmark into the other form. The results on two benchmarks suggest that given two different formulations of essentially the same input, T0 conspicuously acts more biased in question answering form, which is seen during training, compared to premise-hypothesis form which is unlike its training examples.
Self-Supervised Bot Play for Transcript-Free Conversational Recommendation with Justification
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Conversational Recommendation, Recommendation Systems

Conversational recommender systems offer a way for users to engage in multi-turn conversations to find items they enjoy. Dialog agents for conversational recommendation rely on expensive human dialog transcripts, limiting their usage to domains where such data exists. We develop an alternative, two-part framework for training multi-turn conversational recommenders that accommodate a common paradigm of conversation: experts provide and justify suggestions, while users can critique and respond. We can thus adapt conversational recommendation to a wider range of domains where crowd-sourced ground truth dialogs are not available. First, we train a recommender system to jointly suggest items and justify its reasoning via subjective aspects. We then fine-tune this model to incorporate iterative user feedback via self-supervised bot-play. Experiments on three real-world datasets demonstrate that our system can be applied to different recommendation models across diverse domains to achieve state-of-the-art performance in multi-turn recommendation. Human studies show that systems trained with our framework provide more useful, helpful, and knowledgeable suggestions in warm- and cold-start settings.
EASE: Entity-Aware Contrastive Learning of Sentence Embedding
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Clustering, Contrastive Learning, Semantic Textual Similarity, Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings, Short Text Clustering, STS, Text Clustering

We present EASE, a novel method for learning sentence embeddings via contrastive learning between sentences and their related entities.The advantage of using entity supervision is twofold: (1) entities have been shown to be a strong indicator of text semantics and thus should provide rich training signals for sentence embeddings; (2) entities are defined independently of languages and thus offer useful cross-lingual alignment supervision.We evaluate EASE against other unsupervised models both in monolingual and multilingual settings.We show that EASE exhibits competitive or better performance in English semantic textual similarity (STS) and short text clustering (STC) tasks and it significantly outperforms baseline methods in multilingual settings on a variety of tasks.Our EASE model and newly constructed multilingual STC dataset, MewsC-15, have been made publicly available to catalyze future research on sentence embeddings.
Improving Robustness in Multilingual Machine Translation via Data Augmentation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Data Augmentation, Machine Translation, Translation

 Multilingual humans can and do seamlessly switch back and forth between languages when communicating. However, multilingual (machine) translation models are not robust to such sudden changes. In this work, we explore the robustness of multilingual MT models to language switching and propose checks to measure switching capability. We also investigate simple and effective data augmentation methods that can enhance robustness. A glass-box analysis of attention modules demonstrates the effectiveness of these methods in improving robustness.
Frustratingly Simple Regularization to Improve Zero-shot Cross-lingual Robustness
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Cross-Lingual Transfer, XLM-R, Zero-Shot Cross-Lingual Transfer

Large-scale multilingual pretrained encoders, such as mBERT and XLM-R, have demonstrated impressive zero-shot cross-lingual transfer capability across multiple NLP tasks. However, as we show in this paper, these models suffer from two major problems: (1) degradation in zero-shot cross-lingual performance after fine-tuning on a single language, and (2) cross-lingual performance sensitivity to fine-tuning hyperparameters. In order to address these issues, we evaluate two techniques during fine-tuning, namely, Elastic Weight Consolidation (EWC) and L2-distance regularization to assist the multilingual models in retaining their cross-lingual ability after being fine-tuned on a single language. We compare zero-shot cross-lingual performance of mBERT with/without regularization on four different tasks: XNLI, PANX, UDPOS and PAWSX and demonstrate that the model fine-tuned with L2-distance regularization performs better than its vanilla fine-tuned counterpart in zero-shot setting across all the tasks by up to 1.64%. Moreover, by fine-tuning mBERT with different hyperparameter settings on the specified tasks, we demonstrate that L2-distance regularization also makes fine-tuning more robust,  reducing standard deviation of zero-shot results by up to 87%. Based on our experiments, EWC does not provide consistent improvements across languages. Moreover, to test if additional constraint on the encoder parameters would improve the results further, we compared L2-distance regularization with techniques that freeze most of the encoder parameters during fine-tuning, such as bitfit, soft prompting, and adapter-based methods. However, we observe that L2-distance regularization still performs the best.
Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: reinforcement-learning, Reinforcement Learning (RL), Semantic Parsing

Semantic parsing converts natural language paraphrases into structured logical expressions. In this paper, we consider two such formal representations: Propositional Logic (PL) and First-order Logic (FOL). Due to the insufficiency of annotated data in this field, we use dual reinforcement learning (RL) to make full use of labeled and unlabeled data. We further propose a brand new reward mechanism to avoid the trouble of manually defining the reward in RL. To utilize the training data efficiently and make the learning process consistent with humans, we integrate curriculum learning into our framework. Experimental results show that the proposed method outperforms competitors on different datasets. In addition to the technical contribution, we construct a Chinese-PL/FOL dataset to make up for the lack of data in this field. We aim to release our code as well as the dataset to aid further research in related tasks.
SUBS: Subtree Substitution for Compositional Semantic Parsing
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Data Augmentation, Semantic Parsing

Although sequence-to-sequence models often achieve good performance in semantic parsing for i.i.d. data, their performance is still inferior in compositional generalization. Several data augmentation methods have been proposed to alleviate this problem. However, prior work only leveraged  superficial grammar or rules for data augmentation, which resulted in limited improvement. We propose to use subtree substitution for compositional data augmentation, where we consider subtrees with similar semantic functions as exchangeable. Our experiments showed that such augmented data led to significantly better performance on Scan and GeoQuery, and reached new SOTA on compositional split of GeoQuery.
Experience Affected in the Act of Remembering: A Study of Discursivity of Verb Tense Shifts in Memory Narrative
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

This article contributes to the empirical understanding of the discursivity of verb morphology and verb tense shifts in memory narratives. Specifically, we explore how the 2016 presidential election result, as a historic and political event of the past decade, is recounted collectively through the lens of language use. In an online survey, 185 undergraduate students in the Computer Science department at the University of Georgia were asked to remember the day they learned about the 2016 presidential election results and write a narrative of their experience. The results from our analysis show a distinct correlation between the political leaning of the surveyed population and verb tense shifts in their stories.
Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Semitic morphologically-rich languages (MRLs) are plagued by word ambiguity; in a standard text, many (and often most) of the words will be homographs with multiple possible analyses. Previous research on MRLs claimed that standardly trained contextualized embeddings based on word-pieces do not sufficiently capture the internal structure of words with hugely ambiguous homographs. Taking Hebrew as a case study, we investigate the extent to which Hebrew homographs can be disambiguated using contextualized embeddings. We evaluate all existing models for contextualized Hebrew embeddings on 75 Hebrew homograph challenge sets. Our empirical results demonstrate that contemporary Hebrew contextualized embeddings outperform non-contextualized embeddings; they are most effective for disambiguation of segmentation and morphological features, less so regarding pure sense disambiguation.  We show that these embeddings are more effective when the number of word-piece splits is limited, and they are more effective for 2-way and 3-way ambiguities than for 4-way ambiguity. We show that the embeddings are equally effective for homographs of both balanced and skewed distributions. Finally, we show that these embeddings are as effective for homograph disambiguation with extensive supervised training as with a few-shot setup.
SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Out-of-Distribution Generalization, Semantic Parsing

Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing.The canonical utterance is often lengthy and complex due to the compositional structure of formal languages. Learning to generate such canonical utterance requires significant amount of data to reach high performance. Fine-tuning with only few-shot samples, the LMs can easily forget pretrained knowledge, overfit spurious biases, and suffer from compositionally out-of-distribution generalization errors. To tackle these issues, we propose a novel few-shot semantic parsing method -- SEQZERO. SEQZERO decomposes the problem into a sequence of sub-problems, which corresponds to the sub-clauses of the formal language. Based on the decomposition, the LMs only need to generate short answers using prompts for predicting sub-clauses. Thus, SEQZERO avoids generating a long canonical utterance at once. Moreover, SEQZERO employs not only a few-shot model but also a zero-shot model to alleviate the overfitting.In particular, SEQZERO brings out the merits from both models via ensemble equipped with our proposed constrained rescaling.SEQZERO achieves SOTA performance on GeoQuery dataset and a new EcommerceQuery dataset in the few-shot compositional generalization setting.
Exploring the Universal Vulnerability of Prompt-based Learning Paradigm
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Prompt-based learning paradigm bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. In this paper, we explore this universal vulnerability by either injecting \textit{backdoor triggers} or searching for \textit{adversarial triggers} on pre-trained language models using only plain text. In both scenarios, we demonstrate that our triggers can totally control or severely decrease the performance of prompt-based models fine-tuned on arbitrary downstream tasks, reflecting the universal vulnerability of the prompt-based learning paradigm. Further experiments show that adversarial triggers have good transferability among language models. We also find conventional fine-tuning models are not vulnerable to adversarial triggers constructed from pre-trained language models. We conclude by proposing a potential solution to mitigate our attack methods. All the code and data will be made public.
Putting the Con in Context: Identifying Deceptive Actors in the Game of Mafia
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Game of Mafia, text-classification, Text Classification

While neural networks demonstrate a remarkable ability to model linguistic content, capturing contextual information related to a speaker's conversational role is an open area of research. In this work, we analyze the effect of speaker role on language use through the game of Mafia, in which participants are assigned either an honest or a deceptive role. In addition to building a framework to collect a dataset of Mafia game records, we demonstrate that there are differences in the language produced by players with different roles. We confirm that classification models are able to rank deceptive players as more suspicious than honest ones based only on their use of language. Furthermore, we show that training models on two auxiliary tasks outperforms a standard BERT-based text classification approach. We also present methods for using our trained models to identify features that distinguish between player roles, which could be used to assist players during the Mafia game.
Exposing the Limits of Video-Text Models through Contrast Sets
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling, Multiple-choice, text-classification, Text Classification

Recent video-text models can retrieve relevant videos based on text with a high accuracy, but to what extent do they comprehend the semantics of the text? Can they discriminate between similar entities and actions? To answer this, we propose an evaluation framework that probes video-text models with hard negatives. We automatically build contrast sets, where true textual descriptions are manipulated in ways that change their semantics while maintaining plausibility. Specifically, we leverage a pre-trained language model and a set of heuristics to create verb and person entity focused contrast sets. We apply these in the multiple choice video to-text classification setting. We test the robustness of recent methods on the proposed automatic contrast sets, and compare them to additionally collected human-generated counterparts, to assess their effectiveness. We see that model performance suffers across all methods, erasing the gap between recent CLIP-based methods vs. the earlier methods.
DOCmT5: Document-Level Pre-training of Multilingual Language Models
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: de-en, Document Translation, Language Modeling, Language Modelling, Sentence

In this paper, we introduce DOCmT5, a multi-lingual sequence-to-sequence language model pre-trained with large scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, we try to build a general-purpose pre-trained model that can understand and generate long documents.  We propose a simple and effective pre-training objective - Document reordering MachineTranslation (DrMT), in which the input documents that are shuffled and masked need to be translated. DrMT brings consistent improvements over strong baselines on a variety of document-level generation tasks, including over 12 BLEU points for seen-language pair document-level MT, over 7 BLEU points for unseen-language-pair document-level MT and over 3 ROUGE-1 points for seen-language pair cross-lingual summarization. We achieve state-of-the-art (SOTA) on WMT20 De-En and IWSLT15 Zh-En document translation tasks. We also conduct extensive analysis on various factors for document pre-training, including (1) the effects of pre-training data quality and (2) The effects of combining mono-lingual and cross-lingual pre-training. We plan to make our model checkpoints publicly available.
XLTime: A Cross-Lingual Knowledge Transfer Framework for Temporal Expression Extraction
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Causal Inference, Information Retrieval, Multi-Task Learning, Question Answering, Retrieval, Transfer Learning

Temporal Expression Extraction (TEE) is essential for understanding time in natural language. It has applications in Natural Language Processing (NLP) tasks such as question answering, information retrieval, and causal inference. To date, work in this area has mostly focused on English as there is a scarcity of labeled data for other languages. We propose XLTime, a novel framework for multilingual TEE. XLTime works on top of pre-trained language models and leverages multi-task learning to prompt cross-language knowledge transfer both from English and within the non-English languages. It alleviates problems caused by a shortage of data in the target language. We apply XLTime with different language models and show that it outperforms the previous automatic SOTA methods on French, Spanish, Portuguese, and Basque, by large margins. It also closes the gap considerably on the handcrafted HeidelTime method.
Template-free Prompt Tuning for Few-shot NER
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Few-Shot Learning, Few-shot NER, Named Entity Recognition (NER), Sentence

Prompt-based methods have been successfully applied in sentence-level few-shot learning tasks, mostly owing to the sophisticated design of templates and label words. However, when applied to token-level labeling tasks such as NER, it would be time-consuming to enumerate the template queries over all potential entity spans. In this work, we propose a more elegant method to reformulate NER tasks as LM problems without any templates. Specifically, we discard the template construction process while maintaining the word prediction paradigm of pre-training models to predict a class-related pivot word (or label word) at the entity position. Meanwhile, we also explore principled ways to automatically search for appropriate label words that the pre-trained models can easily adapt to. While avoiding the complicated template-based process, the proposed LM objective also reduces the gap between different objectives used in pre-training and fine-tuning, thus it can better benefit the few-shot performance. Experimental results demonstrate the effectiveness of the proposed method over bert-tagger and template-based method under few-shot settings. Moreover, the decoding speed of the proposed method is up to 1930.12 times faster than the template-based method.
Reframing Human-AI Collaboration for Generating Free-Text Explanations
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using a small number of human-written examples (i.e., in a few-shot manner). We find that (1) higher-quality, human-authored prompts result in higher quality generations; and (2) surprisingly, in a head-to-head comparison, humans often prefer explanations generated by GPT-3 to crowdsourced explanations in existing datasets. Our human studies also show, however, that while models often produce factual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. Despite significant subjectivity intrinsic to judging acceptability, our approach is able to consistently filter GPT-3 generated explanations deemed acceptable by humans.
Penguins Don’t Fly: Reasoning about Generics through Instantiations and Exceptions
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Generic statements (e.g., Birds can fly) express generalizations about the world. However, generics are not universally true -- while sparrows and penguins are both birds, penguins can't fly. Understanding cases when a generic statement is true or false is crucial for machine reasoning. In this work, we present a novel framework to generate pragmatically relevant true and false instances of a generic.We use pre-trained language models, constraining the generation based on our computational framework, and produce ${\sim}20k$ \textsc{exemplars} for ${\sim}650$ generics. Our system outperforms few-shot generation from GPT-3 (by 12.5 precision points) and our analysis highlights the importance of constrained decoding for this task and the implications of generics \textsc{exemplars} for non-monotonic reasoning and NLI.
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Answer Generation, Decoder, Retrieval, Visual Question Answering (VQA)

The primary focus of recent work with large-scale transformers has been on optimizing the amount of information packed into the model's parameters.  In this work, we ask a complementary question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6\% absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. Additionally, explicit knowledge integration improves interpretability of model predictions in our analysis.