pwc-archive/papers-with-abstracts huggingface dataset:
Multi-way VNMT for UGC: Improving Robustness and Capacity via Mixture Density Networks
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, Transfer Learning, Translation

This work presents a novel Variational Neural Machine Translation (VNMT) architecture with enhanced robustness properties, which we investigate through a detailed case-study addressing noisy French user-generated content (UGC) translation to English. We show that the proposed model, with results comparable or superior to state-of-the-art VNMT, improves performance over UGC translation in a zero-shot evaluation scenario while keeping optimal translation scores on in-domain test sets. We elaborate on such results by visualizing and explaining how neural learning representations behave when processing UGC noise. In addition, we show that VNMT enforces robustness to the learned embeddings, which can be later used for robust transfer learning approaches.
Pruning Adatperfusion with Lottery Ticket Hypothesis
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Transfer Learning

Pre-trained language models have shown great success in multiple downstream tasks. However, they are computationally expensive to fine-tune. Thus, transfer learning with adapter modules has been introduced to alleviate this problem, helping to extract knowledge of the downstream tasks. And the latest Adapterfusion model can further merge multiple adapters to incorporate knowledge from different tasks. However, merging multiple adapters will inevitably cause redundancies, increasing the training and inference time massively. Therefore, in this paper, we propose an approach to identify the influence of each adapter module and a novel way to prune adapters based on the prestigious Lottery Ticket Hypothesis. Experiments on GLUE datasets show that the pruned Adapterfusion model with our scheme can achieve state-of-the-art results, reducing sizes significantly while keeping performance intact.
Towards a Fast Response Selection: Selecting the Optimal Dialogue Response Once for All
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: All

Response selector, as an essential component of dialogue systems, aims to pick out an optimal response in a candidate pool to continue the dialogue. The current state-of-the-art methods are mainly based on an encoding paradigm called Cross-Encoder, which separately encodes each context-response pair and ranks the responses according to their fitness scores. However, such a paradigm is both inefficient and ineffective. Specifically, it has to repeatedly encode the same context for each response, which results in heavy inference cost. Also, without considering the relationship among the candidates, it is difficult to tell which one is the best candidate purely based on the fitness score of each candidate. To address this problem, we propose a new model called Panoramic-Encoder, which accepts all candidates and the context as inputs at once and allows them to interact with each other through a specially designed attention mechanism. Our method also allows us to naturally integrate some of the effective training techniques, such as the in-batch negative training. Extensive experiments across four benchmark datasets show that our new method significantly outperforms the current state-of-the-art while achieving approximately 3X speed-up at inference time.
FaiRR: Faithful and Robust Deductive Reasoning over Natural Language
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Fact Selection, Logical Reasoning

Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in natural language. Recent works show that such models can also produce the reasoning steps (i.e., the proof graph) that emulate the model's logical reasoning process. Currently, these black-box models generate both the proof graph and intermediate inferences within the same model and thus may be unfaithful. In this work, we frame the deductive logical reasoning task by defining three modular components: rule selection, fact selection, and knowledge composition. The rule and fact selection steps select the candidate rule and facts to be used and then the knowledge composition combines them to generate new inferences. This ensures model faithfulness by assured causal relation from the proof step to the inference reasoning. To test our framework, we propose FaiRR (Faithful and Robust Reasoner) where the above three components are independently modeled by transformers. We observe that FaiRR is robust to novel language perturbations, and is faster at inference than previous works on existing reasoning datasets. Additionally, in contrast to black-box generative models, the errors made by FaiRR are more interpretable due to the modular approach.
RGL: A Simple yet Effective Relation Graph Augmented Prompt-based Tuning Approach for Few-Shot Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Few-Shot Learning, Link Prediction, Natural Language Understanding, Node Classification, Relation

Pre-trained language models (PLMs) which carry generic knowledge can be a good starting point for adapting to downstream applications. However, it is difficult to generalize PLMs to new tasks with only a limited number of labeled samples given. In this work, we show that Relation Graph augmented Learning RGL method can obtain better performance in few-shot natural language understanding tasks. During learning, RGL constructs a relation graph based on the label consistency between samples in the same batch, and learns to solve the resultant node classification and link prediction problems of the relation graphs. In this way, RGL fully exploits the limited supervised information, which can boost the tuning effectiveness. Extensive experiments on benchmark tasks show that RGL consistently improve the performance of prompt-based tuning strategies.
Reinforcement Learning with Large Action Spaces for Neural Machine Translation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, NMT, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Semantic Similarity, Semantic Textual Similarity, Translation

Applying Reinforcement learning (RL) following pre-training is a versatile method for enhancing neural machine translation (NMT) performance. However, recent work has argued that the gains produced by RL for NMT are mostly due to promoting tokens that have already received a fairly high probability in pre-training. We hypothesize that the large action space is a main obstacle to RL's effectiveness in MT, and conduct two sets of experiments that lend support to our hypothesis, focusing on low-resource settings. First, we find that reducing the size of the vocabulary improves RL's effectiveness. Second, we find that effectively reducing the dimension of the action space without changing the vocabulary also yields notable improvement as evaluated by BLEU, semantic similarity, andhuman evaluation. Indeed, by replacing the network's final fully connected layer (that maps the network's internal dimension to the vocabulary dimension), with a layer that generalizes over similar actions, we obtain a substantial improvement in RL performance.
Unlearnable Text for Neural Classifiers
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Classification, Gender Classification, text-classification, Text Classification, Topic Classification

Neural text classification models are known to001explore  statistical patterns during  supervised002learning.  However, such patterns include spurious patterns and superficial regularity in the004training data.  In this paper, we exaggerate superficial regularity in the text to prevent unau-006thorized exploration of personal data.007We  propose a  gradient-based method to construct text modifications, which can make deep009neural  networks  (DNNs)  unlearnable.We010then  analyze text modifications exposed by the gradient-based method and further propose012two  simple hypotheses to manually craft unlearnable text. Experiments on four tasks (sen-014timent classification, topic classification, read-015ing comprehension and gender classification validate the effectiveness of our method,  by which these hypotheses achieve almost  un-018trained performance after training on unlearn-019able text.
Data Augmentation for Low-Resource Dialogue Summarization
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Data Augmentation, Meeting Summarization

We present DADS, a novel Data Augmentation technique for low-resource Dialogue Summarization. Our method generates synthetic examples by replacing sections of text from both the input dialogue and summary while preserving the augmented summary to correspond to a viable summary for the augmented dialogue. We utilize pretrained language models that produce highly likely dialogue alternatives while still being free to generate diverse alternatives. We applied our data augmentation method to the SAMSum dataset in low resource scenarios, mimicking real world problems such as chat, thread, and meeting summarization where large scale supervised datasets with human-written summaries are scarce. Through both automatic and human evaluations, we show that DADS shows strong improvements for low resource scenarios while generating topically diverse summaries without introducing additional hallucinations to the summaries.
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling, Natural Language Understanding

In the age of large transformer language models, linguistic benchmarks play an important role in diagnosing models' abilities and limitations on natural language understanding. However, current benchmarks show some significant shortcomings. In particular, they do not provide insight into how well a language model captures distinct linguistic phenomena essential for language understanding and reasoning. In this paper, we introduce Curriculum, a new large-scale NLI benchmark for evaluation on broad-coverage linguistic phenomena. We show that our benchmark for linguistic phenomena serves as a more difficult challenge for current state-of-the-art models. Our experiments also provide insight into the limitation of existing benchmark datasets. In addition, we find that sequential training on selected linguistic phenomena effectively improves generalizing performance on adversarial NLI under limited training examples.
Rethinking Style Transformer by Energy-based Interpretation: Adversarial Unsupervised Style Transfer using Pretrained Model
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling, Style Transfer, Text Style Transfer

Style control, content preservation, and fluency determine the quality of text style transfer models. To train on a nonparallel corpus, several existing approaches aim to deceive the style discriminator with an adversarial loss. However, adversarial training significantly degrades fluency compared to the other two metrics. In this work, we explain this phenomenon with the energy-based interpretation and leverage a pretrained language model to improve fluency. Specifically, we propose a novel approach of applying the pretrained language model to the text style transfer framework by restructuring the discriminator and the model itself, allowing the generator and the discriminator to also take advantage of the power of the pretrained model. We evaluate our model on four public benchmarks Amazon, Yelp, GYAFC, and Civil Comments and achieve state-of-the-art performance on the overall metrics.
Neural Discourse Deixis Resolution in Dialogue
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

We adapt Lee at el.'s (2018) span-based entity coreference model to the task of discourse deixis resolution. The resulting model achieves state-of-the-art results on the four datasets in the CODI-CRAC 2021 shared task.
Learning Sense Embeddings from Definitions in Dictionaries
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Re-Ranking

We introduce a method for learning to embed word senses as defined in a given set of given dictionaries. In our approach, sense definition pairs, <word, definition> are transformed into low-dimension vectors aimed at maximizing the probability of reconstructing the definitions in an autoencoding setting. The method involves automatically training sense autoencoder for encoding sense definitions, automatically aligning sense definitions, and automatically generating embeddings of arbitrary description. At run-time, queries from users are mapped to the embedding space and re-ranking is performed on the sense definition retrieved. We present a prototype sense definition embedding, SenseNet, that applies the method to two dictionaries. Blind evaluation on a set of real queries shows that the method significantly outperforms a baseline based on the Lesk algorithm. Our methodology clearly supports combining multiple dictionaries resulting in additional improvement in representing sense definitions in dictionaries.
Learning to Embed Multi-Modal Contexts for Situated Conversational Agents
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: coreference-resolution, Coreference Resolution, Decoder, dialog state tracking, Response Generation, Retrieval

The Situated Interactive Multi-Modal Conversations (SIMMC) 2.0 aims to create virtual shopping assistants that can accept complex multi-modal inputs, i.e. visual appearances of objects and user utterances. It consists of four subtasks, multi-modal disambiguation (MM-Disamb), multi-modal coreference resolution (MM-Coref), multi-modal dialog state tracking (MM-DST), and response retrieval and generation. While many task-oriented dialog systems usually tackle each subtask separately, we propose a jointly learned multi-modal encoder-decoder that incorporates visual inputs and performs all four subtasks at once for efficiency. This approach won the MM-Coref and response retrieval subtasks and nominated runner-up for the remaining subtasks using a single unified model at the 10th Dialog Systems Technology Challenge (DSTC10), setting a high bar for the novel task of multi-modal task-oriented dialog systems.
Don’t Forget About Pronouns: Removing Gender Bias in Language Models without Losing Factual Gender Information
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling, Text Generation

The representations in large language models contain various types of gender information. We focus on two types of such signals in English texts: factual gender information, which is a grammatical or semantic property, and gender bias, which is the correlation between a word and specific gender. We can disentangle the model’s embeddings and identify components encoding both information with probing. We aim to diminish the representation of stereotypical bias while preserving factual gender signal. Our filtering method shows that it is possible to decrease the bias of gender-neutral profession names without deteriorating language modeling capabilities. The findings can be applied to language generation and understanding to mitigate reliance on stereotypes while preserving gender agreement in coreferences.
FAQ Search using Transformers
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Question Answering

Many websites have bots as a guiding agent, for answering FAQ questions or directing users to human support. Many of them already have a curated FAQ page that can be used to bootstrap these bots. In this paper, we want to tackle a real-world problem of question answering for Bots. Given a user query, the system needs to pick the most relevant answer from a data source such as FAQ or Manuals. So, the ranking system needs to consider not just the passage but also the provided support questions or titles. This technique also provides the flexibility to add and delete support questions to continuously improve bot's quality, suggestions can be provided by system and the bot developer has control over their data instead of a black box system. We explore novel techniques to improve the results on a few public sets and on our own judged real user data. For the paper, We limit our experiments to transformers since it has proven to be significantly better in all question answering tasks. We show that significant gains can be observed using an extra segment embedding as well as pre-training new separators in transformers.
JEFF - Just Another EFFicient Reading Comprehension Test Generation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Question Generation, Question-Generation, Reading Comprehension

We introduce a method for generating vocabulary questions on reading comprehension of a given English article. In our approach, the method involves selecting target words in the given English article, finding synonyms as answer keys, and generating seemingly reasonable words in context as distractors. At run-time, some target words in the inputted article will be identified as questions, and automatically generating one answer key and three distractors.  We present a AQG (automatic question generation) system, JEFF, that applies the method to generate questions automatically. Evaluation on a set of questions generated by JEFF shows that the method is close to the human-designed ones.
What Role Does BERT Play in the Neural Machine Translation Encoder?
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, NMT, Translation

Pre-trained language models have been widely applied in various natural language processing tasks. But when it comes to neural machine translation, things are a little different. The differences between the embedding spaces created by BERT and NMT encoder may be one of the main reasons for the difficulty of integrating pre-trained LMs into NMT models. Previous studies illustrate the best way of integration is introducing the output of BERT into the encoder with some extra modules. Nevertheless, it is still unrevealed whether these additional modules will affect the embedding spaces created by the NMT encoder or not and what kind of information the NMT encoder takes advantage of from the output of BERT. In this paper, we start by comparing the changes of embedding spaces after introducing BERT into the NMT encoder trained on different machine translation tasks. Although the changing trends of these embedding spaces vary, introducing BERT into the NMT encoder will not affect the space of the last layer significantly. Subsequent evaluation on several semantic and syntactic tasks proves the NMT encoder is facilitated by the rich syntactic information contained in the output of BERT to boost the translation quality.
Quantifying Synthesis and Fusion and their Impact on Machine Translation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Diversity, Machine Translation, Translation

Theoretical work in morphological typology offers the possibility of measuring morphological diversity on a continuous scale. However, literature in NLP typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the theoretical rigidity of such claims, by quantifying the morphological typology at the word and segment level. We consider Payne (2017)'s approach to classify morphology using two indices: synthesis (from 1 for analytic to 3 or more for polysynthetic) and fusion (from 0 for agglutinative to 1 for fusional). For computing synthesis, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study.Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish, and verbs in English-Spanish) and segment level (previous language pairs plus English-German in both directions). We complement the word-level analysis with human evaluation, and overall, we observe a consistent impact of both indexes on machine translation quality.
Deep Speech Synthesis from Articulatory Features
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Speech Synthesis

In the articulatory synthesis task, speech is synthesized from input features containing information about the physical behavior of the human vocal tract. This task provides a promising direction for speech synthesis research, as the articulatory space is compact, smooth, and interpretable. Current works have highlighted the potential for deep learning models to perform  articulatory synthesis. However, it remains unclear whether these models can achieve the efficiency and fidelity of the human speech production system. To help bridge this gap, we propose a time-domain articulatory synthesis methodology and demonstrate its efficacy with both electromagnetic articulography (EMA) and synthetic articulatory feature inputs. Our model is both computationally efficient and highly intelligible, achieving a transcription word error rate (WER) of 7.14\% for the EMA-to-speech task. Through interpolation experiments, we also highlight the generalizability and interpretability of our approach.
Generating a Temporally Coherent Visual Story by Multimodal Recurrent Transformers
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Image Generation, Sentence, Story Visualization, Text to Image Generation, Text-to-Image Generation

Story visualization is a challenging text-to-image generation task for the difficulty of rendering visual details from abstract text descriptions. Besides the difficulty of image generation, the generator also needs to conform to the narrative of a multi-sentence story input. While prior arts in this domain have focused on improving semantic relevance between generated images and input text, controlling the generated images to be temporally consistent still remains a challenge. Moreover, existing generators are trained on single text-image pairs and fail to consider the variations of natural language captions that can describe a given image, causing poor model generalization. To address such problems, we leverage a cyclic training methodology involving pseudo-text descriptions as an intermediate step that decouples the image’s visual appearance from the variations of natural language descriptions. Additionally, to generate a semantically coherent image sequence, we consider an explicit memory controller which can augment the temporal coherence of images in the multi-modal autoregressive transformer. To sum up all components, we call it Cyclic Story visualization by MultimodAl Recurrent Transformers or C-SMART for short. Our method generates high-resolution, high-quality images, outperforming prior works by a significant margin across multiple evaluation metrics on the Pororo-SV dataset.