pwc-archive/papers-with-abstracts huggingface dataset:
Fast and Accurate Transformer-based Translation with Character-Level Encoding and Subword-Level Decoding
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Translation

The  Transformer translation model is fast to train and achieves state-of-the-art results for various translation tasks. However, unknown input words at test time remain a challenge for the  Transformer, especially when unknown words are segmented into inappropriate subword sequences that break morpheme boundaries. This paper improves the Transformer  model   to learn more accurate source representations via character-level encoding. We simply adopt character sequences instead of subword sequences as  input of the   standard  Transformer encoder and propose contextualized   character embedding (CCEmb) to help character-level encoding. Our CCEmb contains information about the current character and its context by adding the embeddings of its contextual character $n$-grams. 
The  CCEmb causes little extra computational cost  and we  show that our model with a  character-level encoder and a standard subword-level Transformer decoder can outperform   the original pure subword-level Transformer, especially for translating source sentences that contain unknown (or rare) words.
Analyzing Dynamic Adversarial Training Data in the Limit
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

To create models that are robust across a wide range of test inputs, training datasets should include diverse examples that span numerous phenomena. Dynamic adversarial data collection (DADC), where annotators craft examples that challenge continually improving models, holds promise as an approach for generating such diverse training sets. Prior work has shown that running DADC over 1-3 rounds can help models fix some error types, but it does not necessarily lead to better generalization beyond adversarial test data. We argue that running DADC over many rounds maximizes its training-time benefits, as the different rounds can together cover many of the task-relevant phenomena. We present the first study of longer-term DADC, where we collect 20 rounds of NLI examples for a small set of premise paragraphs, with both adversarial and non-adversarial approaches. Models trained on DADC examples make 26% fewer errors on our expert-curated test set compared to models trained on non-adversarial data. Our analysis shows that DADC yields examples that are more difficult, more lexically and syntactically diverse, and contain fewer annotation artifacts compared to non-adversarial examples.
Combining Paraphrase Pre-trained Model and Controllable Rules for Unsupervised Sentence Simplification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence, Text Simplification, Text Summarization

Although neural sequence-to-sequence models for sentence simplification achieve some progress, they still suffer from the data sparsity problem and are lack of controllability. This paper proposes a two-stage approach for text simplification. First, considering text simplification is closely related to text summarization and paraphrase, we fine-tune the pre-trained model on the dataset of summarization and paraphrase. Further, in order to achieve interpretation and controllablity, we design controllable scorers to evaluate the simplified sentence from three aspects: adequacy, fluency and simplicity, which are applied to sort the generated sentences and output the best one. Experiments show that our approach improves the previous best performance of the unsupervised model by a considerable margin of 5.53 points, achieving a new state-of-the-art result. Our method even performs competitively with supervised models in both automatic metrics and human evaluation.
Retrieval Data Augmentation Informed by Downstream Question Answering Performance
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, Question Answering, Retrieval

Training retrieval models to fetch contexts for Question Answering (QA) over large corpora requires labeling relevant passages in those corpora. Since obtaining exhaustive manual annotations of all relevant passages is not feasible, prior work uses text overlap heuristics to find passages that are likely to contain the answer, but this is not feasible when the task requires deeper reasoning and answers are not extractable spans (e.g.: multi-hop, discrete reasoning). We address this issue by identifying relevant passages based on whether they are useful for a trained QA model to arrive at the correct answers, and develop a search process guided by the QA model's loss. Our experiments show that this approach enables identifying relevant context for  unseen data greater than 90% of the time on the IIRC dataset and generalizes better to the end QA task than those trained on just the gold retrieval data on IIRC and QASC datasets.
Modeling Multi-hop Question Answering as Single Sequence Prediction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Answer Generation, Decoder, Generative Question Answering, Multi-hop Question Answering, Passage Retrieval, Prediction, Question Answering, Retrieval

Fusion-in-decoder (Fid) (Izacard and Grave, 2020) is a generative question answering (QA) model that leverages passage retrieval with a pre-trained transformer and pushed the state of the art on single-hop QA. However, the complexity of multi-hop QA hinders the effectiveness of the generative QA approach. In this work, we propose a simple generative approach (PathFid) that extends the task beyond just answer generation by explicitly modeling the reasoning process to resolve the answer for multi-hop questions. By linearizing the hierarchical reasoning path of supporting passages, their key sentences, and finally the factoid answer, we cast the problem as a single sequence prediction task. To facilitate complex reasoning with multiple clues, we further extend the unified flat representation of multiple input documents by encoding cross-passage interactions. Our extensive experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets: HotpotQA and IIRC. Besides the performance gains, PathFid is more interpretable, which in turn yields answers that are more faithfully grounded to the supporting passages and facts compared to the baseline Fid model.
Interpretability on clinical analysis from Pattern Disentanglement Insight
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: BIG-bench Machine Learning, Clustering, Disentanglement

Diagnosis of a clinical condition can help medical professionals save time in making a clinical diagnosis and prevent overlooking risks. Therefore we explore the problem using free-text medical notes recorded in electronic health records (EHR). MIMIC III is a de-identified EHR database containing observations from over 40,000 patients in critical care units. Since the text corpus is unstructured in the non-database table format, existing machine learning models may be ineffective at interpreting the results, which is often desirable for clinical diagnosis. Hence, in this paper, we proposed a text mining and pattern discovery solution to discover strong association patterns from patient discharge summaries and the code of international classification of diseases (ICD9-code). The proposed approach offers a straightforward interpretation of the underlying relation of patient characteristics in an unsupervised machine learning setting. The clustering results outperform the baseline clustering algorithm and are comparable to baseline supervised methods.
How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Morphological Inflection

Neural network models have achieved good performance on morphological inflection tasks, including English past tense inflection. However whether they can represent human cognitive mechanisms is still under debate. In this work, we examined transformer models with different training size to show that: 1) neural models correlate with both human behaviors and cognitive theories' predictions on nonce verbs; and the model with small-size training data that matches parents' input distribution has the highest correlation; 2) neural models make different types of errors on regular and irregular verbs, exhibiting a clear distinction between regulars and irregulars. Therefore, we conclude that neural networks have the potential to be good cognitive models for English past tense.
What Works and Doesn't Work, A Deep Decoder for Neural Machine Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Language Modelling, Machine Translation, NMT, Translation

Deep learning has demonstrated performance advantages in a wide range of natural language processing tasks, including neural machine translation (NMT). Transformer NMT models are typically strengthened by deeper encoder layers, but deepening their decoder layers usually results in failure. In this paper, we first identify the cause of the failure of the deep decoder in the Transformer model. Inspired by this discovery, we then propose approaches to improving it, with respect to model structure and model training, to make the deep decoder practical in NMT. Specifically, with respect to model structure, we propose a cross-attention drop mechanism to allow the decoder layers to perform their own different roles, to reduce the difficulty of deep-decoder learning. For model training, we propose a collapse reducing training approach to improve the stability and effectiveness of deep-decoder training. We experimentally evaluated our proposed Transformer NMT model structure modification and novel training methods on several popular machine translation benchmarks. The results showed that deepening the NMT model by increasing the number of decoder layers successfully prevented the deepened decoder from degrading to an unconditional language model. In contrast to prior work on deepening an NMT model on the encoder, our method can deepen the model on both the encoder and decoder at the same time, resulting in a deeper model and improved performance.
Seq2rel: A sequence-to-sequence-based approach for document-level relation extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: coreference-resolution, Coreference Resolution, Document-level Relation Extraction, Joint Entity and Relation Extraction, Relation, Relation Extraction, Sentence

Motivated by the fact that many relations cross the sentence boundary, there has been increasing interest in document-level relation extraction (RE). Document-level RE requires integrating information within and across sentences, capturing complex interactions between mentions of interacting entities. Most document-level RE methods proposed to date are pipeline-based, requiring entities as input. However, previous work has demonstrated that jointly learning to extract entities and relations can improve performance and be more efficient due to shared parameters and training steps. In this paper, we develop a sequence-to-sequence-based approach that can learn the sub-tasks of document-level RE --- entity extraction, coreference resolution and relation extraction --- in an end-to-end fashion. We evaluate our approach on several datasets, in some cases exceeding the performance of existing methods. Finally, we demonstrate that, under our model, the end-to-end approach outperforms a pipeline-based approach. Our code and models will be made publicly available.
Achieving Conversational Goals with Unsupervised Post-hoc Knowledge Injection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Informativeness, Specificity

A limitation of current neural dialog models is that they tend to suffer from a lack of specificity and informativeness in generated responses, primarily due to dependence on training data that covers a limited variety of scenarios and conveys limited knowledge. One way to alleviate this issue is to extract relevant knowledge from external sources at decoding time and incorporate it into the dialog response. In this paper, we propose a post-hoc knowledge-injection technique where we first retrieve a diverse set of relevant knowledge snippets conditioned on both the dialog history and an initial response from an existing dialog model. We construct multiple candidate responses, individually injecting each retrieved snippet into the initial response using a gradient-based decoding method, and then select the final response with an unsupervised ranking step. Our experiments in goal-oriented and knowledge-grounded dialog settings demonstrate that human annotators judge the outputs from the proposed method to be more engaging and informative compared to responses from prior dialog systems. We further show that knowledge-augmentation promotes success in achieving conversational goals in both experimental settings.
A Zero-Resource Approach to Cross-Lingual Query-Focused Abstractive Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Abstractive Text Summarization, Decoder, Translation

We present a novel approach for cross-lingual query-focused abstractive summarization (QFAS) that leverages the translate-then-summarize paradigm. We approach cross-lingual QFAS as a zero-resource problem and introduce a framework to create a synthetic QFAS corpus from a standard summarization corpus using a novel query-generation strategy. Our model summarizes documents in foreign languages for which translation quality is poor. It learns not only to identify and condense salient information relevant to a query, but also to appropriately rephrase grammatical errors and disfluencies that may occur in the noisy translations. Our technique enhances a pre-trained encoder-decoder transformer by introducing query focus to the encoder.  We show that our method for creating synthetic QFAS data leads to more robust models that not only achieve state-of-the-art performance on our corpus, but also perform better on out-of-distribution data as compared to prior work.
Cross-cultural Emotion Classification: the Effect of Emotional Intensity and Acoustic Features
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cultural Vocal Bursts Intensity Prediction, Emotion Classification, Emotion Recognition, Feature Importance

Cross-cultural emotion recognition is attracting increasing research attention; robustness to such differences in emotional expression is important for speech modality emotion recognition. In this work we quantify the accuracy loss when classifying cross-culturally for multiple emotional intensities, and investigate the effect of feature sets, including feature importance. We find that different emotional intensities yield a similar decrease in cross-culture accuracy relative to within-culture, and different acoustic feature sets also yield similar relative cross-culture accuracy. The top 10 important eGeMAPS features for within-cultural and cross-cultural classification share only one common feature, which partially explains differences in accuracy.
Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Answering, Semantic Parsing

Existing studies on semantic parsing focus on mapping a natural-language utterance to a logical form (LF) in one turn. However, because natural language may contain ambiguity and variability, this is a difficult challenge. In this work, we investigate an interactive semantic parsing framework that explains the predicted LF step by step in natural language and enables the user to make corrections through natural-language feedback for individual steps. We focus on question answering over knowledge bases (KBQA) as an instantiation of our framework, aiming to increase the transparency of the parsing process and help the user trust the final answer. We construct INSPIRED, a crowdsourced dialogue dataset derived from the ComplexWebQuestions dataset. Our experiments show that this framework has the potential to greatly improve overall parse accuracy. Furthermore, we develop a pipeline for dialogue simulation to evaluate our framework w.r.t. a variety of state-of-the-art KBQA models without further crowdsourcing effort. The results demonstrate that our framework promises to be effective across such models.
One-Shot Learning from a Demonstration with Hierarchical Latent Language
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Minecraft, One-Shot Learning

Humans have the capability, aided by the expressive compositionality of their language, to learn quickly by demonstration. They are able to describe unseen task-performing procedures and generalize their execution to other contexts. In this work, we introduce DescribeWorld, an environment designed to test this sort of generalization skill in grounded agents, where tasks are linguistically and procedurally composed of elementary concepts. The agent observes a single task demonstration in a Minecraft-like grid world, and is then asked to carry out the same task in a new map.To enable such a level of generalization, we propose a neural agent infused with hierarchical latent language—both at the level of task inference and subtask planning. Our agent first generates a textual description of the demonstrated unseen task, then leverages this description to replicate it. Through multiple evaluation scenarios and a suite of generalization tests, we find that agents that perform text-based inference are better equipped for the challenge under a random split of tasks.
Mitigating Gender Bias in Machine Translation through Adversarial Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Translation

Machine translation and other NLP systems often contain significant biases regarding sensitive attributes, such as gender or race, that worsen system performance and perpetuate harmful stereotypes. Recent preliminary research suggests that adversarial learning can be used as part of a model-agnostic bias mitigation method that requires no data modifications. However, adapting this strategy for machine translation and other modern NLP domains requires (1) restructuring training objectives in the context of fine-tuning pretrained large language models and (2) developing measures for gender or other protected variables for tasks in which these attributes must be deduced from the data itself.

We present an adversarial learning framework that addresses these challenges to mitigate gender bias in seq2seq machine translation. Our framework improves the disparity in translation quality for sentences with male vs. female entities by 86% for English-German translation and 91% for English-French translation, with minimal effect on translation quality. The results suggest that adversarial learning is a promising technique for mitigating gender bias in machine translation.
Structured Pruning Learns Compact and Accurate Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Model Compression

The growing size of neural language models has led to increased attention in model compression. Pruning methods start from a large model and gradually remove model weights---they can significantly reduce the model size but hardly achieve impressive runtime efficiency.
On the other hand, distillation methods start from a shallower, compact model and can obtain large speedups---however, they are costly to train on large amounts of unlabeled data. In this work, we show that structured pruning can match the distillation counterparts in both latency ($>$10$\times$) and accuracy ($>$92\%) and result in highly compact and efficient subnetworks. Unlike distillation, our task-specific pruning approach, {\ours}, does not need to pre-specify the model architecture nor rely on unlabeled data. Our solution is to jointly prune layers and sub-modules such as heads and hidden units in Transformer models through $l_0$ regularization while ensuring that the resulting model is parallelizable. We also propose a layerwise distillation approach to further guide pruning. Finally, our pruned structures reveal interesting patterns---for example, more than 70\% of feed-forward and 50\% of self-attention layers can be easily pruned, while the first and last 1-2 layers are likely to remain for highly compressed models.
Can Rationalization Improve Robustness?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence

A growing line of work has investigated the development of neural NLP models that can produce rationales---subsets of input that can explain their model predictions.
In this paper, we ask whether such rationale models can also provide robustness to adversarial attacks in addition to their interpretable nature. Since these models need to first generate rationales (``rationalizer'') before making predictions (``predictor''), they have the potential to ignore noise or adversarially added text by simply masking it out of the generated rationale. To this end, we systematically generate various types of `AddText' attacks for both token and sentence-level rationalization tasks, and perform an extensive empirical evaluation of state-of-the-art rationale models across five different tasks.
Our experiments reveal that the rationale models show the promise to improve robustness, while they struggle in certain scenarios---when the rationalizer is sensitive to positional bias or lexical choices of attack text. Further, leveraging human rationale as supervision does not always translate to better performance. Our study is a first step towards exploring the interplay between interpretability and robustness in the rationalize-then-predict framework.
Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Tracking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Dialogue State Tracking, Multi-domain Dialogue State Tracking, Transfer Learning

Dialogue State Tracking (DST) aims to keep track of users' intentions during the course of a conversation. In DST, modelling the relations among domains and slots is still an under-studied problem. Existing approaches that have considered such relations generally fall short in: (1) fusing prior slot-domain membership relations and dialogue-aware dynamic slot relations explicitly, and (2) generalizing to unseen domains. To address these issues, we propose a novel \textbf{D}ynamic \textbf{S}chema \textbf{G}raph \textbf{F}usion \textbf{Net}work (\textbf{DSGFNet}), which generates a dynamic schema graph to explicitly fuse the prior slot-domain membership relations and dialogue-aware dynamic slot relations. It also uses the schemata to facilitate knowledge transfer to new domains. DSGFNet consists of a dialogue utterance encoder, a schema graph encoder, a dialogue-aware schema graph evolving network, and a schema graph enhanced dialogue state decoder. Empirical results on benchmark datasets, including SGD, MultiWOZ2.1, and MultiWOZ2.2, show that DSGFNet outperforms the existing methods.