pwc-archive/papers-with-abstracts huggingface dataset:
Breaking Down Multilingual Machine Translation
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Machine Translation, Translation

While multilingual training is now an essential ingredient in machine translation (MT) systems, recent work has demonstrated that it has different effects in different multilingual settings, such as many-to-one, one-to-many, and many-to-many learning. These training settings expose the encoder and the decoder in a machine translation model with different data distributions. In this paper, we examine how different varieties of multilingual training contribute to learning these two components of the MT model. Specifically, we compare bilingual models with encoders and/or decoders initialized by multilingual training. We show that multilingual training is beneficial to encoders in general, while it only benefits decoders for low-resource languages (LRLs). We further find the important attention heads for each language pair and compare their correlations during inference. Our analysis sheds light on how multilingual translation models work and also enables us to propose methods to improve performance by training with highly related languages. Our many-to-one models for high-resource languages and one-to-many models
for LRL outperform the best results reported
by Aharoni et al. (2019).
Update Frequently, Update Fast: Retraining Semantic Parsing Systems in a Fraction of Time
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Semantic Parsing

Currently used semantic parsing systems deployed in voice assistants can require weeks to train.
Datasets for these models often receive small and frequent updates, data patches. Each patch requires training a new model. To reduce training time, one can fine-tune the previously trained model on each patch, but naive fine-tuning exhibits catastrophic forgetting -- degradation of the model performance on the data not represented in the data patch.

In this work, we propose a simple method that alleviates catastrophic forgetting and show that it is possible to match the performance of a model trained from scratch in less than 10\% of a time via fine-tuning. The key to achieving this is supersampling and EWC regularization. We demonstrate the effectiveness of our method on multiple splits of the Facebook TOP and SNIPS datasets.
Is Knowledge Embedding Fully Exploited in Language Understanding? An Empirical Study
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Knowledge Graph Completion, Knowledge Graphs, Natural Language Understanding

The recent development of knowledge embedding (KE) enables machines to represent knowledge graphs (KGs) with low-dimensional embeddings, which facilitates utilizing KGs for various downstream natural language understanding (NLU) tasks. However, less work has been done on systematically evaluating the impact of KE on NLU. In this work, we conduct a comprehensive analysis of utilizing KE on four downstream knowledge-driven NLU tasks using two representative knowledge-guided frameworks, including knowledge augmentation and knowledge attention. From the experimental results, we find that: (1) KE models that have better performance on knowledge graph completion do not necessarily help knowledge-driven NLU tasks better in the knowledge-guided frameworks; (2) KE could effectively benefit NLU tasks from two aspects including entity similarity and entity relation information; (3) KE could further benefit pre-trained language models which have already learned rich knowledge from pre-training. We hope the results could help and guide future studies to utilize KE in NLU tasks. Our source code will be released to support further exploration.
Causal Augmentation for Causal Sentence Classification
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Classification, counterfactual, Sentence, Sentence Classification

Scarcity of corpora with annotated causal texts can lead to poor robustness when training state-of-the-art language models for causal sentence classification. In particular, we find that these models misclassify on augmented sentences that have been negated or strengthened in terms of their causal meaning. This is worrying because minor linguistic changes in causal sentences can have disparate meanings. To resolve such issues, we propose to generate counterfactual causal sentences by creating contrast sets (Gardner et al., 2020). However, we notice an important finding that simply introducing edits is not sufficient to train models with counterfactuals. We thus introduce heuristics, like sentence shortening or multiplying key causal terms, to emphasize semantically important keywords to the model. We demonstrate these findings on different training setups and across two out-of-domain corpora. Our proposed mixture of augmented edits consistently achieves improved performance compared to baseline across two models and both within and out of corpus' domain, suggesting our proposed augmentation also helps the model generalize.
Faithful and Plausible Explanations of Medical Code Predictions
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Decision Making

Machine learning models that offer excellent predictive performance often lack the interpretability necessary to support integrated human machine decision-making. In clinical medicine and other high-risk settings, domain experts may be unwilling to trust model predictions without explanations. Work in explainable AI must balance competing objectives along two different axes: 1) Models should ideally be both accurate and simple. 2) Explanations must balance faithfulness to the model's decision-making with their plausibility to a domain expert.
We propose to train a proxy model that mimics the behavior of a trained model and provides control over these trade-offs. We evaluate our approach on the task of assigning ICD codes to clinical notes to demonstrate that the proxy model is faithful to the trained model's behavior and produces quality explanations.
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Open-Domain Question Answering, Question Answering

We study open-domain question answering with \emph{structured, unstructured} and \emph{semi-structured} knowledge sources, including text, tables, lists and knowledge bases.  
Departing from prior work, we propose a unifying approach that homogenizes all sources by reducing them to text and applies the retriever-reader model which has so far been limited to text sources only.
Our approach greatly improves the results on knowledge-base QA tasks by 11 points, compared to latest graph-based methods.
More importantly, we demonstrate that our \emph{unified knowledge} (\uniqa{}) model is a simple and yet effective way to combine heterogeneous sources of knowledge, advancing the state-of-the-art results on two popular question answering benchmarks, NaturalQuestions and WebQuestions, by 3.5 and 2.6 points, respectively.
LiSTra, Automatic Speech Translation: English to Lingala case study
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), automatic-speech-translation, Decoder, Machine Translation, speech-recognition, Speech Recognition, Translation

In recent years there have been great interests in addressing the low resourcefulness of African languages and provide baseline models for different Natural Language Processing tasks. Several initiatives on the continent use the Bible as a data source to provide proof of concept for some NLP tasks. In this work, we present the Lingala Speech Translation (LiSTra) dataset, release a full pipeline for the construction of such dataset in other languages, and report baselines using both the traditional cascade approach (Automatic Speech Recognition -> Machine Translation) and a revolutionary transformer-based End-2-End architecture with a custom interactive attention that allows information sharing between the recognition decoder and the translation decoder.
Subtopic Clustering with a Query-Specific Siamese Similarity Metric
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Clustering

We propose a Query-Specific Siamese Similarity Metric (QS3M) for query-specific clustering of text documents. It uses fine-tuned BERT embeddings and trains a non-linear projection into a query-specific similarity space. We build on the idea of Siamese networks but include a third component, a representation of the query. The empirical evaluation for clustering employs two TREC datasets with two different clustering benchmarks each. When used to obtain query-relevant clusters, QS3M achieves a 12% performance improvement over a recently published BERT-based reference method and significantly outperforms other unsupervised baselines.
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Natural Language Understanding

Despite the success of fine-tuning pretrained language encoders like BERT for downstream natural language understanding (NLU) tasks, it is still poorly understood how neural networks change after fine-tuning. In this work, we use centered kernel alignment (CKA), a method for comparing learned representations, to measure the similarity of representations in task-tuned models across layers. In experiments across twelve NLU tasks, we discover a consistent block diagonal structure in the similarity of representations within fine-tuned RoBERTa and ALBERT models, with strong similarity within clusters of earlier and later layers, but not between them. The similarity of later layer representations implies that later layers only marginally contribute to task performance, and we verify in experiments that the top few layers of fine-tuned Transformers can be discarded without hurting performance, even with no further tuning.
Neural Predictive Text for Grammatical Error Prevention
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Grammatical Error Correction, Language Modeling, Language Modelling

In this paper we study the potential of two neural language models, an LSTM and an autoregressive language model GPT-2, to predict possible correction tokens in erroneous sentences and to predict the next token in randomly sliced correct sentences, in the aim of establishing a new Grammatical Error Correction (GEC) subarea, for which we coin the term Grammatical Error Prevention (GEP). Systems that could assist in GEP, such as language models, are expected to predict elements and therefore prevent grammatical errors in advance. Our findings show that GPT-2 can predict 29% of the correct tokens with one prediction. Accuracy rises up to 44% when the top 3 predictions are considered. To test the pedagogical capacity of such a model, we also experimented with real English as a second language (ESL) learners. By equipping GPT-2 to generate text that functions as potential continuation of the learners' sentences, we created a small corpus of the learners' writings and analyzed their errors along with their frequencies.
QUASER: Question Answering with Scalable Extractive Rationalization
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Question Answering

Designing NLP models that produce predictions by first extracting a set of relevant input sentences (i.e., rationales), is gaining importance as a means to improving model interpretability and to producing supporting evidence for users. Current unsupervised approaches are trained to extract rationales that maximize prediction accuracy, which is invariably obtained by exploiting spurious correlations in datasets, and leads to unconvincing rationales. In this paper, we introduce unsupervised generative models to extract dual-purpose rationales, which must not only be able to support a subsequent answer prediction, but also support a reproduction of the input query. We show that such models can produce more meaningful rationales, that are less influenced by dataset artifacts, and as a result, also achieve the state-of-the-art on rationale extraction metrics on four datasets from the ERASER benchmark, significantly improving upon previous unsupervised methods.
Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Cross-Lingual Transfer, Domain Adaptation, Machine Translation, NMT, Translation

Adapter layers are lightweight, learnable units inserted between transformer layers. Recent work explores using such layers for neural machine translation (NMT), to adapt pre-trained models to new domains or language pairs.
We propose strategies to compose language and domain adapters. Our goals are both parameter-efficient adaptation to multiple domains and languages simultaneously, and cross-lingual transfer in domains where parallel data is unavailable for certain language pairs.
We find that a naive combination of domain-specific and language-specific adapters often results in translations into the wrong language. We study other ways to combine the adapters to alleviate this issue and maximize cross-lingual transfer.
With our best adapter combinations, we obtain improvements of 3-4 BLEU on average for source languages that do not have in-domain data. For target languages without in-domain data, we achieve a similar improvement by combining adapters with back-translation.
A Statistical Typology of (Textual) Language in Finer Granularity
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5

We propose a character-level perspective for a new understanding and visualization of language, in its textual representation in computing, using relative line length and character vocabulary size from parallel corpora as parameters. We discover an emergent pattern with a natural, continuous order to languages. We highlight some of the outlier languages and discuss the opportunities and challenges in line for character- and byte-level development in language technology.
Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Common Sense Reasoning, Open-Ended Question Answering, Task Planning

The ability to sequence unordered events is evidence of comprehension and reasoning about real world tasks/procedures, and is essential for applications such as task planning and multi-source instruction summarization.
It often requires thorough understanding of temporal common sense and multimodal information, since these procedures are often conveyed by a combination of texts and images.
While humans are capable of reasoning about and sequencing unordered procedural instructions, the extent to which the current machine learning methods possess such a capability is still an open question.
In this work, we benchmark models' capability of reasoning over and sequencing unordered multimodal instructions by curating datasets from online instructional manuals and collecting comprehensive human annotations.
We find current state-of-the-art models not only perform significantly worse than humans but also seem incapable of efficiently utilizing  multimodal information.
To improve machines' performance on multimodal event sequencing, we propose sequence-aware pretraining techniques exploiting the sequential alignment properties of both texts and images, resulting in > 5% improvements on perfect match ratio.
SumHiS: Extractive Summarization Exploiting Hidden Sctructure
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Clustering, Extractive Summarization

Extractive summarization is a task of highlighting the most important parts of the text. We introduce a new approach to extractive summarization task using hidden clustering structure of the text. Experimental results on CNN/DailyMail demonstrate that our approach generates more accurate summaries than both extractive and abstractive methods, achieving state-of-the-art results in terms of ROUGE-2 metric exceeding the previous approaches by 10\%. Additionally, we show that hidden structure of the text could be interpreted as aspects.
A Simple Information-Based Approach to Unsupervised Domain-Adaptive Aspect-Based Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Sentiment Analysis, Term Extraction

Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis task which aims to extract the aspects from sentences and identify their corresponding sentiments. Aspect term extraction (ATE) is the crucial step for ABSA. Due to the expensive annotation for aspect terms, we often lack labeled target domain data for fine-tuning. To address this problem, many approaches have been proposed recently to transfer common knowledge in an unsupervised way, but such methods have too many modules and require expensive multi-stage preprocessing. In this paper, we propose a simple but effective technique based on mutual information maximization, which can serve as an additional component to enhance any kind of model for cross-domain ABSA and ATE. Furthermore, we provide some analysis of this approach. Experiment results show that our proposed method outperforms the state-of-the-art methods for cross-domain ABSA by 4.32\% Micro-F1 on average over 10 different domain pairs. Apart from that, our method can be extended to other sequence labeling tasks, such as named entity recognition (NER). Codes will be released.
e-CARE: a New Dataset for Exploring Explainable Causal Reasoning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: valid

Understanding causality has vital importance for various Natural Language Processing (NLP) applications. Beyond the labeled instances, conceptual explanations of the causality can provide deep understanding of the causal fact to facilitate the causal reasoning process. However, such explanation information still remains absent in existing causal reasoning resources. In this paper, we fill this gap by presenting a human-annotated explainable CAusal REasoning dataset (e-CARE), which contains over 20K causal reasoning questions, together with natural language formed explanations of the causal questions. 
Experimental results show that generating valid explanations for causal facts still remains especially challenging for the state-of-the-art models, and the explanation information can be helpful for promoting the accuracy and stability of causal reasoning models.
Mukayese: Turkish NLP Strikes Back
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Benchmarking, Language Modeling, Language Modelling, Sentence, Sentence segmentation

Having sufficient resources for a language X lifts it from the $\textit{under-resourced}$ languages class, but does not necessarily lift it from the $\textit{under-researched}$ class. In this paper, we address the problem of the absence of organized benchmarks in the Turkish language. We demonstrate that languages such as Turkish are left behind the State-of-the-Art in NLP applications. As a solution, we present Mukayese, a set of NLP benchmarks for the Turkish language that contains several NLP tasks. For each benchmark, we work on one or more datasets and present two or more baselines. Moreover, we present four new benchmarking datasets in Turkish for language modeling, sentence segmentation, and spellchecking and correction.
Toward More Meaningful Resources for Lower-resourced Languages
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

In this paper, we describe our perspective on how meaningful resources for lower-resourced languages can be developed in connection with the speakers of those languages. We examine two massively multilingual resources in detail. We explore the contents of the names stored in Wikidata for a few lower-resourced languages and find that many of them are not in fact in the languages they claim to be and require non-trivial effort to correct. We discuss quality issues present in WikiAnn and evaluate whether it is a useful supplement to hand annotated data. We then discuss the importance of creating annotation for lower-resourced languages in a thoughtful and ethical way that includes the languages' speakers as part of the development process. We conclude with recommended guidelines for resource development.
Unified NMT models for the Indian subcontinent transcending script-barriers
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, NMT, Translation

Highly accurate machine translation systems are very important in societies and countries where multilinguality is very common, and where English often does not suffice. The Indian subcontinent is such a region, with all the Indic languages currently being under-represented in the NLP ecosystem. It is essential to advance the state-of-the-art of such low-resource languages atleast by using whatever data is available in open-source, which itself is something not very explored in the Indic ecosystem. In our work, we focus on improving the performance of very-low-resource Indic languages, especially of countries in addition to India. Specifically, we propose how unified models can be built that can exploit the data from comparatively resource-rich languages of the same region. We propose strategies to unify different types of unexplored scripts, especially Perso-Arabic scripts and Indic scripts to build multilingual models for all the Indic languages despite the script barrier. We also study how augmentation techniques like back-translation can be made use-of to build unified models that achieve state-of-the-art result among open source models, especially just using openly available raw data.
OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Transfer, Machine Translation, Retrieval, Sentence, Sentence Retrieval, Zero-Shot Cross-Lingual Transfer

Aligning parallel sentences in multilingual corpora is essential to curate data for downstream applications such as Machine Translation. In this work, we present OneAligner, an alignment model specially designed for sentence retrieval tasks. This model is able to train on only one language pair and transfers, in a cross-lingual fashion, to low-resource language pairs with negligible degradation in performance. When trained with all language pairs of a large-scale parallel multilingual corpus (OPUS-100), this model achieves the state-of-the-art result on the Tateoba dataset, outperforming an equally-sized previous model by $8.0$ points in accuracy while using less than $0.6\%$ of their parallel data. When finetuned on a single rich-resource language pair, be it English-centered or not, our model is able to match the performance of the ones finetuned on all language pairs under the same data budget with less than $2.0$ points decrease in accuracy. Furthermore, with the same setup, scaling up the number of rich-resource language pairs monotonically improves the performance, reaching a minimum of $0.4$ points discrepancy in accuracy, essentially obviating the need to collect any low-resource parallel data. Finally, we conclude through empirical results and analyses that the performance on the retrieval tasks depends mostly on the monolingual and parallel data size, up to a certain size threshold, rather than on what language pairs are used for training or evaluation.