pwc-archive/papers-with-abstracts huggingface dataset:
IST-Unbabel Participation in the WMT20 Quality Estimation Shared Task
Authors: João Moura, Miguel Vera, Daan van Stigt, Fabio Kepler, André F. T. Martins
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Translation

We present the joint contribution of IST and Unbabel to the WMT 2020 Shared Task on Quality Estimation. Our team participated on all tracks (Direct Assessment, Post-Editing Effort, Document-Level), encompassing a total of 14 submissions. Our submitted systems were developed by extending the OpenKiwi framework to a transformer-based predictor-estimator architecture, and to cope with glass-box, uncertainty-based features coming from neural machine translation systems.
TMUOU Submission for WMT20 Quality Estimation Shared Task
Authors: Akifumi Nakamachi, Hiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: regression, Sentence

We introduce the TMUOU submission for the WMT20 Quality Estimation Shared Task 1: Sentence-Level Direct Assessment. Our system is an ensemble model of four regression models based on XLM-RoBERTa with language tags. We ranked 4th in Pearson and 2nd in MAE and RMSE on a multilingual track.
NICT Kyoto Submission for the WMT’20 Quality Estimation Task: Intermediate Training for Domain and Task Adaptation
Authors: Raphael Rubino
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Domain Adaptation, Language Modeling, Language Modelling, Machine Translation, Self-Supervised Learning, Sentence, Task 2, Translation

This paper describes the NICT Kyoto submission for the WMT’20 Quality Estimation (QE) shared task. We participated in Task 2: Word and Sentence-level Post-editing Effort, which involved Wikipedia data and two translation directions, namely English-to-German and English-to-Chinese. Our approach is based on multi-task fine-tuned cross-lingual language models (XLM), initially pre-trained and further domain-adapted through intermediate training using the translation language model (TLM) approach complemented with a novel self-supervised learning task which aim is to model errors inherent to machine translation outputs. Results obtained on both word and sentence-level QE show that the proposed intermediate training method is complementary to language model domain adaptation and outperforms the fine-tuning only approach.
HW-TSC’s Participation at WMT 2020 Quality Estimation Shared Task
Authors: Minghan Wang, Hao Yang, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Lizhi Lei, Ying Qin, Shimin Tao, Shiliang Sun, Yimeng Chen, Liangyou Li
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Sentence, Transfer Learning

This paper presents our work in the WMT 2020 Word and Sentence-Level Post-Editing Quality Estimation (QE) Shared Task. Our system follows standard Predictor-Estimator architecture, with a pre-trained Transformer as the Predictor, and specific classifiers and regressors as Estimators. We integrate Bottleneck Adapter Layers in the Predictor to improve the transfer learning efficiency and prevent from over-fitting. At the same time, we jointly train the word- and sentence-level tasks with a unified model with multitask learning. Pseudo-PE assisted QE (PEAQE) is proposed, resulting in significant improvements on the performance. Our submissions achieve competitive result in word/sentence-level sub-tasks for both of En-De/Zh language pairs.
Tencent submission for WMT20 Quality Estimation Shared Task
Authors: Haijiang Wu, Zixuan Wang, Qingsong Ma, Xinjie Wen, Ruichen Wang, Xiaoli Wang, Yulin Zhang, Zhipeng Yao, Siyao Peng
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Sentence, Task 2, Translation

This paper presents Tencent’s submission to the WMT20 Quality Estimation (QE) Shared Task: Sentence-Level Post-editing Effort for English-Chinese in Task 2. Our system ensembles two architectures, XLM-based and Transformer-based Predictor-Estimator models. For the XLM-based Predictor-Estimator architecture, the predictor produces two types of contextualized token representations, i.e., masked XLM and non-masked XLM; the LSTM-estimator and Transformer-estimator employ two effective strategies, top-K and multi-head attention, to enhance the sentence feature representation. For Transformer-based Predictor-Estimator architecture, we improve a top-performing model by conducting three modifications: using multi-decoding in machine translation module, creating a new model by replacing the transformer-based predictor with XLM-based predictor, and finally integrating two models by a weighted average. Our submission achieves a Pearson correlation of 0.664, ranking first (tied) on English-Chinese.
NLPRL System for Very Low Resource Supervised Machine Translation
Authors: Rupjyoti Baruah, Rajesh Kumar Mundotiya, Amit Kumar, Anil Kumar Singh
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Translation

This paper describes the results of the system that we used for the WMT20 very low resource (VLR) supervised MT shared task. For our experiments, we use a byte-level version of BPE, which requires a base vocabulary of size 256 only. BPE based models are a kind of sub-word models. Such models try to address the Out of Vocabulary (OOV) word problem by performing word segmentation so that segments correspond to morphological units. They are also reported to work across different languages, especially similar languages due to their sub-word nature. Based on BLEU cased score, our NLPRL systems ranked ninth for HSB to GER and tenth in GER to HSB translation scenario.
Low-Resource Translation as Language Modeling
Authors: Tucker Berckmann, Berkan Hiziroglu
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Decoder, Language Modeling, Language Modelling, Machine Translation, Translation

We present our submission to the very low resource supervised machine translation task at WMT20. We use a decoder-only transformer architecture and formulate the translation task as language modeling. To address the low-resource aspect of the problem, we pretrain over a similar language parallel corpus. Then, we employ an intermediate back-translation step before fine-tuning. Finally, we present an analysis of the system’s performance.
UdS-DFKI@WMT20: Unsupervised MT and Very Low Resource Supervised MT for German-Upper Sorbian
Authors: Sourav Dutta, Jesujoba Alabi, Saptarashmi Bandyopadhyay, Dana Ruiter, Josef van Genabith
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Translation, Unsupervised Machine Translation

This paper describes the UdS-DFKI submission to the shared task for unsupervised machine translation (MT) and very low-resource supervised MT between German (de) and Upper Sorbian (hsb) at the Fifth Conference of Machine Translation (WMT20). We submit systems for both the supervised and unsupervised tracks. Apart from various experimental approaches like bitext mining, model pre-training, and iterative back-translation, we employ a factored machine translation approach on a small BPE vocabulary.
Data Selection for Unsupervised Translation of German–Upper Sorbian
Authors: Lukas Edman, Antonio Toral, Gertjan van Noord
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Translation, Unsupervised Machine Translation

This paper describes the methods behind the systems submitted by the University of Groningen for the WMT 2020 Unsupervised Machine Translation task for German–Upper Sorbian. We investigate the usefulness of data selection in the unsupervised setting. We find that we can perform data selection using a pretrained model and show that the quality of a set of sentences or documents can have a great impact on the performance of the UNMT system trained on it. Furthermore, we show that document-level data selection should be preferred for training the XLM model when possible. Finally, we show that there is a trade-off between quality and quantity of the data used to train UNMT systems.
The LMU Munich System for the WMT20 Very Low Resource Supervised MT Task
Authors: Jindřich Libovický, Viktor Hangya, Helmut Schmid, Alexander Fraser
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Transfer Learning, Translation

We present our systems for the WMT20 Very Low Resource MT Task for translation between German and Upper Sorbian. For training our systems, we generate synthetic data by both back- and forward-translation. Additionally, we enrich the training data with German-Czech translated from Czech to Upper Sorbian by an unsupervised statistical MT system incorporating orthographically similar word pairs and transliterations of OOV words. Our best translation system between German and Sorbian is based on transfer learning from a Czech-German system and scores 12 to 13 BLEU higher than a baseline system built using the available parallel data only.
NRC Systems for Low Resource German-Upper Sorbian Machine Translation 2020: Transfer Learning with Lexical Modifications
Authors: Rebecca Knowles, Samuel Larkin, Darlene Stewart, Patrick Littell
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Transfer Learning, Translation

We describe the National Research Council of Canada (NRC) neural machine translation systems for the German-Upper Sorbian supervised track of the 2020 shared task on Unsupervised MT and Very Low Resource Supervised MT. Our models are ensembles of Transformer models, built using combinations of BPE-dropout, lexical modifications, and backtranslation.
The University of Helsinki and Aalto University submissions to the WMT 2020 news and low-resource translation tasks
Authors: Yves Scherrer, Stig-Arne Grönroos, Sami Virpioja
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Multi-Task Learning, Translation

This paper describes the joint participation of University of Helsinki and Aalto University to two shared tasks of WMT 2020: the news translation between Inuktitut and English and the low-resource translation between German and Upper Sorbian. For both tasks, our efforts concentrate on efficient use of monolingual and related bilingual corpora with scheduled multi-task learning as well as an optimized subword segmentation with sampling. Our submission obtained the highest score for Upper Sorbian -> German and was ranked second for German -> Upper Sorbian according to BLEU scores. For English–Inuktitut, we reached ranks 8 and 10 out of 11 according to BLEU scores.
The NITS-CNLP System for the Unsupervised MT Task at WMT 2020
Authors: Salam Michael Singh, Thoudam Doren Singh, Sivaji Bandyopadhyay
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Decoder, Machine Translation, Translation, Unsupervised Machine Translation

We describe NITS-CNLP’s submission to WMT 2020 unsupervised machine translation shared task for German language (de) to Upper Sorbian (hsb) in a constrained setting i.e, using only the data provided by the organizers. We train our unsupervised model using monolingual data from both the languages by jointly pre-training the encoder and decoder and fine-tune using backtranslation loss. The final model uses the source side (de) monolingual data and the target side (hsb) synthetic data as a pseudo-parallel data to train a pseudo-supervised system which is tuned using the provided development set(dev set).
Adobe AMPS’s Submission for Very Low Resource Supervised Translation Task at WMT20
Authors: Keshaw Singh
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Translation

In this paper, we describe our systems submitted to the very low resource supervised translation task at WMT20. We participate in both translation directions for Upper Sorbian-German language pair. Our primary submission is a subword-level Transformer-based neural machine translation model trained on original training bitext. We also conduct several experiments with backtranslation using limited monolingual data in our post-submission work and include our results for the same. In one such experiment, we observe jumps of up to 2.6 BLEU points over the primary system by pretraining on a synthetic, backtranslated corpus followed by fine-tuning on the original parallel training data.
On the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation
Authors: Sheila Castilho
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, Sentence, Translation

Document-level evaluation of machine translation has raised interest in the community especially since responses to the claims of “human parity” (Toral et al., 2018; Läubli et al., 2018) with document-level human evaluations have been published. Yet, little is known about best practices regarding human evaluation of machine translation at the document-level. This paper presents a comparison of the differences in inter-annotator agreement between quality assessments using sentence and document-level set-ups. We report results of the agreement between professional translators for fluency and adequacy scales, error annotation, and pair-wise ranking, along with the effort needed to perform the different tasks. To best of our knowledge, this is the first study of its kind.
How Should Markup Tags Be Translated?
Authors: Greg Hanneman, Georgiana Dinu
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Data Augmentation, Machine Translation, TAG, Translation

The ability of machine translation (MT) models to correctly place markup is crucial to generating high-quality translations of formatted input. This paper compares two commonly used methods of representing markup tags and tests the ability of MT models to learn tag placement via training data augmentation. We study the interactions of tag representation, data augmentation size, tag complexity, and language pair to show the drawbacks and benefits of each method. We construct and release new test sets containing tagged data for three language pairs of varying difficulty.
The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT
Authors: Jörg Tiedemann
Date: 2020-11-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Few-Shot Learning, Machine Translation, Translation

This paper describes the development of a new benchmark for machine translation that provides training and test data for thousands of language pairs covering over 500 languages and tools for creating state-of-the-art translation models from that collection. The main goal is to trigger the development of open translation tools and models with a much broader coverage of the World’s languages. Using the package it is possible to work on realistic low-resource scenarios avoiding artificially reduced setups that are common when demonstrating zero-shot or few-shot learning. For the first time, this package provides a comprehensive collection of diverse data sets in hundreds of languages with systematic language and script annotation and data splits to extend the narrow coverage of existing benchmarks. Together with the data release, we also provide a growing number of pre-trained baseline models for individual language pairs and selected language groups.
Feature-level Incongruence Reduction for Multimodal Translation
Authors: Zhifeng Li, Yu Hong, Yuchen Pan, Jian Tang, Jianmin Yao, Guodong Zhou
Date: 2021-06-01 00:00:00
Proceeding: NAACL (ALVR) 2021 6
Tasks: Machine Translation, Translation

Caption translation aims to translate image annotations (captions for short). Recently, Multimodal Neural Machine Translation (MNMT) has been explored as the essential solution. Besides of linguistic features in captions, MNMT allows visual(image) features to be used. The integration of multimodal features reinforces the semantic representation and considerably improves translation performance. However, MNMT suffers from the incongruence between visual and linguistic features. To overcome the problem, we propose to extend MNMT architecture with a harmonization network, which harmonizes multimodal features(linguistic and visual features)by unidirectional modal space conversion. It enables multimodal translation to be carried out in a seemingly monomodal translation pipeline. We experiment on the golden Multi30k-16 and 17. Experimental results show that, compared to the baseline,the proposed method yields the improvements of 2.2% BLEU for the scenario of translating English captions into German (En→De) at best,7.6% for the case of English-to-French translation(En→Fr) and 1.5% for English-to-Czech(En→Cz). The utilization of harmonization network leads to the competitive performance to the-state-of-the-art.
Error Causal inference for Multi-Fusion models
Authors: Chengxi Li, Brent Harrison
Date: 2021-06-01 00:00:00
Proceeding: NAACL (ALVR) 2021 6
Tasks: Caption Generation, Causal Inference

In this paper, we propose an error causal inference method that could be used for finding dominant features for a faulty instance under a well-trained multi-modality input model, which could apply to any testing instance. We evaluate our method using a well-trained multi-modalities stylish caption generation model and find those causal inferences that could provide us the insights for next step optimization.
Leveraging Partial Dependency Trees to Control Image Captions
Authors: Wenjie Zhong, Yusuke Miyao
Date: 2021-06-01 00:00:00
Proceeding: NAACL (ALVR) 2021 6
Tasks: Image Captioning

Controlling the generation of image captions attracts lots of attention recently. In this paper, we propose a framework leveraging partial syntactic dependency trees as control signals to make image captions include specified words and their syntactic structures. To achieve this purpose, we propose a Syntactic Dependency Structure Aware Model (SDSAM), which explicitly learns to generate the syntactic structures of image captions to include given partial dependency trees. In addition, we come up with a metric to evaluate how many specified words and their syntactic dependencies are included in generated captions. We carry out experiments on two standard datasets: Microsoft COCO and Flickr30k. Empirical results show that image captions generated by our model are effectively controlled in terms of specified words and their syntactic structures.The code is available on GitHub.
Grounding Plural Phrases: Countering Evaluation Biases by Individuation
Authors: Julia Suter, Letitia Parcalabescu, Anette Frank
Date: 2021-06-01 00:00:00
Proceeding: NAACL (ALVR) 2021 6
Tasks: Phrase Grounding

Phrase grounding (PG) is a multimodal task that grounds language in images. PG systems are evaluated on well-known benchmarks, using Intersection over Union (IoU) as evaluation metric. This work highlights a disconcerting bias in the evaluation of grounded plural phrases, which arises from representing sets of objects as a union box covering all component bounding boxes, in conjunction with the IoU metric. We detect, analyze and quantify an evaluation bias in the grounding of plural phrases and define a novel metric, c-IoU, based on a union box’s component boxes. We experimentally show that our new metric greatly alleviates this bias and recommend using it for fairer evaluation of plural phrases in PG tasks.
Learning to Learn Semantic Factors in Heterogeneous Image Classification
Authors: Boyue Fan, Zhenting Liu
Date: 2021-06-01 00:00:00
Proceeding: NAACL (ALVR) 2021 6
Tasks: Bilevel Optimization, Classification, Few-Shot Learning, image-classification, Image Classification, Meta-Learning

Few-shot learning is to recognize novel classes with a few labeled samples per class. Although numerous meta-learning methods have made significant progress, they struggle to directly address the heterogeneity of training and evaluating task distributions, resulting in the domain shift problem when transitioning to new tasks with disjoint spaces. In this paper, we propose a novel method to deal with the heterogeneity. Specifically, by simulating class-difference domain shift during the meta-train phase, a bilevel optimization procedure is applied to learn a transferable representation space that can rapidly adapt to heterogeneous tasks. Experiments demonstrate the effectiveness of our proposed method.
Reference and coreference in situated dialogue
Authors: Sharid Loáiciga, Simon Dobnik, David Schlangen
Date: 2021-06-01 00:00:00
Proceeding: NAACL (ALVR) 2021 6

In recent years several corpora have been developed for vision and language tasks. We argue that there is still significant room for corpora that increase the complexity of both visual and linguistic domains and which capture different varieties of perceptual and conversational contexts. Working with two corpora approaching this goal, we present a linguistic perspective on some of the challenges in creating and extending resources combining language and vision while preserving continuity with the existing best practices in the area of coreference annotation.
Reconstructing Implicit Knowledge with Language Models
Authors: Maria Becker, Siting Liang, Anette Frank
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Sentence

In this work we propose an approach for generating statements that explicate implicit knowledge connecting sentences in text. We make use of pre-trained language models which we refine by fine-tuning them on specifically prepared corpora that we enriched with implicit information, and by constraining them with relevant concepts and connecting commonsense knowledge paths. Manual and automatic evaluation of the generations shows that by refining language models as proposed, we can generate coherent and grammatically sound sentences that explicate implicit knowledge which connects sentence pairs in texts – on both in-domain and out-of-domain test data.
Investigating the Effect of Background Knowledge on Natural Questions
Authors: Vidhisha Balachandran, Bhuwan Dhingra, Haitian Sun, Michael Collins, William Cohen
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Natural Questions, Retrieval

Existing work shows the benefits of integrating KBs with textual evidence for QA only on questions that are answerable by KBs alone (Sun et al., 2019). In contrast, real world QA systems often have to deal with questions that might not be directly answerable by KBs. Here, we investigate the effect of integrating background knowledge from KBs for the Natural Questions (NQ) task. We create a subset of the NQ data, Factual Questions (FQ), where the questions have evidence in the KB in the form of paths that link question entities to answer entities but still must be answered using text, to facilitate further research into KB integration methods. We propose and analyze a simple, model-agnostic approach for incorporating KB paths into text-based QA systems and establish a strong upper bound on FQ for our method using an oracle retriever. We show that several variants of Personalized PageRank based fact retrievers lead to a low recall of answer entities and consequently fail to improve QA performance. Our results suggest that fact retrieval is a bottleneck for integrating KBs into real world QA datasets