pwc-archive/papers-with-abstracts huggingface dataset:
The LMU Munich Unsupervised Machine Translation System for WMT19
Authors: Dario Stojanovski, Viktor Hangya, Matthias Huck, Alex Fraser, er
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Denoising, Language Modeling, Language Modelling, Machine Translation, Translation, Unsupervised Machine Translation, Word Embeddings

We describe LMU Munich{'}s machine translation system for Germanâ†’Czech translation which was used to participate in the WMT19 shared task on unsupervised news translation. We train our model using monolingual data only from both languages. The final model is an unsupervised neural model using established techniques for unsupervised translation such as denoising autoencoding and online back-translation. We bootstrap the model with masked language model pretraining and enhance it with back-translations from an unsupervised phrase-based system which is itself bootstrapped using unsupervised bilingual word embeddings.
Combining Local and Document-Level Context: The LMU Munich Neural Machine Translation System at WMT19
Authors: Dario Stojanovski, Alex Fraser, er
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Sentence, Translation

We describe LMU Munich{'}s machine translation system for Englishâ†’German translation which was used to participate in the WMT19 shared task on supervised news translation. We specifically participated in the document-level MT track. The system used as a primary submission is a context-aware Transformer capable of both rich modeling of limited contextual information and integration of large-scale document-level context with a less rich representation. We train this model by fine-tuning a big Transformer baseline. Our experimental results show that document-level context provides for large improvements in translation quality, and adding a rich representation of the previous sentence provides a small additional gain.
IITP-MT System for Gujarati-English News Translation Task at WMT 2019
Authors: Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, Pushpak Bhattacharyya
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, NMT, Translation

We describe our submission to WMT 2019 News translation shared task for Gujarati-English language pair. We submit constrained systems, i.e, we rely on the data provided for this language pair and do not use any external data. We train Transformer based subword-level neural machine translation (NMT) system using original parallel corpus along with synthetic parallel corpus obtained through back-translation of monolingual data. Our primary systems achieve BLEU scores of 10.4 and 8.1 for Gujarati→English and English→Gujarati, respectively. We observe that incorporating monolingual data through back-translation improves the BLEU score significantly over baseline NMT and SMT systems for this language pair.
The En-Ru Two-way Integrated Machine Translation System Based on Transformer
Authors: Doron Yu
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation

Machine translation is one of the most popular areas in natural language processing. WMT is a conference to assess the level of machine translation capabilities of organizations around the world, which is the evaluation activity we participated in. In this review we participated in a two-way translation track from Russian to English and English to Russian. We used official training data, 38 million parallel corpora, and 10 million monolingual corpora. The overall framework we use is the Transformer neural machine translation model, supplemented by data filtering, post-processing, reordering and other related processing methods. The BLEU value of our final translation result from Russian to English is 38.7, ranking 5th, while from English to Russian is 27.8, ranking 10th.
DFKI-NMT Submission to the WMT19 News Translation Task
Authors: Jingyi Zhang, Josef van Genabith
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: NMT, Translation

This paper describes the DFKI-NMT submission to the WMT19 News translation task. We participated in both English-to-German and German-to-English directions. We trained Transformer models and adopted various techniques for effectively training our models, including data selection, back-translation and in-domain fine-tuning. We give a detailed analysis of the performance of our system.
A Test Suite and Manual Evaluation of Document-Level NMT at WMT19
Authors: Kate{\v{r}}ina Rysov{\'a}, Magdal{\'e}na Rysov{\'a}, Tom{\'a}{\v{s}} Musil, Lucie Pol{\'a}kov{\'a}, Ond{\v{r}}ej Bojar
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, NMT, Sentence, Translation

As the quality of machine translation rises and neural machine translation (NMT) is moving from sentence to document level translations, it is becoming increasingly difficult to evaluate the output of translation systems. We provide a test suite for WMT19 aimed at assessing discourse phenomena of MT systems participating in the News Translation Task. We have manually checked the outputs and identified types of translation errors that are relevant to document-level translation.
Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses
Authors: Maja Popovi{\'c}
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Sentence, Translation

We present a test set for evaluating an MT system{'}s capability to translate ambiguous conjunctions depending on the sentence structure. We concentrate on the English conjunction {``}but{''} and its French equivalent {``}mais{''} which can be translated into two different German conjunctions. We evaluate all English-to-German and French-to-German submissions to the WMT 2019 shared translation task. The evaluation is done mainly automatically, with additional fast manual inspection of unclear cases. All systems almost perfectly recognise the target conjunction {``}aber{''}, whereas accuracies for the other target conjunction {``}sondern{''} range from 78{\%} to 97{\%}, and the errors are mostly caused by replacing it with the alternative conjunction {``}aber{''}. The best performing system for both language pairs is a multilingual Transformer {``}TartuNLP{''} system trained on all WMT 2019 language pairs which use the Latin script, indicating that the multilingual approach is beneficial for conjunction disambiguation. As for other system features, such as using synthetic back-translated data, context-aware, hybrid, etc., no particular (dis)advantages can be observed. Qualitative manual inspection of translation hypotheses shown that highly ranked systems generally produce translations with high adequacy and fluency, meaning that these systems are not only capable of capturing the right conjunction whereas the rest of the translation hypothesis is poor. On the other hand, the low ranked systems generally exhibit lower fluency and poor adequacy.
The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation
Authors: Aless Raganato, ro, Yves Scherrer, J{\"o}rg Tiedemann
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, NMT, Sentence, Translation, Word Sense Disambiguation

Supervised Neural Machine Translation (NMT) systems currently achieve impressive translation quality for many language pairs. One of the key features of a correct translation is the ability to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. Existing evaluation benchmarks on WSD capabilities of translation systems rely heavily on manual work and cover only few language pairs and few word types. We present MuCoW, a multilingual contrastive test suite that covers 16 language pairs with more than 200 thousand contrastive sentence pairs, automatically built from word-aligned parallel corpora and the wide-coverage multilingual sense inventory of BabelNet. We evaluate the quality of the ambiguity lexicons and of the resulting test suite on all submissions from 9 language pairs presented in the WMT19 news shared translation task, plus on other 5 language pairs using NMT pretrained models. The MuCoW test suite is available at http://github.com/Helsinki-NLP/MuCoW.
SAO WMT19 Test Suite: Machine Translation of Audit Reports
Authors: Tereza Vojt{\v{e}}chov{\'a}, Michal Nov{\'a}k, Milo{\v{s}} Klou{\v{c}}ek, Ond{\v{r}}ej Bojar
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation

This paper describes a machine translation test set of documents from the auditing domain and its use as one of the {``}test suites{''} in the WMT19 News Translation Task for translation directions involving Czech, English and German. Our evaluation suggests that current MT systems optimized for the general news domain can perform quite well even in the particular domain of audit reports. The detailed manual evaluation however indicates that deep factual knowledge of the domain is necessary. For the naked eye of a non-expert, translations by many systems seem almost perfect and automatic MT evaluation with one reference is practically useless for considering these details. Furthermore, we show on a sample document from the domain of agreements that even the best systems completely fail in preserving the semantics of the agreement, namely the identity of the parties.
WMDO: Fluency-based Word Mover's Distance for Machine Translation Evaluation
Authors: Julian Chow, Lucia Specia, Pranava Madhyastha
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation, Word Embeddings

We propose WMDO, a metric based on distance between distributions in the semantic vector space. Matching in the semantic space has been investigated for translation evaluation, but the constraints of a translation{'}s word order have not been fully explored. Building on the Word Mover{'}s Distance metric and various word embeddings, we introduce a fragmentation penalty to account for fluency of a translation. This word order extension is shown to perform better than standard WMD, with promising results against other types of metrics.
Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation
Authors: Yinuo Guo, Junfeng Hu
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation

This paper describes Meteor++ 2.0, our submission to the WMT19 Metric Shared Task. The well known Meteor metric improves machine translation evaluation by introducing paraphrase knowledge. However, it only focuses on the lexical level and utilizes consecutive n-grams paraphrases. In this work, we take into consideration syntactic level paraphrase knowledge, which sometimes may be skip-grams. We describe how such knowledge can be extracted from Paraphrase Database (PPDB) and integrated into Meteor-based metrics. Experiments on WMT15 and WMT17 evaluation datasets show that the newly proposed metric outperforms all previous versions of Meteor.
YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources
Authors: Chi-kiu Lo
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Semantic Similarity, Semantic Textual Similarity, Translation

We present YiSi, a unified automatic semantic machine translation quality evaluation and estimation metric for languages with different levels of available resources. Underneath the interface with different language resources settings, YiSi uses the same representation for the two sentences in assessment. Besides, we show significant improvement in the correlation of YiSi-1{'}s scores with human judgment is made by using contextual embeddings in multilingual BERT{--}Bidirectional Encoder Representations from Transformers to evaluate lexical semantic similarity. YiSi is open source and publicly available.
EED: Extended Edit Distance Measure for Machine Translation
Authors: Peter Stanchev, Weiyue Wang, Hermann Ney
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Translation

Over the years a number of machine translation metrics have been developed in order to evaluate the accuracy and quality of machine-generated translations. Metrics such as BLEU and TER have been used for decades. However, with the rapid progress of machine translation systems, the need for better metrics is growing. This paper proposes an extension of the edit distance, which achieves better human correlation, whilst remaining fast, flexible and easy to understand.
Filtering Pseudo-References by Paraphrasing for Automatic Evaluation of Machine Translation
Authors: Ryoma Yoshimura, Hiroki Shimanaka, Yukio Matsumura, Hayahide Yamagishi, Mamoru Komachi
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Sentence, Translation

In this paper, we introduce our participation in the WMT 2019 Metric Shared Task. We propose an improved version of sentence BLEU using filtered pseudo-references. We propose a method to filter pseudo-references by paraphrasing for automatic evaluation of machine translation (MT). We use the outputs of off-the-shelf MT systems as pseudo-references filtered by paraphrasing in addition to a single human reference (gold reference). We use BERT fine-tuned with paraphrase corpus to filter pseudo-references by checking the paraphrasability with the gold reference. Our experimental results of the WMT 2016 and 2017 datasets show that our method achieved higher correlation with human evaluation than the sentence BLEU (SentBLEU) baselines with a single reference and with unfiltered pseudo-references.
NICT's Supervised Neural Machine Translation Systems for the WMT19 Translation Robustness Task
Authors: Raj Dabre, Eiichiro Sumita
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Domain Adaptation, Machine Translation, NMT, Transfer Learning, Translation

In this paper we describe our neural machine translation (NMT) systems for Japanese↔English translation which we submitted to the translation robustness task. We focused on leveraging transfer learning via fine tuning to improve translation quality. We used a fairly well established domain adaptation technique called Mixed Fine Tuning (MFT) (Chu et. al., 2017) to improve translation quality for Japanese↔English. We also trained bi-directional NMT models instead of uni-directional ones as the former are known to be quite robust, especially in low-resource scenarios. However, given the noisy nature of the in-domain training data, the improvements we obtained are rather modest.
System Description: The Submission of FOKUS to the WMT 19 Robustness Task
Authors: Cristian Grozea
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: fr-en, Machine Translation, Translation

This paper describes the systems of Fraunhofer FOKUS for the WMT 2019 machine translation robustness task. We have made submissions to the EN-FR, FR-EN, and JA-EN language pairs. The first two were made with a baseline translator, trained on clean data for the WMT 2019 biomedical translation task. These baselines improved over the baselines from the MTNT paper by 2 to 4 BLEU points, but where not trained on the same data. The last one used the same model class and training procedure, with induced typos in the training data to increase the model robustness.
JHU 2019 Robustness Task System Description
Authors: Matt Post, Kevin Duh
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: fr-en

We describe the JHU submissions to the French{--}English, Japanese{--}English, and English{--}Japanese Robustness Task at WMT 2019. Our goal was to evaluate the performance of baseline systems on both the official noisy test set as well as news data, in order to ensure that performance gains in the latter did not come at the expense of general-domain performance. To this end, we built straightforward 6-layer Transformer models and experimented with a handful of variables including subword processing (FR→EN) and a handful of hyperparameters settings (JA↔EN). As expected, our systems performed reasonably.
Improving Robustness of Neural Machine Translation with Multi-task Learning
Authors: Shuyan Zhou, Xiangkai Zeng, Yingqi Zhou, Antonios Anastasopoulos, Graham Neubig
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Multi-Task Learning, NMT, Translation

While neural machine translation (NMT) achieves remarkable performance on clean, in-domain text, performance is known to degrade drastically when facing text which is full of typos, grammatical errors and other varieties of noise. In this work, we propose a multi-task learning algorithm for transformer-based MT systems that is more resilient to this noise. We describe our submission to the WMT 2019 Robustness shared task based on this method. Our model achieves a BLEU score of 32.8 on the shared task French to English dataset, which is 7.1 BLEU points higher than the baseline vanilla transformer trained with clean text.
Findings of the WMT 2019 Shared Tasks on Quality Estimation
Authors: Erick Fonseca, Lisa Yankovskaya, Andr{\'e} F. T. Martins, Mark Fishel, Christian Federmann
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Sentence, Translation

We report the results of the WMT19 shared task on Quality Estimation, i.e. the task of predicting the quality of the output of machine translation systems given just the source text and the hypothesis translations. The task includes estimation at three granularity levels: word, sentence and document. A novel addition is evaluating sentence-level QE against human judgments: in other words, designing MT metrics that do not need a reference translation. This year we include three language pairs, produced solely by neural machine translation systems. Participating teams from eleven institutions submitted a variety of systems to different task variants and language pairs.
Findings of the WMT 2019 Shared Task on Automatic Post-Editing
Authors: Rajen Chatterjee, Christian Federmann, Matteo Negri, Marco Turchi
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Automatic Post-Editing, Machine Translation, Translation

We present the results from the 5th round of the WMT task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a {``}black-box{''} machine translation system by learning from human corrections. Keeping the same general evaluation setting of the previous four rounds, this year we focused on two language pairs (English-German and English-Russian) and on domain-specific data (In-formation Technology). For both the language directions, MT outputs were produced by neural systems unknown to par-ticipants. Seven teams participated in the English-German task, with a total of 18 submitted runs. The evaluation, which was performed on the same test set used for the 2018 round, shows a slight progress in APE technology: 4 teams achieved better results than last year{'}s winning system, with improvements up to -0.78 TER and +1.23 BLEU points over the baseline. Two teams participated in theEnglish-Russian task submitting 2 runs each. On this new language direction, characterized by a higher quality of the original translations, the task proved to be particularly challenging. None of the submitted runs improved the very high results of the strong system used to produce the initial translations(16.16 TER, 76.20 BLEU).
Findings of the WMT 2019 Biomedical Translation Shared Task: Evaluation for MEDLINE Abstracts and Biomedical Terminologies
Authors: Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Mariana Neves, Felipe Soares, Amy Siu, Karin Verspoor, Maika Vicente Navarro
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: de-en, es-en, fr-en, Translation

In the fourth edition of the WMT Biomedical Translation task, we considered a total of six languages, namely Chinese (zh), English (en), French (fr), German (de), Portuguese (pt), and Spanish (es). We performed an evaluation of automatic translations for a total of 10 language directions, namely, zh/en, en/zh, fr/en, en/fr, de/en, en/de, pt/en, en/pt, es/en, and en/es. We provided training data based on MEDLINE abstracts for eight of the 10 language pairs and test sets for all of them. In addition to that, we offered a new sub-task for the translation of terms in biomedical terminologies for the en/es language direction. Higher BLEU scores (close to 0.5) were obtained for the es/en, en/es and en/pt test sets, as well as for the terminology sub-task. After manual validation of the primary runs, some submissions were judged to be better than the reference translations, for instance, for de/en, en/es and es/en.
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions
Authors: Philipp Koehn, Francisco Guzm{\'a}n, Vishrav Chaudhary, Juan Pino
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Sentence, Translation

Following the WMT 2018 Shared Task on Parallel Corpus Filtering, we posed the challenge of assigning sentence-level quality scores for very noisy corpora of sentence pairs crawled from the web, with the goal of sub-selecting 2{\%} and 10{\%} of the highest-quality data to be used to train machine translation systems. This year, the task tackled the low resource condition of Nepali-English and Sinhala-English. Eleven participants from companies, national research labs, and universities participated in this task.
RTM Stacking Results for Machine Translation Performance Prediction
Authors: Ergun Bi{\c{c}}ici
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Machine Translation, Mixture-of-Experts, Prediction, Sentence, Translation

We obtain new results using referential translation machines with increased number of learning models in the set of results that are stacked to obtain a better mixture of experts prediction. We combine features extracted from the word-level predictions with the sentence- or document-level features, which significantly improve the results on the training sets but decrease the test set results.
QE BERT: Bilingual BERT Using Multi-task Learning for Neural Quality Estimation
Authors: Hyun Kim, Joon-Ho Lim, Hyun-Ki Kim, Seung-Hoon Na
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Multi-Task Learning, Sentence, Translation

For translation quality estimation at word and sentence levels, this paper presents a novel approach based on BERT that recently has achieved impressive results on various natural language processing tasks. Our proposed model is re-purposed BERT for the translation quality estimation and uses multi-task learning for the sentence-level task and word-level subtasks (i.e., source word, target word, and target gap). Experimental results on Quality Estimation shared task of WMT19 show that our systems show competitive results and provide significant improvements over the baseline.
MIPT System for World-Level Quality Estimation
Authors: Mikhail Mosyagin, Varvara Logacheva
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Translation

We explore different model architectures for the WMT 19 shared task on word-level quality estimation of automatic translation. We start with a model similar to Shef-bRNN, which we modify by using conditional random fields for sequence labelling. Additionally, we use a different approach for labelling gaps and source words. We further develop this model by including features from different sources such as BERT, baseline features for the task and transformer encoders. We evaluate the performance of our models on the English-German dataset for the corresponding shared task.