pwc-archive/papers-with-abstracts huggingface dataset:
The WMT'18 Morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English
Authors: Franck Burlot, Yves Scherrer, Vinit Ravishankar, Ond{\v{r}}ej Bojar, Stig-Arne Gr{\"o}nroos, Maarit Koponen, Tommi Nieminen, Fran{\c{c}}ois Yvon
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Translation

Progress in the quality of machine translation output calls for new automatic evaluation procedures and metrics. In this paper, we extend the Morpheval protocol introduced by Burlot and Yvon (2017) for the English-to-Czech and English-to-Latvian translation directions to three additional language pairs, and report its use to analyze the results of WMT 2018{'}s participants for these language pairs. Considering additional, typologically varied source and target languages also enables us to draw some generalizations regarding this morphology-oriented evaluation procedure.
Testsuite on Czech--English Grammatical Contrasts
Authors: Silvie Cinkov{\'a}, Ond{\v{r}}ej Bojar
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Translation

We present a pilot study of machine translation of selected grammatical contrasts between Czech and English in WMT18 News Translation Task. For each phenomenon, we run a dedicated test which checks if the candidate translation expresses the phenomenon as expected or not. The proposed type of analysis is not an evaluation in the strict sense because the phenomenon can be correctly translated in various ways and we anticipate only one. What is nevertheless interesting are the differences between various MT systems and the single reference translation in their general tendency in handling the given phenomenon.
A Pronoun Test Suite Evaluation of the English--German MT Systems at WMT 2018
Authors: Liane Guillou, Christian Hardmeier, Ekaterina Lapshinova-Koltunski, Sharid Lo{\'a}iciga
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, NMT, Translation

We evaluate the output of 16 English-to-German MT systems with respect to the translation of pronouns in the context of the WMT 2018 competition. We work with a test suite specifically designed to assess system quality in various fine-grained categories known to be problematic. The main evaluation scores come from a semi-automatic process, combining automatic reference matching with extensive manual annotation of uncertain cases. We find that current NMT systems are good at translating pronouns with intra-sentential reference, but the inter-sentential cases remain difficult. NMT systems are also good at the translation of event pronouns, unlike systems from the phrase-based SMT paradigm. No single system performs best at translating all types of anaphoric pronouns, suggesting unexplained random effects influencing the translation of pronouns with NMT.
The Word Sense Disambiguation Test Suite at WMT18
Authors: Annette Rios, Mathias M{\"u}ller, Rico Sennrich
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Translation, Word Sense Disambiguation

We present a task to measure an MT system{'}s capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German{--}English Word Sense Disambiguation (WSD) test set ContraWSD (Rios Gonzales et al., 2017), but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German{--}English submissions to the WMT{'}18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81{\%}â†’93{\%} accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.
The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional
Authors: Jeremy Gwinnup, S, Joshua vick, Michael Hutt, Grant Erdmann, John Duselis, James Davis
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Multimodal Machine Translation, Translation

AFRL-Ohio State extends its usage of visual domain-driven machine translation for use as a peer with traditional machine translation systems. As a peer, it is enveloped into a system combination of neural and statistical MT systems to present a composite translation.
CUNI System for the WMT18 Multimodal Translation Task
Authors: Jind{\v{r}}ich Helcl, Jind{\v{r}}ich Libovick{\'y}, Du{\v{s}}an Vari{\v{s}}
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Image Classification, Machine Translation, Multimodal Machine Translation, Translation

We present our submission to the WMT18 Multimodal Translation Task. The main feature of our submission is applying a self-attentive network instead of a recurrent neural network. We evaluate two methods of incorporating the visual features in the model: first, we include the image representation as another input to the network; second, we train the model to predict the visual features and use it as an auxiliary objective. For our submission, we acquired both textual and multimodal additional data. Both of the proposed methods yield significant improvements over recurrent networks and self-attentive textual baselines.
Sheffield Submissions for WMT18 Multimodal Translation Shared Task
Authors: Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, Lucia Specia
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Data Augmentation, Machine Translation, Multimodal Machine Translation, NMT, Re-Ranking, Translation, Word Sense Disambiguation

This paper describes the University of Sheffield{'}s submissions to the WMT18 Multimodal Machine Translation shared task. We participated in both tasks 1 and 1b. For task 1, we build on a standard sequence to sequence attention-based neural machine translation system (NMT) and investigate the utility of multimodal re-ranking approaches. More specifically, n-best translation candidates from this system are re-ranked using novel multimodal cross-lingual word sense disambiguation models. For task 1b, we explore three approaches: (i) re-ranking based on cross-lingual word sense disambiguation (as for task 1), (ii) re-ranking based on consensus of NMT n-best lists from German-Czech, French-Czech and English-Czech systems, and (iii) data augmentation by generating English source data through machine translation from French to English and from German to English followed by hypothesis selection using a multimodal-reranker.
Translation of Biomedical Documents with Focus on Spanish-English
Authors: Mirela-Stefania Duma, Wolfgang Menzel
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Domain Adaptation, Machine Translation, Translation

For the WMT 2018 shared task of translating documents pertaining to the Biomedical domain, we developed a scoring formula that uses an unsophisticated and effective method of weighting term frequencies and was integrated in a data selection pipeline. The method was applied on five language pairs and it performed best on Portuguese-English, where a BLEU score of 41.84 placed it third out of seven runs submitted by three institutions. In this paper, we describe our method and results with a special focus on Spanish-English where we compare it against a state-of-the-art method. Our contribution to the task lies in introducing a fast, unsupervised method for selecting domain-specific data for training models which obtain good results using only 10{\%} of the general domain data.
Ensemble of Translators with Automatic Selection of the Best Translation -- the submission of FOKUS to the WMT 18 biomedical translation task --
Authors: Cristian Grozea
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, NMT, Sentence, Translation

This paper describes the system of Fraunhofer FOKUS for the WMT 2018 biomedical translation task. Our approach, described here, was to automatically select the most promising translation from a set of candidates produced with NMT (Transformer) models. We selected the highest fidelity translation of each sentence by using a dictionary, stemming and a set of heuristics. Our method is simple, can use any machine translators, and requires no further training in addition to that already employed to build the NMT models. The downside is that the score did not increase over the best in ensemble, but was quite close to it (difference about 0.5 BLEU).
LMU Munich's Neural Machine Translation Systems at WMT 2018
Authors: Matthias Huck, Dario Stojanovski, Viktor Hangya, Alex Fraser, er
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Domain Adaptation, Machine Translation, Translation, Unsupervised Machine Translation

We present the LMU Munich machine translation systems for the English{--}German language pair. We have built neural machine translation systems for both translation directions (English→German and German→English) and for two different domains (the biomedical domain and the news domain). The systems were used for our participation in the WMT18 biomedical translation task and in the shared task on machine translation of news. The main focus of our recent system development efforts has been on achieving improvements in the biomedical domain over last year{'}s strong biomedical translation engine for English→German (Huck et al., 2017a). Considerable progress has been made in the latter task, which we report on in this paper.
Hunter NMT System for WMT18 Biomedical Translation Task: Transfer Learning in Neural Machine Translation
Authors: Abdul Khan, P, Subhadarshi a, Jia Xu, Lampros Flokas
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Domain Adaptation, Ensemble Learning, Machine Translation, NMT, Transfer Learning, Translation

This paper describes the submission of Hunter Neural Machine Translation (NMT) to the WMT{'}18 Biomedical translation task from English to French. The discrepancy between training and test data distribution brings a challenge to translate text in new domains. Beyond the previous work of combining in-domain with out-of-domain models, we found accuracy and efficiency gain in combining different in-domain models. We conduct extensive experiments on NMT with \textit{transfer learning}. We train on different in-domain Biomedical datasets one after another. That means parameters of the previous training serve as the initialization of the next one. Together with a pre-trained out-of-domain News model, we enhanced translation quality with 3.73 BLEU points over the baseline. Furthermore, we applied ensemble learning on training models of intermediate epochs and achieved an improvement of 4.02 BLEU points over the baseline. Overall, our system is 11.29 BLEU points above the best system of last year on the EDP 2017 test set.
Neural Machine Translation with the Transformer and Multi-Source Romance Languages for the Biomedical WMT 2018 task
Authors: Brian Tubay, Marta R. Costa-juss{\`a}
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Speech Recognition, Translation

The Transformer architecture has become the state-of-the-art in Machine Translation. This model, which relies on attention-based mechanisms, has outperformed previous neural machine translation architectures in several tasks. In this system description paper, we report details of training neural machine translation with multi-source Romance languages with the Transformer model and in the evaluation frame of the biomedical WMT 2018 task. Using multi-source languages from the same family allows improvements of over 6 BLEU points.
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance
Authors: Qingsong Ma, Ond{\v{r}}ej Bojar, Yvette Graham
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Sentence, Translation

This paper presents the results of the WMT18 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT18 News Translation Task with automatic metrics. We collected scores of 10 metrics and 8 research groups. In addition to that, we computed scores of 8 standard metrics (BLEU, SentBLEU, chrF, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric{'}s scores correlate with WMT18 official manual ranking of systems) and in terms of segment-level correlation (how often a metric agrees with humans in judging the quality of a particular sentence relative to alternate outputs). This year, we employ a single kind of manual evaluation: direct assessment (DA).
Findings of the WMT 2018 Shared Task on Quality Estimation
Authors: Lucia Specia, Fr{\'e}d{\'e}ric Blain, Varvara Logacheva, Ram{\'o}n Astudillo, Andr{\'e} F. T. Martins
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Sentence, Translation

We report the results of the WMT18 shared task on Quality Estimation, i.e. the task of predicting the quality of the output of machine translation systems at various granularity levels: word, phrase, sentence and document. This year we include four language pairs, three text domains, and translations produced by both statistical and neural machine translation systems. Participating teams from ten institutions submitted a variety of systems to different task variants and language pairs.
Findings of the WMT 2018 Shared Task on Automatic Post-Editing
Authors: Rajen Chatterjee, Matteo Negri, Raphael Rubino, Marco Turchi
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Automatic Post-Editing, Machine Translation, NMT, Translation

We present the results from the fourth round of the WMT shared task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a {``}black-box{''} machine translation system by learning from human corrections. Keeping the same general evaluation setting of the three previous rounds, this year we focused on one language pair (English-German) and on domain-specific data (Information Technology), with MT outputs produced by two different paradigms: phrase-based (PBSMT) and neural (NMT). Five teams submitted respectively 11 runs for the PBSMT subtask and 10 runs for the NMT subtask. In the former subtask, characterized by original translations of lower quality, top results achieved impressive improvements, up to -6.24 TER and +9.53 BLEU points over the baseline {``}\textit{do-nothing}{''} system. The NMT subtask proved to be more challenging due to the higher quality of the original translations and the availability of less training data. In this case, top results show smaller improvements up to -0.38 TER and +0.8 BLEU points.
Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering
Authors: Philipp Koehn, Huda Khayrallah, Kenneth Heafield, Mikel L. Forcada
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Outlier Detection, Sentence, Translation

We posed the shared task of assigning sentence-level quality scores for a very noisy corpus of sentence pairs crawled from the web, with the goal of sub-selecting 1{\%} and 10{\%} of high-quality data to be used to train machine translation systems. Seventeen participants from companies, national research labs, and universities participated in this task.
Meteor++: Incorporating Copy Knowledge into Machine Translation Evaluation
Authors: Yinuo Guo, Chong Ruan, Junfeng Hu
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Sentence, Text Generation, Translation

In machine translation evaluation, a good candidate translation can be regarded as a paraphrase of the reference. We notice that some words are always copied during paraphrasing, which we call \textbf{copy knowledge}. Considering the stability of such knowledge, a good candidate translation should contain all these words appeared in the reference sentence. Therefore, in this participation of the WMT{'}2018 metrics shared task we introduce a simple statistical method for copy knowledge extraction, and incorporate it into Meteor metric, resulting in a new machine translation metric \textbf{Meteor++}. Our experiments show that Meteor++ can nicely integrate copy knowledge and improve the performance significantly on WMT17 and WMT15 evaluation sets.
ITER: Improving Translation Edit Rate through Optimizable Edit Costs
Authors: Joybrata Panja, Sudip Kumar Naskar
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Translation

The paper presents our participation in the WMT 2018 Metrics Shared Task. We propose an improved version of Translation Edit/Error Rate (TER). In addition to including the basic edit operations in TER, namely - insertion, deletion, substitution and shift, our metric also allows stem matching, optimizable edit costs and better normalization so as to correlate better with human judgement scores. The proposed metric shows much higher correlation with human judgments than TER.
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation
Authors: Hiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, Sentence, Sentence Embeddings, Translation

We introduce the RUSE metric for the WMT18 metrics shared task. Sentence embeddings can capture global information that cannot be captured by local features based on character or word N-grams. Although training sentence embeddings using small-scale translation datasets with manual evaluation is difficult, sentence embeddings trained from large-scale data in other tasks can improve the automatic evaluation of machine translation. We use a multi-layer perceptron regressor based on three types of sentence embeddings. The experimental results of the WMT16 and WMT17 datasets show that the RUSE metric achieves a state-of-the-art performance in both segment- and system-level metrics tasks with embedding features only.
Keep It or Not: Word Level Quality Estimation for Post-Editing
Authors: Prasenjit Basu, Santanu Pal, Sudip Kumar Naskar
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Language Modeling, Language Modelling, Machine Translation, Translation

The paper presents our participation in the WMT 2018 shared task on word level quality estimation (QE) of machine translated (MT) text, i.e., to predict whether a word in MT output for a given source context is correctly translated and hence should be retained in the post-edited translation (PE), or not. To perform the QE task, we measure the similarity of the source context of the target MT word with the context for which the word is retained in PE in the training data. This is achieved in two different ways, using \textit{Bag-of-Words} (\textit{BoW}) model and \textit{Document-to-Vector} (\textit{Doc2Vec}) model. In the \textit{BoW} model, we compute the cosine similarity while in the \textit{Doc2Vec} model we consider the Doc2Vec similarity. By applying the Kneedle algorithm on the F1mult vs. similarity score plot, we derive the threshold based on which OK/BAD decisions are taken for the MT words. Experimental results revealed that the Doc2Vec model performs better than the BoW model on the word level QE task.
RTM results for Predicting Translation Performance
Authors: Ergun Bi{\c{c}}ici
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Language Modeling, Language Modelling, Machine Translation, NMT, Prediction, Sentence, Translation

With improved prediction combination using weights based on their training performance and stacking and multilayer perceptrons to build deeper prediction models, RTMs become the 3rd system in general at the sentence-level prediction of translation scores and achieve the lowest RMSE in English to German NMT QET results. For the document-level task, we compare document-level RTM models with sentence-level RTM models obtained with the concatenation of document sentences and obtain similar results.
Neural Machine Translation for English-Tamil
Authors: Himanshu Choudhary, Aditya Kumar Pathak, Rajiv Ratan Saha, Ponnurangam Kumaraguru
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, NMT, Translation

A huge amount of valuable resources is available on the web in English, which are often translated into local languages to facilitate knowledge sharing among local people who are not much familiar with English. However, translating such content manually is very tedious, costly, and time-consuming process. To this end, machine translation is an efficient approach to translate text without any human involvement. Neural machine translation (NMT) is one of the most recent and effective translation technique amongst all existing machine translation systems. In this paper, we apply NMT for English-Tamil language pair. We propose a novel neural machine translation technique using word-embedding along with Byte-Pair-Encoding (BPE) to develop an efficient translation system that overcomes the OOV (Out Of Vocabulary) problem for languages which do not have much translations available online. We use the BLEU score for evaluating the system performance. Experimental results confirm that our proposed MIDAS translator (8.33 BLEU score) outperforms Google translator (3.75 BLEU score).
The Benefit of Pseudo-Reference Translations in Quality Estimation of MT Output
Authors: Melania Duma, Wolfgang Menzel
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Machine Translation, NMT, Sentence

In this paper, a novel approach to Quality Estimation is introduced, which extends the method in (Duma and Menzel, 2017) by also considering pseudo-reference translations as data sources to the tree and sequence kernels used before. Two variants of the system were submitted to the sentence level WMT18 Quality Estimation Task for the English-German language pair. They have been ranked 4th and 6th out of 13 systems in the SMT track, while in the NMT track ranks 4 and 5 out of 11 submissions have been reached.
Supervised and Unsupervised Minimalist Quality Estimators: Vicomtech's Participation in the WMT 2018 Quality Estimation Task
Authors: Thierry Etchegoyhen, Eva Mart{\'\i}nez Garcia, Andoni Azpeitia
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: Language Modeling, Language Modelling, Machine Translation, Sentence, Translation

We describe Vicomtech{'}s participation in the WMT 2018 shared task on quality estimation, for which we submitted minimalist quality estimators. The core of our approach is based on two simple features: lexical translation overlaps and language model cross-entropy scores. These features are exploited in two system variants: uMQE is an unsupervised system, where the final quality score is obtained by averaging individual feature scores; sMQE is a supervised variant, where the final score is estimated by a Support Vector Regressor trained on the available annotated datasets. The main goal of our minimalist approach to quality estimation is to provide reliable estimators that require minimal deployment effort, few resources, and, in the case of uMQE, do not depend on costly data annotation or post-editing. Our approach was applied to all language pairs in sentence quality estimation, obtaining competitive results across the board.
Sheffield Submissions for the WMT18 Quality Estimation Shared Task
Authors: Julia Ive, Carolina Scarton, Fr{\'e}d{\'e}ric Blain, Lucia Specia
Date: 2018-10-01 00:00:00
Proceeding: WS 2018 10
Tasks: All, Machine Translation

In this paper we present the University of Sheffield submissions for the WMT18 Quality Estimation shared task. We discuss our submissions to all four sub-tasks, where ours is the only team to participate in all language pairs and variations (37 combinations). Our systems show competitive results and outperform the baseline in nearly all cases.