pwc-archive/papers-with-abstracts huggingface dataset:
ICL’s Submission to the WMT21 Critical Error Detection Shared Task
Authors: Genze Jiang, Zhenhao Li, Lucia Specia
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Feature Engineering

This paper presents Imperial College London’s submissions to the WMT21 Quality Estimation (QE) Shared Task 3: Critical Error Detection. Our approach builds on cross-lingual pre-trained representations in a sequence classification model. We further improve the base classifier by (i) adding a weighted sampler to deal with unbalanced data and (ii) introducing feature engineering, where features related to toxicity, named-entities and sentiment, which are potentially indicative of critical errors, are extracted using existing tools and integrated to the model in different ways. We train models with one type of feature at a time and ensemble those models that improve over the base classifier on the development (dev) set. Our official submissions achieve very competitive results, ranking second for three out of four language pairs.
Papago’s Submission for the WMT21 Quality Estimation Shared Task
Authors: Seunghyun Lim, Hantae Kim, Hyunjoong Kim
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Knowledge Distillation, Multi-Task Learning, Sentence

This paper describes Papago submission to the WMT 2021 Quality Estimation Task 1: Sentence-level Direct Assessment. Our multilingual Quality Estimation system explores the combination of Pretrained Language Models and Multi-task Learning architectures. We propose an iterative training pipeline based on pretraining with large amounts of in-domain synthetic data and finetuning with gold (labeled) data. We then compress our system via knowledge distillation in order to reduce parameters yet maintain strong performance. Our submitted multilingual systems perform competitively in multilingual and all 11 individual language pair settings including zero-shot.
NICT Kyoto Submission for the WMT’21 Quality Estimation Task: Multimetric Multilingual Pretraining for Critical Error Detection
Authors: Raphael Rubino, Atsushi Fujita, Benjamin Marie
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Sentence, Translation, XLM-R

This paper presents the NICT Kyoto submission for the WMT’21 Quality Estimation (QE) Critical Error Detection shared task (Task 3). Our approach relies mainly on QE model pretraining for which we used 11 language pairs, three sentence-level and three word-level translation quality metrics. Starting from an XLM-R checkpoint, we perform continued training by modifying the learning objective, switching from masked language modeling to QE oriented signals, before finetuning and ensembling the models. Results obtained on the test set in terms of correlation coefficient and F-score show that automatic metrics and synthetic data perform well for pretraining, with our submissions ranked first for two out of four language pairs. A deeper look at the impact of each metric on the downstream task indicates higher performance for token oriented metrics, while an ablation study emphasizes the usefulness of conducting both self-supervised and QE pretraining.
QEMind: Alibaba’s Submission to the WMT21 Quality Estimation Shared Task
Authors: Jiayi Wang, Ke Wang, Boxing Chen, Yu Zhao, Weihua Luo, Yuqi Zhang
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Sentence, Translation

Quality Estimation, as a crucial step of quality control for machine translation, has been explored for years. The goal is to to investigate automatic methods for estimating the quality of machine translation results without reference translations. In this year’s WMT QE shared task, we utilize the large-scale XLM-Roberta pre-trained model and additionally propose several useful features to evaluate the uncertainty of the translations to build our QE system, named 
          QEMind
        . The system has been applied to the sentence-level scoring task of Direct Assessment and the binary score prediction task of Critical Error Detection. In this paper, we present our submissions to the WMT 2021 QE shared task and an extensive set of experimental results have shown us that our multilingual systems outperform the best system in the Direct Assessment QE task of WMT 2020.
Direct Exploitation of Attention Weights for Translation Quality Estimation
Authors: Lisa Yankovskaya, Mark Fishel
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Sentence, Translation

The paper presents our submission to the WMT2021 Shared Task on Quality Estimation (QE). We participate in sentence-level predictions of human judgments and post-editing effort. We propose a glass-box approach based on attention weights extracted from machine translation systems. In contrast to the previous works, we directly explore attention weight matrices without replacing them with general metrics (like entropy). We show that some of our models can be trained with a small amount of a high-cost labelled data. In the absence of training data our approach still demonstrates a moderate linear correlation, when trained with synthetic data.
IST-Unbabel 2021 Submission for the Quality Estimation Shared Task
Authors: Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, André F. T. Martins
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11

We present the joint contribution of IST and Unbabel to the WMT 2021 Shared Task on Quality Estimation. Our team participated on two tasks: Direct Assessment and Post-Editing Effort, encompassing a total of 35 submissions. For all submissions, our efforts focused on training multilingual models on top of OpenKiwi predictor-estimator architecture, using pre-trained multilingual encoders combined with adapters. We further experiment with and uncertainty-related objectives and features as well as training on out-of-domain direct assessment data.
The IICT-Yverdon System for the WMT 2021 Unsupervised MT and Very Low Resource Supervised MT Task
Authors: Àlex R. Atrio, Gabriel Luthier, Axel Fahy, Giorgos Vernikos, Andrei Popescu-Belis, Ljiljana Dolamic
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Translation

In this paper, we present the systems submitted by our team from the Institute of ICT (HEIG-VD / HES-SO) to the Unsupervised MT and Very Low Resource Supervised MT task. We first study the improvements brought to a baseline system by techniques such as back-translation and initialization from a parent model. We find that both techniques are beneficial and suffice to reach performance that compares with more sophisticated systems from the 2020 task. We then present the application of this system to the 2021 task for low-resource supervised Upper Sorbian (HSB) to German translation, in both directions. Finally, we present a contrastive system for HSB-DE in both directions, and for unsupervised German to Lower Sorbian (DSB) translation, which uses multi-task training with various training schedules to improve over the baseline.
Unsupervised Translation of German–Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language
Authors: Lukas Edman, Ahmet Üstün, Antonio Toral, Gertjan van Noord
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Decoder, Machine Translation, Translation, Unsupervised Machine Translation

This paper describes the methods behind the systems submitted by the University of Groningen for the WMT 2021 Unsupervised Machine Translation task for German–Lower Sorbian (DE–DSB): a high-resource language to a low-resource one. Our system uses a transformer encoder-decoder architecture in which we make three changes to the standard training procedure. First, our training focuses on two languages at a time, contrasting with a wealth of research on multilingual systems. Second, we introduce a novel method for initializing the vocabulary of an unseen language, achieving improvements of 3.2 BLEU for DE->DSB and 4.0 BLEU for DSB->DE.Lastly, we experiment with the order in which offline and online back-translation are used to train an unsupervised system, finding that using online back-translation first works better for DE->DSB by 2.76 BLEU. Our submissions ranked first (tied with another team) for DSB->DE and third for DE->DSB.
The LMU Munich Systems for the WMT21 Unsupervised and Very Low-Resource Translation Task
Authors: Jindřich Libovický, Alexander Fraser
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Translation

We present our submissions to the WMT21 shared task in Unsupervised and Very Low Resource machine translation between German and Upper Sorbian, German and Lower Sorbian, and Russian and Chuvash. Our low-resource systems (German↔Upper Sorbian, Russian↔Chuvash) are pre-trained on high-resource pairs of related languages. We fine-tune those systems using the available authentic parallel data and improve by iterated back-translation. The unsupervised German↔Lower Sorbian system is initialized by the best Upper Sorbian system and improved by iterated back-translation using monolingual data only.
Language Model Pretraining and Transfer Learning for Very Low Resource Languages
Authors: Jyotsana Khatri, Rudra Murthy, Pushpak Bhattacharyya
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Language Modeling, Language Modelling, Transfer Learning, Translation

This paper describes our submission for the shared task on Unsupervised MT and Very Low Resource Supervised MT at WMT 2021. We submitted systems for two language pairs: German ↔ Upper Sorbian (de ↔ hsb) and German-Lower Sorbian (de ↔ dsb). For de ↔ hsb, we pretrain our system using MASS (Masked Sequence to Sequence) objective and then finetune using iterative back-translation. Final finetunng is performed using the parallel data provided for translation objective. For de ↔ dsb, no parallel data is provided in the task, we use final de ↔ hsb model as initialization of the de ↔ dsb model and train it further using iterative back-translation, using the same vocabulary as used in the de ↔ hsb model.
NRC-CNRC Systems for Upper Sorbian-German and Lower Sorbian-German Machine Translation 2021
Authors: Rebecca Knowles, Samuel Larkin
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Transfer Learning, Translation

We describe our neural machine translation systems for the 2021 shared task on Unsupervised and Very Low Resource Supervised MT, translating between Upper Sorbian and German (low-resource) and between Lower Sorbian and German (unsupervised). The systems incorporated data filtering, backtranslation, BPE-dropout, ensembling, and transfer learning from high(er)-resource languages. As measured by automatic metrics, our systems showed strong performance, consistently placing first or tied for first across most metrics and translation directions.
NoahNMT at WMT 2021: Dual Transfer for Very Low Resource Supervised Machine Translation
Authors: Meng Zhang, Minghao Wu, Pengfei Li, Liangyou Li, Qun Liu
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Translation

This paper describes the NoahNMT system submitted to the WMT 2021 shared task of Very Low Resource Supervised Machine Translation. The system is a standard Transformer model equipped with our recent technique of dual transfer. It also employs widely used techniques that are known to be helpful for neural machine translation, including iterative back-translation, selected finetuning, and ensemble. The final submission achieves the top BLEU for three translation directions.
MTEQA at WMT21 Metrics Shared Task
Authors: Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, Pavel Pecina
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Translation

In this paper, we describe our submission to the WMT 2021 Metrics Shared Task. We use the automatically-generated questions and answers to evaluate the quality of Machine Translation (MT) systems. Our submission builds upon the recently proposed MTEQA framework. Experiments on WMT20 evaluation datasets show that at the system-level the MTEQA metric achieves performance comparable with other state-of-the-art solutions, while considering only a certain amount of information from the whole translation.
Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task
Authors: Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, Alon Lavie
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: CPU

In this paper, we present the joint contribution of Unbabel and IST to the WMT 2021 Metrics Shared Task. With this year’s focus on Multidimensional Quality Metric (MQM) as the ground-truth human assessment, our aim was to steer COMET towards higher correlations with MQM. We do so by first pre-training on Direct Assessments and then fine-tuning on z-normalized MQM scores. In our experiments we also show that reference-free COMET models are becoming competitive with reference-based models, even outperforming the best COMET model from 2020 on this year’s development data. Additionally, we present COMETinho, a lightweight COMET model that is 19x faster on CPU than the original model, while also achieving state-of-the-art correlations with MQM. Finally, in the “QE as a metric” track, we also participated with a QE model trained using the OpenKiwi framework leveraging MQM scores and word-level annotations.
Multilingual Machine Translation Evaluation Metrics Fine-tuned on Pseudo-Negative Examples for WMT 2021 Metrics Task
Authors: Kosuke Takahashi, Yoichi Ishibashi, Katsuhito Sudoh, Satoshi Nakamura
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Attribute, Machine Translation

This paper describes our submission to the WMT2021 shared metrics task. Our metric is operative to segment-level and system-level translations. Our belief toward a better metric is to detect a significant error that cannot be missed in the real practice cases of evaluation. For that reason, we used pseudo-negative examples in which attributes of some words are transferred to the reversed attribute words, and we build evaluation models to handle such serious mistakes of translations. We fine-tune a multilingual largely pre-trained model on the provided corpus of past years’ metric task and fine-tune again further on the synthetic negative examples that are derived from the same fine-tune corpus. From the evaluation results of the WMT21’s development corpus, fine-tuning on the pseudo-negatives using WMT15-17 and WMT18-20 metric corpus achieved a better Pearson’s correlation score than the one fine-tuned without negative examples. Our submitted models,hyp+src_hyp+ref and hyp+src_hyp+ref.negative, are the plain model using WMT18-20 and the one additionally fine-tuned on negative samples, respectively.
RoBLEURT Submission for WMT2021 Metrics Task
Authors: Yu Wan, Dayiheng Liu, Baosong Yang, Tianchi Bi, Haibo Zhang, Boxing Chen, Weihua Luo, Derek F. Wong, Lidia S. Chao
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Denoising

In this paper, we present our submission to Shared Metrics Task: RoBLEURT (Robustly Optimizing the training of BLEURT). After investigating the recent advances of trainable metrics, we conclude several aspects of vital importance to obtain a well-performed metric model by: 1) jointly leveraging the advantages of source-included model and reference-only model, 2) continuously pre-training the model with massive synthetic data pairs, and 3) fine-tuning the model with data denoising strategy. Experimental results show that our model reaching state-of-the-art correlations with the WMT2020 human annotations upon 8 out of 10 to-English language pairs.
Linguistic Evaluation for the 2021 State-of-the-art Machine Translation Systems for German to English and English to German
Authors: Vivien Macketanz, Eleftherios Avramidis, Shushen Manakhimova, Sebastian Möller
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Machine Translation, Translation

We are using a semi-automated test suite in order to provide a fine-grained linguistic evaluation for state-of-the-art machine translation systems. The evaluation includes 18 German to English and 18 English to German systems, submitted to the Translation Shared Task of the 2021 Conference on Machine Translation. Our submission adds up to the submissions of the previous years by creating and applying a wide-range test suite for English to German as a new language pair. The fine-grained evaluation allows spotting significant differences between systems that cannot be distinguished by the direct assessment of the human evaluation campaign. We find that most of the systems achieve good accuracies in the majority of linguistic phenomena but there are few phenomena with lower accuracy, such as the idioms, the modal pluperfect and the German resultative predicates. Two systems have significantly better test suite accuracy in macro-average in every language direction, Online-W and Facebook-AI for German to English and VolcTrans and Online-W for English to German. The systems show a steady improvement as compared to previous years.
Pruning Neural Machine Translation for Speed Using Group Lasso
Authors: Maximiliana Behnke, Kenneth Heafield
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Decoder, Machine Translation, Translation

Unlike most work on pruning neural networks, we make inference faster. Group lasso regularisation enables pruning entire rows, columns or blocks of parameters that result in a smaller dense network. Because the network is still dense, efficient matrix multiply routines are still used and only minimal software changes are required to support variable layer sizes. Moreover, pruning is applied during training so there is no separate pruning step. Experiments on top of English->German models, which already have state-of-the-art speed and size, show that two-thirds of feedforward connections can be removed with 0.2 BLEU loss. With 6 decoder layers, the pruned model is 34% faster; with 2 tied decoder layers, the pruned model is 14% faster. Pruning entire heads and feedforward connections in a 12–1 encoder-decoder architecture gains an additional 51% speed-up. These push the Pareto frontier with respect to the trade-off between time and quality compared to strong baselines. In the WMT 2021 Efficiency Task, our pruned and quantised models are 1.9–2.7x faster at the cost 0.9–1.7 BLEU in comparison to the unoptimised baselines. Across language pairs, we see similar sparsity patterns: an ascending or U-shaped distribution in encoder feedforward and attention layers and an ascending distribution in the decoder.
Contrastive Learning for Context-aware Neural Machine Translation Using Coreference Information
Authors: Yongkeun Hwang, Hyeongu Yun, Kyomin Jung
Date: 2021-11-01 00:00:00
Proceeding: WMT (EMNLP) 2021 11
Tasks: Contrastive Learning, coreference-resolution, Coreference Resolution, Data Augmentation, Document Level Machine Translation, Machine Translation, NMT, Sentence, Translation

Context-aware neural machine translation (NMT) incorporates contextual information of surrounding texts, that can improve the translation quality of document-level machine translation. Many existing works on context-aware NMT have focused on developing new model architectures for incorporating additional contexts and have shown some promising results. However, most of existing works rely on cross-entropy loss, resulting in limited use of contextual information. In this paper, we propose CorefCL, a novel data augmentation and contrastive learning scheme based on coreference between the source and contextual sentences. By corrupting automatically detected coreference mentions in the contextual sentence, CorefCL can train the model to be sensitive to coreference inconsistency. We experimented with our method on common context-aware NMT models and two document-level translation tasks. In the experiments, our method consistently improved BLEU of compared models on English-German and English-Korean tasks. We also show that our method significantly improves coreference resolution in the English-German contrastive test suite.
A comprehensive and bias-free evaluation of genomic variant clinical interpretation tools
Authors: Minh Trang, Nguyen; Anh Vu, Mai Nguyen; Hoang Anh, Tran; Nguyet Minh, Do; Thanh Nguyen, Nguyen
Date: 2021-10-28 00:00:00
Proceeding: The 13th International Conference on KNOWLEDGE AND SYSTEMS ENGINEERING 2021 10

The advancement of Next Generation Sequencing (NGS) generates a huge pool of raw sequencing data and genomic variants, while the diverse selection of variant annotation tools adds even more confusion to the mix. Choosing the right tools for clinical interpretation of genomic variants is still challenging due to the lack of comprehensive evaluation studies in this field. Here, we introduced a bias-free analysis approach to assess ten well-known variant annotation tools in terms of clinical interpretation. Our results revealed notable correlations of contemporary methods when applied to the Clin Var dataset. Moreover, allele frequency is still a strong predictor, emphasizing the importance of biological insight in the prediction of clinical interpretation. Our analysis and evaluation scripts are available for public use at https://github.com/nmtrang001Var_Annot_Eval.
Infrared Image Super-Resolution via Transfer Learning and PSRGAN
Authors: Yongsong Huang; Zetao Jiang; Rushi Lan; Shaoqin Zhang; Kui Pi
Date: 2021-05-06 00:00:00
Proceeding: IEEE Signal Processing Letters 2021 5
Tasks: Generative Adversarial Network, Image Super-Resolution, Infrared image super-resolution, Super-Resolution, Transfer Learning

Recent advances in single image super-resolution (SISR) demonstrate the power of deep learning for achieving better performance. Because it is costly to recollect the training data and retrain the model for infrared (IR) image super-resolution, the availability of only a few samples for restoring IR images presents an important challenge in the field of SISR. To solve this problem, we first propose the progressive super-resolution generative adversarial network (PSRGAN) that includes the main path and branch path. The depthwise residual block (DWRB) is used to represent the features of the IR image in the main path. Then, the novel shallow lightweight distillation residual block (SLDRB) is used to extract the features of the readily available visible image in the other path. Furthermore, inspired by transfer learning, we propose the multistage transfer learning strategy for bridging the gap between different high-dimensional feature spaces that can improve the PSRGAN performance. Finally, quantitative and qualitative evaluations of two public datasets show that PSRGAN can achieve better results compared to the SR methods.