MT - T elescope: A n interactive platform for contrastive evaluation of MT systems (2021)

Abstract: We present MT-Telescope, a visualization platform designed to facilitate comparative analysis of the output quality of two Machine Translation (MT) systems. While automated MT evaluation metrics are commonly used to evaluate MT systems at a corpus-level, our platform supports fine-grained segment-level analysis and interactive visualisations that expose the fundamental differences in the performance of the compared systems. MT-Telescope also supports dynamic corpus filtering to enable focused analysis on specific phenomena such as; translation of named entities, handling of terminology, and the impact of input segment length on translation quality. Furthermore, the platform provides a bootstrapped t-test for statistical significance as a means of evaluating the rigor of the resulting system ranking. MT-Telescope is open source, written in Python, and is built around a user friendly and dynamic web interface. Complementing other existing tools, our platform is designed to facilitate and promote the broader adoption of more rigorous analysis practices in the evaluation of MT quality.

## introduction when developing mt systems or comparing experiments across papers, it has been common practice for researchers and developers to rely on automated metrics such as bleu @xcite and meteor @xcite as a means of quantifying the relative performance difference between two models. commercial deployment of systems and the establishment of state-of-theart in academia is often driven by these metrics alone. automated metrics have long been an essential means for assessing quality improvements and driving progress in the field of mt. recent state-of-the-art metrics such as comet @xcite , prism @xcite , and bleurt @xcite , show much higher levels of correlation with human judgement than their predecessors. notwithstanding the strength of available metrics, when applied and reported at corpus-level, they are only able to provide a general indication of whether one system is superior, based on a single score which in some cases is limited to an arithmetic mean of segment-level score predictions @xcite . we contend that the broad definition of 'improvement' as an increase in a relevant corpus-level score is insufficient, especially when the relative difference between high-performing mt systems is negligible. exposure of the changing distribution of performance at segment-level on targeted phenomena is fundamental to our understanding of translation quality. manual inspection at this level is often too time-consuming and inefficient to be done rigorously and on a regular basis. mt-telescope was inspired by other recent work on developing holistic approaches for finegrained comparison of mt systems, such as compare-mt @xcite and mt-compareval @xcite and other more general comparative tools such as vizseq @xcite . despite the intention of such tools in addressing the above problem, none have been widely adopted as a standard method of evaluating mt. mt-telescope was specifically developed to leverage the best of existing approaches in a manner that is as user friendly as possible, with features specifically tailored to the mt use case. the platform supports fine-grained segment-level analysis and interactive visualisations that provide relevant and informative quality intelligence. in particular, the platform also supports focused analysis of mt-specific phenomena through interactive corpus filtering. mt-telescope is differentiated from existing mt-specific tools by exposing features such as named entities and glossary handling which play a fundamental role in determining the suitability of an mt system for a production environment. furthermore, the platform applies a bootstrapped t-test for statistical significance @xcite as a means of exposing the experimental rigor of system comparisons. these features are not widely available in other tools and provide a uniquely tailored solution to mt comparison that is highly informative and easy to use. the fundamental goal of mt-telescope is to widen access to state-of-art, robust mt comparison, to the benefit of the mt community at large. mt-telescope is open source, written in python and uses a dynamic web interface implemented in streamlit 2 . in this manner, mt-telescope provides a uniquely accessible framework that requires little technical skill to operate and exposes information about the critical differences between mt outputs that is interactive, informative and highly customizable. ## mt-telescope: features in this section, we describe the main features and visualizations implemented in mt-telescope and illustrate the user experience with examples: 2 https://streamlit.io/ 2.1 user input and data mt-telescope is opened in a web browser and takes four text (.txt) files as input; source and reference segments and one set of mt outputs for each of the compared systems. users drag and drop these files directly onto the interface to begin evaluation. comet @xcite ) is provided as a default metric given its proven value in the wmt metrics shared task 2020 @xcite . optionally the user can choose an alternate metric using a selection box. currently available metrics include bleu, meteor and chrf, and a selection of more recently proposed metrics such as prism, bleurt, and bertscore. ## mt-telescope: dynamic corpus filtering given a test corpus, mt-telescope provides functionality to dynamically evaluate sub-samples of the system outputs as a means of focused analysis tailored to particular phenomena relevant to mt. on selection of any of the available filtering criteria, the mt-telescope dynamic corpus filtering feature (dcf) updates the output evaluation in real-time to allow the user to 'zoom in' on relevant data points. currently, mt-telescope supports filtering by named entity, glossary and source segment length, as well as an option to remove duplicates. whenever any of these options is selected, the interface will output the size of the sub-sample as a percentage of the original test corpus. ## statistical significance testing by default, mt-telescope implements the bootstrapped t-test for statistical significance promoted for use in comparison of mt systems by @xcite . specifically, we iteratively re-sample a portion of the test set (of size p ) n times, compare corpus-level results of each sub-sample and record the comparative conclusions. the ratio of wins of a single system is a reasonable proxy to the probability that that system is better than the other. in other words, if one system outperforms the other system 95% of the time, we conclude that the former is better with a significance of @xmath0.05 @xcite . this is particularly useful in cases where the relative difference between systems is minimal and acts as a measure of the robustness of any resulting decision. in our implementation p is an optional parameter which defaults to 0.5 (50%) or 500 seg-ments, whichever is larger, to ensure reasonable stability in the output conclusion. n is also user defined and by default is set at 300 iterations. ## related tools mt-telescope is similar in spirit and largely inspired by recently proposed tools such as compare-mt @xcite , mt-compareval @xcite , and vizseq @xcite . compare-mt also provides a holistic analysis comparing two mt systems, although with different features. using compare-mt, the user can, for example, look at performance according to n-gram frequency and part-of-speech (pos) accuracy. mt-compareval also provides comparative analysis of segment-level errors with highlighting of variant n-grams. the tool also provides some limited aggregate analysis. both of the above tools also offer statistical significance testing in the form of a bootstrapped t-test. vizseq @xcite , whilst only tangentially related, is one of the only comparative tools that offers a web-based interface. moreover, vizseq has impressive coverage in terms of natural language generation metrics. however, vizseq was developed for multi-model comparison and is primarily focused at corpus-level. other tools such as pet @xcite and ap-praise @xcite are complementary to mt-telescope in that they offer features which leverage annotation and post-edition. ## conclusions and future work mt-telescope is designed to provide robust and insightful comparative analysis specific to the mt use case with state-of-the-art metrics. data visualizations are dynamic, interactive and highly customizable. the tools have been built specifically with ease of use in mind, in the hope of expanding access to high quality mt evaluation. there is tremendous scope in the adaptation of the dcf framework to target many other phenomena and future work will be focused primarily in this area. we envisage for example adding filters for specific discourse phenomenon such as pronoun translation. ideally such filter would allow researchers to measure context usage in nmt without having to rely only on contrastive evaluation @xcite and/or human evaluation. we also plan to extend mt-telescope to handle a (possibly empty) set of references. this will bring more flexibility to the tool allowing more informed decision when multiple references are available while also supporting quality estimation @xcite when references are not available. finally we hope to implement exporting functionality to allow saving of analysis output in commonly used formats (e.g. json and pdf). given that mt-telescope is an open source platform, we are excited to encourage other users to contribute to its growth with suggestions and new features.