A Parallel Evaluation Data Set of Software Documentation with Document Structure Annotation (2020)

Abstract: This paper accompanies the software documentation data set for machine translation, a parallel evaluation data set of data originating from the SAP Help Portal, that we released to the machine translation community for research purposes. It offers the possibility to tune and evaluate machine translation systems in the domain of corporate software documentation and contributes to the availability of a wider range of evaluation scenarios. The data set comprises of the language pairs English to Hindi, Indonesian, Malay and Thai, and thus also increases the test coverage for the many low-resource language pairs. Unlike most evaluation data sets that consist of plain parallel text, the segments in this data set come with additional metadata that describes structural information of the document context. We provide insights into the origin and creation, the particularities and characteristics of the data set as well as machine translation results.

## introduction the software documentation data set for machine translation is created by sap foot_0 as evaluation data for the machine translation (mt) research community. the data originates from the sap help portal foot_1 that contains documentation for sap products and user assistance for product-related questions. the current language scope is english (en) to hindi (hi), indonesian (id), malay (ms) and thai (th). the data has been processed in a way that makes it suitable as development and test data for machine translation purposes. for each language pair about 4k segments are available, split into development and test data. the segments are provided in their document context and are annotated with additional metadata about the document structure. the metadata provides information such as document and paragraph boundaries as well as the segment's text type, for example whether it is a title or table element. such information will surely be valuable when developing and evaluating document-level mt or for tuning systems for specific text types in order to increase the overall translation quality in this domain. the software documentation data set for machine translation as described in this paper is available under the creative commons license attribution-non commercial 4.0 international (cc by-nc 4.0). it is available on github under https: //github.com/sap/software-documentationdata-set-for-machine-translation. it has been released by sap for the 7th workshop on asian translation (wat 2020). @xcite we will first provide some context, explaining the role of test data in machine translation and referring to related work (section 2). we will then describe the origin of the software documentation data set in section 3, including the data preparation and data selection. section 4 is dedicated to the characteristics of the data set. benchmarking results of mt systems on the test sets are provided in section 5. section 6 concludes. ## context test sets are typically used for comparison in mt evaluation campaigns, such as wmt 4 and wat 5 , in which different participants, or rather their systems, compete against each other on specific tasks. subsequently, those test sets are typically also used in research publications to demonstrate the effectiveness of the approach at hand and to compare to previous results. as such, test sets play a crucial role in showing the progress of machine translation. for many years, test sets have been prevalently drawn from news articles. @xcite however, to be able to assess machine translation quality in a wider range of usage scenarios, it is important to also evaluate in other domains than news, and thus to create and establish test sets from a wider range of domains. clearly, specific usage scenarios have other challenges than what is represented in the news domain. thus, quality results, and claims about human parity (e.g. hassan et al., 2018), that have been achieved in the news domain can usually not be directly transferred to other domains. accordingly, data sets and shared tasks have been created for other domains as well, e.g. biomedical foot_6 and patents @xcite . with the software documentation data set, we provide the possibility to tune and evaluate mt systems in the domain of corporate software documentation, and thus contribute to a clearer picture of the quality of machine translation across domains. similarly, the focus of machine translation has often been on high-resource language pairs, such as english-german. with an evaluation data set for four language pairs that are rather on the lower end of availability of resources, we contribute to a better test coverage for the many low-resource language pairs. with the recent improvements in machine translation quality, up to claims of human parity, flaws in the evaluation setups and interpretation of results have been pointed out (toral et al., 2018; läubli et al., 2018; bojar et al., 2018). subsequently, more emphasis has been put on carefully evaluating machine translation, in particular to be able to evaluate segments within their document context, e.g. by barrault et al. (2019). by creating data sets that consist of documents corresponding to help pages, we contribute to this endeavor. the document structure annotation can also provide additional useful information during human evaluation. similarly, machine translation approaches have started to look beyond translating independent sentences. methods for taking more context into account have emerged, with the goal to improve the translation quality (miculicich et al., 2018; maruf and haffari, 2018; yu et al., 2020, amongst others). by providing development and test data with document context and metadata, we hope to strengthen such developments. resources that are related to the data set at hand in terms of the covered domain are the data sets from the wmt16 shared task of machine translation of it domain (bojar et al., 2016, section 4) and the documentation data set by salesforce (hashimoto et al., 2019). the data set from the it translation shared task consists of answers from a help desk, thus it covers a different text type than software documentation that likely also comes with a different style. furthermore, the focus of the data set is on european languages, and it does not contain more context than short one-paragraph answers. the data set described and experimented with by hashimoto et al. (2019) is very similar in nature to ours. note however that the language scope is different: all language pairs in the data set by salesforce are rather high-resource. ## characteristics of the data set the selected documents with reduced redundancy, see section 3.4, were divided into development and test data sets and constitute the software documentation data set for machine translation. we will look into its characteristics in this section. while the number of segments of the development and test sets are in the same range across language pairs, the number of documents and the total amount of words are different for en-hi and en-id compared to the other two language pairs. this difference is also reflected in the distribution of words per segment, see figure 5 : there is a larger number of short segments for en-hi and en-id. for en-ms and en-th, we see a more balanced distribution of short and medium length segments in both, development and test sets. figure 6 shows the distributions of textual element annotations in the data sets' metadata. they explain, to some extent, the distribution of segment length: we see a larger number of section segments for en-ms and en-th. sections usually contain longer segments than table elements, which are frequent for en-hi and en-id. finally, we look at the redundancy in the released data set, i.e. the number of all source-target pairs over the number of unique source-target pairs, shown in in summary, we conclude that the data sets for en-hi and en-id are comparable concerning the criteria analyzed in this section. they differ somewhat from the en-ms and en-th data sets that also have characteristics in common. we would have preferred to provide a more homogeneous data set. however, given the different sizes and features of the original resources and the constraints imposed by adding contextual metadata, this was not feasible. on the other hand, the charts in this section indicate that the development and test sets of each language pair share the same characteris- ## machine translation results in this section, we provide reference mt results on the test sets of the software documentation data set for machine translation. they should serve as comparison for future research evaluated on this data set. the results can be found in the first reference mt result is based on the baseline system of the wat 2020 nict-sap it and wikinews task foot_13 , provided by the organizers of the task. there are two multilingual systems, one for en to {hi, id, ms, th} and one for {hi, id, ms, th} to en. they are trained on wikinews data from the asian language treebank (alt) project (riza et al., 2016) and it data from opus foot_14 (ubuntu, gnome and kde4) (tiedemann, 2012). a transformer-big configuration (vaswani et al., 2017) was used. more details can be found in the workshop overview (organizers, to appear). the second reference mt system that we report on is a popular general purpose online mt provider. the contextual metadata of the test set is not used in either reference system. we report case-sensitive bleu (papineni et al., 2002) and chrf (popović, 2016) scores as calcul-cated by sacrebleu 18 (post, 2018). the bleu scores for en-th are generated based on charactersegmented input. for the wat system, we additionally report the official bleu scores as provided by the task evaluation (cf. column bleu * in ## conclusion we released the software documentation data set for machine translation to the mt research community, a high-quality real-world data set with content from the sap help portal. to our knowledge, it is the first data collection with explicit text structure annotation and the first it-specific evaluation data set for english to hindi, indonesian, malay and thai. it will advance automatic quality assessment of context-aware mt systems, giving users the flexibility to consider all or only selected or no text structure metadata. moreover, it facilitates the development and testing of machine translation systems for low-resource language pairs for the translation of software documentation in a corporate context. as a starting point, we provided mt results that serve as benchmarks in future research.