h_da@ R epro H umn – Reproduction of Human Evaluation and Technical Pipeline (2023)

Abstract: How reliable are human evaluation results? Is it possible to replicate human evaluation? This work takes a closer look at the evaluation of the output of a Text-to-Speech (TTS) system. Unfortunately, our results indicate that human evaluation is not as straightforward to replicate as expected. Additionally, we also present results on reproducing the technical background of the TTS system and discuss potential reasons for the reproduction failure.

## introduction replication of research results in natural language processing (nlp) has gained considerable attention in the past years. while quite some progress has been achieved with initiatives such as the responsible research checklist 1 and the reproduction checklist 2 @xcite , the question about the reproduction of human evaluation is widely unanswered. the work presented here is part of the reprohum project 3 , which aimes to reproduce human evaluation. in our experiment, we tried to reproduce the evaluation of a low-resource text-to-speech (tts) system for german. as the results of our reproduction indicated that we were unsuccessful, we also had a closer look at the technical aspects of the work and attempted to reproduce those elements for our study as well. our major contributions are therefore: 1) the results on the reproduction of the human evaluation of the tts output, 2) the results of the reconstruction of the language data required for the tts system and 3) the results of the reconstruction of the tts model required to create the tts output, which is then judged during the human evaluation. ## the original tts experiment the basis for our work is the paper by @xcite . its aim is to present the possibility to create tts systems with little training data and reduced training time. this is achieved by using a large multilingual model, which is then fine-tuned towards the target language based on the reduced training data and reduced training time. a specific focus is put to model articulatory features of the language. the technical basis for the model is tacotron2 (shen and pang) and fastspeech2 @xcite . where tacotron2 is based on a recurrent sequenceto-sequence network, fastspeech2 is based on a feed-forward transformer network. the basis for the multilingual model is data from english, greek, spanish, finnish, russian, hungarian, dutch and french. the german data is derived from the hui corpus (see section 6 below). while the multi-lingual model required lots of resources, both in time and hardware, the adaptation to german was performed using 30 minutes of speech and training for about 2 hours. in order to allow for a comparison and to verify the lowresource approach, the authors also trained both fastspeech2 and tacotron2 exclusively on german, using 29 hours of recorded speech. ## reproduction -experimental setup following the original study, we set up a google form survey, where each participant is presented with two stimuli and asked to judge, which of the two sounds more natural. figure 3 shows the interface we used to conduct the survey. as we were dealing with german speech output and german students were asked to judge the tts output, we also addressed participants in german. participants could choose from three different options: either one of the outputs is better than the other, or both are equally good. prior to starting the evaluation, we submitted all relevant information to the university of aberdeen ethics board for evaluation, which approved of our experimental setup, the way we dealt with the data and the personal information collected from the participants. the participants were recruited by email from our university. other than sending out an email via a central email address, we did not collect any personal data from our participants. ## reproduction -results in the end, 37 participants took part in our experiment, which is comparable to the original study. in general, the output from the proposed fastspeech2 model is considered better than the baseline system in 41 % of the cases, while the baseline system is considered better in 13 % of the cases. when comparing the two fastspeech2 versions, 46 % of the participants did not hear any noticeable difference. this is comparable to the original evaluation, where 43 % of the participants did not hear a difference. see also figure 4 . when evaluating tacotron2, 26 % of the participants preferred the low-resource model, while 23 % preferred the original version. but, 51 % of the participants did not hear a difference between the two versions. compared to the original evaluation, where 52 % of the participants preferred the low-resource version, while 11 % preferred the original system and only 37 % did not hear a difference. the results are also shown in figure 5 . as shown in ## technical reproduction in light of these results for the reproduction of the human evaluation we had a closer look at the background of the tts system. first, we tried to reproduce the data and then we aimed to reproduce the tts model. ## discussion figure 7 : pipline to create the tts model according to @xcite these are quite baffling, since none of our approaches reached the same values or results. there are a number of potential reasons for this: the differences in results when reproducing the tts models could be explainable by different hardware or slightly different software versions, especially since we faced issues that the original authors obviously did not encounter. regarding the different results for the reproduction of the human evaluation, one reason could be the different group of people. while both studies employed students to evaluate the synthesis output, in the original study, the students are from the field of computational linguistics and natural language processing and as such more used to hearing and judging synthetic speech. in our study, the students did not have any particular training in judging synthetic speech. another reason could be that the stimuli were somehow mixed up. if that would be case, we would have to transpose the results and would have results that are more comparable to the original study. the problem might be related to the problems with reproducing the original data set and/or the original tts models, since the stimuli were recreated for the purpose of this study 4 , which could have lead to a variance in sound quality compared to the original stimuli. comparing our results to the results of 4 florian lux personal communication. @xcite , who ran the exact same experiment, the chances that the stimuli were transposed somewhere in the process are increasing, as their results also indicate low reproducibility, except if a transposition is assumed. as their results are based on a larger number of participants, they are more pronounced than ours and statistically more reliable. the authors state a range of other potential error sources, which have to be taken into account in addition to our experiments. additionally, it is certainly remarkable that in both reproductions the lowest coefficients of variation were achieved for the "no preference" option. ## conclusion in general, we can support the conclusion of the previous study, that the low-resource speech synthesis (both using tacotron2 and fastspeech2) are viable approaches to produce reasonable tts output based on limited resources (time, computing and available speech data). our results also show, that the reproduction of human evaluation and possibly human annotation as well are important research areas. as quantitative results can only give so much information, while human evaluation in various domains (i.e. synthetic speech, but also text quality in natural language generation) can provide a more detailed insight into the data. unfortunately, the way human evaluation is currently reported, the reproduction of human evaluation has not been successful. with respect to the whole pipeline, of a technical reproduction based on which a human evaluation can take place, it is important to make sure, that research artifacts are stored properly, documented thoroughly and potential pitfalls (i.e. dying links) are noted. our results indicate that more research is necessary into the issue of human evaluation. related to this, it would be interesting to study human annotation tasks, which are related to human evaluation and are the basis of a wide range of models built in the context of nlp.