Reproducing a Comparative Evaluation of G erman Text-to-Speech Systems (2023)

Abstract: This paper describes the reproduction of a human evaluation in Language-Agnostic Meta- Learning for Low-Resource Text-to-Speech with Articulatory Features reported in Lux and Vu (2022). It is a contribution to the ReproNLP 2023 Shared Task on Reproducibility of Evaluations in NLP. The original evaluation assessed the naturalness of audio generated by different Text-to-Speech (TTS) systems for German, and our goal was to repeat the experiment with a different set of evaluators. We reproduced the evaluation based on data and instructions provided by the original authors, with some uncertainty concerning the randomisation of question order. Evaluators were recruited via email to relevant mailing lists and we received 157 responses over the course of three weeks. Our initial results show low reproducibility, but when we assume that the systems of the original and repeat evaluation experiment have been transposed, the reproducibility assessment improves markedly. We do not know if and at what point such a transposition happened; however, an initial analysis of our audio and video files provides some evidence that the system assignment in our repeat experiment is correct.

## introduction the work reported in this paper has been carried out as part of a multi-lab, multi-test study in the context of the reprohum project (belz et al., 2023) and the repronlp shared task. the goal of the project is to assess the reproducibility of human evaluations in natural language processing and to find out which factors contribute to making human evaluations more or less reproducible. our contribution attempts to reproduce an evaluation in a paper from track c of repronlp 2023, lux and vu (2022), which presents a language-agnostic low-resource approach for text-to-speech (tts). the human evaluation is carried out on german audios generated with four different text-to-speech systems. we first (section 2) describe the approaches of the original experiment and our reproduction in detail. in section 3, we present the answer distribution of our results (section 3.1) and the reproduction targets (section 3.2). in section 3.3 we then compare the results of both studies in terms of the scores obtained by each model and report the coefficients of variation (cv*), which quantify the variability of original-reproduced measurement pairs. we also report pearson's correlation coefficients between the original and the reproduction system measurement sets. these results show very low reproducibility (large cv* and low pearson correlations) and we notice a strong cross-similarity between the system results, meaning that the original results for one system are very similar to repeat results for the other, and vice versa. therefore, in section 3.4 we also re-evaluate the results with an assumed system transposition and find improved reproducibility (lower cv* and very high pearson correlations). in the light of these results, after ruling out some error sources (section 4), we compare the mel-frequency cepstral coefficients (mfcc) of the audio and video files used in the repeat evaluation (section 4.1). the results indicate that the system assignments in our repeat experiment are likely to be correct. in section 5, we discuss our findings and in section 6 we briefly compare our results with those of another reproduction submitted to repronlp 2023. all our resources are publicly available. foot_0 ## evaluation experiments in this section we first (section 2.1) describe the original experiment and then (section 2.2) our reproduction. ## results below, we first (section 3.1) present the results obtained in our reproduction study. we then show the reproduction targets (section 3.2) and compare our results to the original study, assessing their reproducibility (section 3.3). since we find that the original results for fastspeech are very similar to the repeat results for tacotron and vice versa, we add section 3.4, where we redo the comparisons and reproducibility assessments after transposing the system labels of our results. note that we cannot be certain that such a transposition happened. ## analysing potential error sources our analysis show a more positive reproducibility assessment for system-transposed results. the current section is an attempt to assess potential sources of this supposed transposition error. we were able to verify the following: we cannot assess potential error sources (1a) and ( 2 ), since we do not have access to the required materials from the original study. therefore, below we analyse the likelihood of option (1b) by comparing the audio files with the generated videos. ## discussion and conclusions the positive aspects of this evaluation were that the original authors were able to provide the exact prompts, instructions, and questions used for the evaluation as well as information on how they set up the evaluation, so the setup was relatively straightforward. however, the question randomisation in the survey form could not be reproduced. as for reproducibility, our initial assessment completely fails to confirm the results of the original study (see section 3.3). once we assume a transposition of systems (section 3.4), we can paint a more positive picture with strong positive correlations and agreement. however, even in the transposed scenario, the per-label aggregation does not fully agree with the per-user preference data (pppd): in a binomial test, we reject the null hypothesis that the per-label aggregated means could be drawn from the per-user preference data distribution. it appears that, when we aggregate on the question level, as opposed to the user level, we smooth over some within-rater variability. the low inter-annotator agreement (krippendorff's alpha) further underscores that there are disagreements between the different evaluators. for both these assessments, the binomial tests and the interannotator agreement, we do not have any comparison to the original study since these data were not reported. finally, it is unclear in which study the hypothesised transposition happened. we can only confirm that for one of the systems there is a relatively clear preference for the proposed low-resource model (as opposed to the baseline), but we do not know for certain whether this is fastspeech or tacotron. test outcome reproducibility table/figure ref. type (i) results high cv* values not reproduced