Reproducing a Comparison of Hedged and Non-hedged NLG Texts (2021)

Abstract: This paper describes an attempt to reproduce an earlier experiment, previously conducted by the author, that compares hedged and non-hedged NLG texts as part of the ReproGen shared challenge. This reproduction effort was only able to partially replicate results from the original study. The analyisis from this reproduction effort suggests that whilst it is possible to replicate the procedural aspects of a previous study, replicating the results can prove more challenging as differences in participant type can have a potential impact.

## introduction there has been within recent years a great interest in understanding and quantifying the reproducibility of experiments across several areas of scientific research. this also includes experiments in the field of natural language understanding (nlu), where researchers have questioned the degree to which experiments and results can reliably be reproduced. recent working exploring the reproducibility of past nlu work has found significant issues such as only a minority of systems reproducing previously reported scores and systems not working due to non-functional code or resource limits @xcite . additionally, there has been growing awareness of systematic issues with regards to how human evaluations are being conducted. in particular, the lack of standardisation and significant under reporting of key human evaluation details @xcite . these twin concerns has led to the creation of the reprogen shared task @xcite , which attempts to check the reproducibility of human evaluations within the field of natural language generation (nlg). as part of the reprogen shared task 1 , a reproduction experiment was attempted for a previous 1 reprogen - https://reprogen.github.io/ work that the author had previously conducted in 2007. in this previous work a human evaluation was conducted between nlg texts containing hedge phrases and those that do not @xcite . this past experiment was conducted to better understand the impact of hedge phrases can have when introduced into a data-to-text nlg system. this was done in order to understand how such systems should communicate potentially emotionally sensitive information to a given reader. in this paper we will describe the experimental setup used and the differences that were made in the reproduction experiment (section 2), the results obtained and how they compare to the ones originally obtained (section 3), and finally we will discuss the significance of the results obtained in this reproduction effort (section 4). ## reproduction results the results from the reproduction experiment along with the original experiment results for the native and fluent english speaker groups are shown in for native speakers, the results of the reproduction confirm the initial findings that native speakers prefer the non-hedged over the hedged texts. interestingly, like the original study native speakers tend to prefer the non-hedged texts to a higher degree than compared to fluent speakers. although this effect is less pronounced than compared to the original study. a two-sample t-test was conducted to compare the mean rating score of the native and fluent speaker groups for both scenarios 2 . for the first scenario the result was t(9)=-0.301, @xmath0.769 and for the second scenario it was t(9)=-0.056, @xmath1.956. the statistically non-significant p-values for both scenarios indicate that the mean rating scores given by both groups for each scenario are not statistically different from each other. analysis of free-text comments from fluent speakers across both scenarios showed that participants found the hedges "didn't add value" and that the non-hedged texts were more "formal" and "professional". these comments align with the general comments from native speakers from the original study. it is possible that the use of fluent speakers with professional experience of using english results in cultural expectations that are closer to that of native speakers than compared to fluent speaking students of the original study. therefore the need for hedges to act as "emotional navigators" are significantly diminished for non-native fluent speakers. ## conclusion in this paper we have conducted a reproduction of a previous nlg study. unfortunately, we have only been able to only partially replicate the results from the original study. whilst, we were able to confirm the findings for native speakers we were not able to do so for fluent speakers. this suggest two things. firstly, that reproduction is a necessary step to better understand the validity of results obtained in initial experiments. and until those results have been validated by a reproduction effort such results should be taken with a degree of scepticism. the second key point is that results obtained in earlier studies cannot be generalised beyond a particular target group of human participants until a reproduction effort confirms the same effect with a different audience. in the case of this study, the original experiment was conducted with master students. it is possible the effects found maybe limited to that audience in particular. therefore, it is critical that key demographic information is recorded in human evaluiations to enable future reproduction efforts to have the correct participant mix for their experiments. the two key limitations of this reproduction effort is the differences in participant types and the lack of non-fluent english speakers recruited for the study. therefore, due to the second limitation in particular, it was not possible to confirm or reject a key claim from the previous study that non-fluent speakers prefer texts that contain hedge phrases. this remains an area open for a possible future follow-up reproduction effort.