Evaluating Text Generation Quality Using Spectral Distances of Surprisal (2025)

Abstract: We propose a novel automatic evaluation metric for open-ended text generation, which is a substantial improvement of the recently developed method, Fourier analysis of cross-entropy (FACE), hence, FACE-2. FACE-2 is a psycholinguistically inspired metric that extracts the dynamic patterns (spectrum) of text surprisal. Examined with open-ended text generation tasks, FACE-2 significantly outperforms a broad set of baseline metrics in revealing the model scaling effect, which scales up to models of 70B parameters, while many other existing metrics fail to capture this effect. We have also confirmed the advantage of FACE-2 in producing stronger agreement with human preferences from a large human-annotated dataset. We advocate for including metrics that mine the dynamics of likelihood in evaluating open-ended text generation, which covers broader aspects of human language than only using static likelihood-based or semantic-based metrics. Code repository: https://github.com/CLCS-SUSTech/FACEScore.

## introduction the surprisal (likelihood) of texts is an important source of information for evaluating the outcome of natural language generation tasks, especially for open-ended generation. ever since the early works that directly use surprisal for evaluation, such as gltr @xcite , @xcite , and @xcite , more sophisticated methods have been recently developed to further harness the potentials of surprisal: some utilize the curvature of surprisal (e.g., detectgpt and fast-detect) @xcite ; @xcite , and some rely on its variance (e.g., gpt-who) @xcite . these works have achieved impressive effect in telling apart model-generated texts from those "authentic" human-written ones. a less-taken way is to use the dynamic property of surprisal, that is, how surprisal changes over curves are spectra of surprisal fit with gam. to better align with the evaluation of open-ended text generation for modern llms, we introduce face-2, an improved version of the original face metric (referred as face-1 throughout the study). our major improvements are: (i) including optimized processing steps of surprisal which is the basis for nlg evaluation; (ii) implementing several new distance functions on spectrum representations which are mathematically more sound. we conclude our contribution as follows: ## face-2 workflow the general workflow of face-2, as described in figure 2 , can be summarized in the following steps: step 1 -estimate surprisal given d as text set, we use an language model evaluator to calculate the surprisal of each sentence in d. for a sentence of n tokens t 1 , . . . , t n , its surprisal sequence @xmath0, . . . , s n -1 is defined as s i ≜log p(t i |t 1 , . . . , t i-1 ), i.e., the negative logprobability for each token estimated by the language model. there |d| pieces of such surprisal sequences produced at this step. based on fouriergpt @xcite , we additionally apply a z-score normalization to the spectrum to improve the numerical stability of the spectrum value. the surprisal sequence becomes: step 2 -fourier transform we treat s as a signal in time domain, and apply the discrete fourier transform (defined in @xcite : note that here we change the index subscript from i to n to avoid confusion with the imaginary unit i, which is denoted by j instead (j 2 = -1). thus s(n) is equivalent to s i . the obtained series of complex numbers @xmath1, where ω @xmath2 n is the k-th frequency component, @xmath3, . . . , n -1. ## experiments in order to bring the evaluation of open-ended text generation in line with the current development of llm, we evaluate face-2, our main objective of experiments, a simple baseline metric, surprisal, which directly uses the log-likelihood of the evaluator model, and all other metrics mentioned in section 2.2 in our extensive large-scale experiments. our experiment would include some widely used llms range from 0.5b to 70b, and test these metrics' agreement with model size and human preference. we evaluate the performance of face-2 compared other metrics, with respect to the following two desiderata: 1. model scaling effect: how well do the metric scores align with the scaling effect of model size, that is, larger models are better than smaller ones. foot_0 (section 4.2) 2. agreement with human preferences: how well do the metric scores align with human preference for text quality. (section 4. ## discussion and conclusion in this study, we address the issue that most of the metrics designed for open-ended text generation are published in the era where llms are not fully developed, therefore it is necessary to rethink their effectiveness in evaluating current llms. spectrum-based methods exhibit high potential; therefore, we proposed an improved version of fourier analysis of cross-entropy, face-2, a metric set to evaluate open-ended text generation in the current llm era.we also carried out largescale experiments within up to 70b llms to update the effectiveness conclusions of many old evalua-tion metrics in the current llm era. our results show that most of the old conclusions have failed, whereas spectrum-based methods maintain its effectiveness to a certain degree. we observed the outperformance of face-2 in experiments of model scaling effects, and the effectiveness and robustness of both spectrum-based methods in experiments of agreement with human preferences. when compared with its predecessor and other metrics, face-2 demonstrates the following advantages: firstly, it can more effectively distinguish texts generated by models of various sizes, i.e., better reflecting the scaling effect. this is observed across various model families and languages, which, more impressively, is achieved without depending on large evaluating models. secondly, with the newly introduced metrics (emd, kl, and js) and the new z-score normalization step, it outperforms face-1 and a broad set of baseline metrics. thirdly, the advantage of face in agreement with human preferences are confirmed in a larger dataset generated by more recent models. in sum, the new evaluation method for openended text generation, face-2, is effective, robust, and computationally efficient. it reaches state-ofthe-art performance. however, no metric can fully capture the complex nature of human languages. as llms keep evolving, the gap between generated and real content will keep shrink. we believe that seeking metrics to magnify this gap is a meaningful response to the eternal question of where the boundary between ai and human is.