ARGENT : Automatic Reference-free Evaluation for Open-Ended Text Generation without Source Inputs (2025)

Abstract: With increased accessibility of machine-generated texts, the need for their evaluation has also grown. There are broadly two types of text generation tasks. In open-ended generation tasks (OGTs), the model generates de novo text without any input on which to base it, such as story generation. In reflective generation tasks (RGTs), the model output is generated to reflect an input sequence, such as in machine translation. There are many studies on RGT evaluation, where the metrics typically compare one or more gold-standard references to the model output. Evaluation of OGTs has received less attention and is more challenging: since the task does not aim to reflect an input, there are usually no reference texts. In this paper, we propose a new perspective that unifies OGT evaluation with RGT evaluation, based on which we develop an automatic, reference-free generative text evaluation model (ARGENT), and review previous literature from this perspective. Our experiments demonstrate the effectiveness of these methods across informal, formal, and domain-specific texts. We conduct a meta-evaluation to compare existing and proposed metrics, finding that our approach aligns more closely with human judgement.

## introduction natural language generation (nlg) has progressed significantly in the last decade. this progress has been made through the use of encoder-decoder @xcite and decoder only architectures @xcite . in the last few years, the use of these transformer-based architectures @xcite and increased compute capacity to create generative large language models (llms) such as @xcite ; @xcite has attracted attention from both academia and the public. however, the lack of robust evaluation metrics for generated text has limited the ability to make informed choices among candidate outputs produced by one or more llms. nlg tasks can be categorised on a spectrum between two categories: reflective generation tasks (rtgs) foot_0 and open-ended generation tasks (otgs). in rgts, the output closely reflects the content of the input and must remain faithful to it, such as machine translation and summarisation. ogts, by contrast, involve generating novel content that is not directly grounded in the input, such as story generation or synthetic medical report creation. rather than a strict dichotomy, generation tasks are better understood as positioning on a spectrum of constraint. for example, image captioning and text expansion lie between highly constrained tasks such as translation and unconstrained tasks such as storytelling. many studies on rgts, such as machine translation and summarisation, evaluate output quality by comparing model-generated texts to one or more pre-written human references, using similarity metrics such as bleu @xcite , me-teor @xcite , beer @xcite , bertscore @xcite , bleurt @xcite , and comet @xcite . however, these approaches often depend heavily on reference selection, which can significantly impact evaluation outcomes. more recent work on quality estimation (qe), such as comet-qe @xcite , addresses this issue by evaluating outputs in relation to source inputs without requiring human references @xcite . while this mitigates the problem of reference selection, it remains applicable only to rgts, as it still relies on source inputs. in contrast, open-ended ogts, such as story or dialogue generation, remain under-explored in this context, largely due to the difficulty of defining appropriate references for outputs that are not input-grounded @xcite . as a result, ogt evaluation often relies on distribution-level comparisons between model-generated and human-written corpora in the target domain. common approaches include statistical metrics such as self-bleu @xcite and generation perplexity @xcite , as well as divergence-based techniques such as mauve @xcite , which estimates the difference between synthetic and human text distributions using kullback-leibler (kl) divergence. these evaluation methods have two major problems: (1) in ogt evaluation, they are unable to assess the quality of each individual output; (2) there is no unified conceptual framework for comparing metrics across rgt and ogt paradigms. this limits the transfer of insights and tools between these domains, especially transferring tools from rgt to ogt. this paper addresses these issues by proposing a unified evaluation framework that bridges rgt and ogt evaluation. within this framework, we introduce a new reference-free method for evaluating ogts without source inputs at the level of individual outputs, which we call argent (automatic reference-free generated text evaluation). to benchmark argent, we also develop a meta-evaluation framework to assess the effectiveness of evaluation metrics themselves. the contributions of this paper are as follows: ## bridging ogt with rgt evaluation from a unified framework evaluating language generation differs fundamentally from evaluating traditional classification or regression tasks. in classification, there exists a finite list of output classes; in regression, outputs lie on a continuous and measurable scale. in contrast, most language generation tasks do not have a single correct answer, and many do not even have a finite set of acceptable answers. instead, evaluation typically relies on a set of human-written references. moreover, language generation lacks an inherent numerical ground truth, which requires the use of similarity functions to compare generated text to references. we illustrate this complexity in appendix a with a simple translation example to demonstrate how evaluation outcomes vary depending on (1) the references selected, and (2) the similarity function used. in any evaluation of a text generation model, we can identify the following components: the corresponding optimal reference, which depends on both the model output and the chosen similarity function, is defined as: key points arising from this formulation include: ## auto-evaluation for language quality the large reference space in ogt evaluation leads to a challenge: how can we identify the closest reference to a given model output? one solution is to use output-oriented human annotation, in which a human judge corrects errors in an output by making the minimum number of changes, to give an error-free text. this revised text can then serve as the closest reference, and the output-reference pair can be used for evaluation. this technique has been applied in in rgts, such as machine translation, where it has been shown to gives scores more aligned with human judgement than pre-written references with a translation edit rate metric @xcite . however, such output-oriented evaluation is costly and does not scale. we could overcome this with an automatic evaluation, but autoevaluation may itself vary in quality, with some methods providing results more aligned with human judgement than others. we therefore need to consider ways in which we might measure the quality of auto-evaluations. the remainder of this paper discusses a new reference-free auto-evaluation method, argent, and meta-evaluations of argent and existing true evaluation reference closest to the candidate perfect similarity metrics an evaluation reference(s) similarity metrics pre-written output-oriented bleu rouge bert score bleurt auto-evaluation rerence: real text candidate: corrupted real text bleurt, corruption count based real webtext train/dev + corrupted texts clinical train/dev + corrupted texts train test test webtext test + synthetic + annotation pairs of academic paper + synthetic synthetic data only hold out clinical text spearsman correlation machine ranking & human ranking real text detection accuracy area size of pick up graph language quality auto-evaluation generative evaluation theory previous works ideal evaluation argent this work meta-evaluation proxy pair pretrain on corrupted real texts zero-shot learning on synthetic texts metrics under different dataset conditions. figure 1 shows the relationships between evaluation, ideal evaluation, auto-evaluation methods argent, and meta-evaluation presented in this paper. ## experiments data and metrics: to evaluate our framework, we conducted experiments on three types of text: formal, informal, and domain-specific. we report results using three meta-evaluation criteria: correlation with human scores, pairwise accuracy, and the area under the pick-up rate curve. details of the datasets and meta evaluations used for each type are provided in the corresponding subsections below. auto-evaluation models: unless stated otherwise, all argent auto-evaluation models reported in this paper are based on the bert-base cased architecture (12 layers, 768 hidden units, 12 attention heads) @xcite . argent models are pre-trained on corrupted texts and applied directly to test tasks, consisting of either machine-generated or real texts, without fine-tuning on the test data. for pre-training, we use a batch size of 32, a learning rate of 1e-5, and train for 3 epochs. the model contains approximately 110 million parameters and was trained on a single nvidia a100 gpu. pre-training dataset: unless stated otherwise, all pre-training datasets are constructed by applying inflection and local shuffling to real texts. we perform a grid search over inflection and shuffling probabilities in the range {0.2, 0.4, 0.6, 0.8, 1.0} for each corruption method. for shufflection, we use a pair of probability values, one for inflection and one for shuffling, that give the best performance for each method individually. each corrupted text in the pre-training dataset is assigned a quality score using both the corruption-count-based method and the bleurt-based method. ## conclusion in this work, we proposed a unified framework for evaluating machine-generated text that applies to both rgts and ogts. building on this framework, we developed argent, a novel reference-free auto-evaluation method for assessing the language quality of open-ended generation. argent requires no human annotation and operates without relying on source inputs or reference corpora. we evaluated argent across diverse text types and benchmarked it against several commonly used evaluation methods. our results show that ar-gent outperforms all competing models except for mauve on the webtext dataset, where it ranks second. however, unlike mauve, argent does not require a human reference corpus during evaluation and can assign quality scores at the level of individual outputs, rather than only at the model level. finally, we reviewed the existing evaluation literature through the lens of our proposed framework, categorising prior methods based on reference selection strategies and similarity metric design.