Generative AI for Technical Writing: Comparing Human and LLM Assessments of Generated Content (2025)

Abstract: Large language models (LLMs) have recently gained significant attention for their capabilities in natural language processing (NLP), particularly generative artificial intelligence (AI). LLMs can also be useful tools for software documentation technical writers. We present an assessment of technical documentation content generated by three different LLMs using retrieval-augmented technology (RAG) with product documentation as a knowledge base. The LLM-generated responses were analyzed in three ways: 1) manual error analysis by a technical writer, 2) automatic assessment using deterministic metrics (BLEU, ROUGE, token overlap), and 3) evaluation of correctness by LLM as a judge. The results of these assessments were compared using a Network Analysis and linear regression models to investigate statistical relationships, model preferences, and the distribution of human and LLM scores. The analyses concluded that human quality evaluation is more related to the LLM correctness judgment than deterministic metrics, even when using different analysis frameworks.

## introduction technical communication means creating content based on factual data, as consistently and clearly as possible, so that users can easily understand complex technical concepts. various professionals are involved in it, such as technical translators, developers, information architects, and technical writers (society for technical communication, n.d.). technical documentation provides specialized and task-oriented information for the user on how to use and interact with a given product. it is not feasible to cover all possible use cases; instead, the focus should be on the main functionalities or use cases to maintain objectivity. @xcite . llms can be useful not only for code generation but also for technical writing because they can simplify the documentation process by generating drafts when prompted with code snippets. this can facilitate the work of technical writers and reduce the effort needed for research. a good model could lower the technical barriers, automate lengthy tasks, and act as an extra solution for their problems @xcite . however, due to several facts, one of them possibly being outdated training data, llm-based chatbots can also hallucinate information that does not accurately reflect reality. an alternative to tackle this issue is rag. it allows external data to be incorporated into the model, which improves its ability to provide more relevant or up-to-date responses, based on the data used to implement the rag method @xcite . this study leverages the content produced by a chatbot using multiple llms (gpt-3.5 turbo, gpt-4.0, and mistral ai 7b) and rag technology on a specific topic: network as code (nac) technical product documentation. briefly, nac simplifies the programming of networks, such as automatically adjusting streaming capabilities and improving bandwidth, for example, for online games or concert streaming @xcite . the llm responses were evaluated by a technical writer using an error-typology framework and an llm as a judge based on answer correctness. the aim of this paper is to understand how different (automatic and human) evaluations are distributed according to the attributed scores and how, or if, they relate. additionally, the ultimate purpose is not to show which evaluation is the best but to offer insights on how performing different analyses, for instance, automatic @xcite and qualitative can complement one another. we first discuss relevant work on evaluating llm output quality (section 2), and then present the dataset and evaluation approaches used in this study (section 3). we report the comparison of human, deterministic, and llm evaluations using linear regression models, score distributions section 4.1), and network analysis (section 4.2). finally, we discuss the implications of the comparison (section 5) and present the conclusions and directions for future work (section 6). ## discussion as seen in the (4.2), human evaluation is still somehow correlated with deterministic metrics, even though not strongly, while llm evaluation is peripheral, only associating with the human one. overall, the deterministic metrics had higher correlations amongst themselves, mostly likely due to the token overlap between ground truth and generated content. it is important to remember that "hallucinated" responses can also have high token overlap without necessarily generating more accurate responses relevant to a real use-case scenario. on the other hand, human and llm quality analyses seem more closely related to each other, even when using different frameworks to evaluate quality. this shows that an increase in the human evaluation variable relates to an increase in the llm one. additionally, the linear regression model plots seemed to reflect similarities in model preference by both human and llm judgment. however, as the distribution of human quality evaluation scores had several peaks, it could indicate that the human evaluation had a more varied score distribution, due to the detailed error typology followed by the quality analysis type, while the llm analysis mostly considered correctness and relevance as the main reference. furthermore, as illustrated by the qualitative example (4.3), the assessments of a human evaluator and the llm do not always correspond to each other. ## conclusion and future work this paper analyzed technical documentation content generated by three different llms using rag technology and compared the responses to product documentation. different types of analyses were done: a qualitative analysis using dqf-mqm error typology by a technical writer, an automatic llm correctness assessment, and a deterministic evaluation using bleu, rouge, and token overlap. the evaluation scores were contrasted and compared using network analysis, linear regression models, and a histogram with score distribution. deterministic metrics have strong relationships with each other, while human analysis correlates moderately with llm analysis and weakly with deterministic metrics. the dqf-mqm translation quality evaluation framework was found to be a useful model also for the evaluation of technical writing content and shows potential for improving evaluation methods in the generative ai field. the current study is limited by involving only one technical writer as an evaluator, but the approach can provide a basis for further studies involving multiple evaluators. future work could also include integrating such qualitative evaluation frameworks to evaluation packages like continuous-eval for a more granular approach to evaluating llm output. overall, the evaluation results indicate that llms are not yet ready as producers of novel content or standalone solutions for technical writing content. future research is likely to continue exploring methods for enhancing chatbot responses for technical documentation purposes through rag or different prompting techniques such as chain-of-thought @xcite . developing novel and reliable quality evaluation methods is therefore also an essential challenge for positive advancements in this area.