Automatic normalization of noisy technical reports with an LLM : What effects on a downstream task? (2025)

Abstract: This study explores the automatic normalization of noisy and highly technical anomaly reports by an LLM. Different prompts are tested to instruct the LLM to clean the text without changing the structure, vocabulary or specialized lexicon. The evaluation of this task is made in two steps. First, the Character Error Rate (CER) is calculated to assess the changes made compared to a gold standard on a small sample. Second, an automatic sequence labeling task is performed on the original and on the corrected datasets with a transformer-based classifier. If some configurations of LLM and prompts can reach satisfying CER scores, the sequence labeling task shows that the normalization has a small negative impact on performance.

## introduction this study focuses on the automatic cleaning of technical and noisy texts and its impact on an automatic fine-grained semantic labeling task. the goal is to assess the capacity of a generative llm to automatically rectify noise phenomenons in technical texts that are going to be automatically processed afterwards. the dataset used in this work is composed of french written anomaly reports during ariane 5 rocket maintenance operations. these types of maintenance records have proven to be not only filled with words from a technical specialized lexicon, but also extremely noisy (text in uppercase, missing accents, spelling errors and misuse of punctuation) @xcite . as such, we chose to explore the cleaning of the noise by an automatic rectification task performed by prompting a generic pretrained large language model (llm). three different llms were evaluated, with four different prompts covering different levels of information. a first intrinsic evaluation compared the output of the llm compared to a gold standard. a second, extrinsic evaluation consisted in measuring the performance of an automatic sequence labeling task. we compare the same classifier when trained and applied to the corrected versus original of the annotated data. the results of the automatic semantic labeling allow us to determine whether these corrections are beneficial to the fine-grain semantic analysis of those texts. this article is organized as follows. section 2 is a short review of related work on noise in technical text data. in section 3, we present the dataset of french anomaly reports and the methods used to correct the noise. the results of the intrinsic evaluation are presented in section 4 and the sequence labelling task and its results are presented in section 5. ## related work reporting anomalies is a common procedure in the space and aviation domain, as it is encouraged and has become part of the general culture among professionals. numerous studies have been conducted on using nlp (natural language processing) on aviation anomaly reports showing that a number of different techniques can be of use in the treatment of such texts @xcite , ranging from text classification to information retrieval @xcite , @xcite . the same kind of anomaly reports dataset used in this work, focusing on maintenance operations on ariane 5 rockets, has already been the object of nlp experiments in @xcite ; @xcite but with other objectives (assessing risk level) and based on a coarser grain text analysis. maintenance reports have also been shown to be particularly noisy and technical texts @xcite , and thus are difficult to process by the usual nlp pipelines conceived for (and from) standardized texts @xcite , @xcite . several studies have already explored ways to clean this type of texts, from rule-based approaches @xcite to lexical normalization techniques @xcite . the use of generative llms for correction has also been studied, for post-ocr noisy texts in @xcite and @xcite and seems to provide better error reduction rates. @xcite has also shown promising results in the use of an llm to clean noisy texts while preserving their semantic integrity. @xcite confirm these results, but also shows that the performance of the llm varies according to the type of noise and that some models have a better ability to perform this task. ## corpus, noise and automatic normalization our dataset contains 1050 anomaly reports written in french in an industrial setting. this sample was randomly extracted from a much larger database with tens of thousands similar items. these reports are produced systematically every time an irregularity (however trivial) is encountered by an operator in a critical environment. as can be seen in for this experiment, we selected three smallsized quantized llms that could be run locally on a workstation (a constraint due to the confidentiality of the target data), able to process french and which reach state-of-the-art performance in generic benchmarks: meta-llama-3.1-8b-instruct, meta-llama-3-8b-instruct @xcite and mistral-7b-instruct-v0.3 @xcite . for each model, four different prompts were defined with incrementally additional information. the first one included only the context of creation of this dataset (i.e. operators reporting anomalies during the maintenance of a rocket) and the requested task (i.e. remove the noise phenomenons without altering the meaning). in the second prompt we added the goal of this operation : to prepare the text for further processing by a non-specified nlp program. for the third one, a list of the different expected types of noise to rectify was given. and finally, in the fourth prompt, two reports and their rectified versions were given as examples (few-shot prompting) (cf. appendix a). as is commonly recommended in such cases, all four prompts were written in standard english, with the explicit indication that the source and target texts are in french @xcite . llm temperature was set to zero, resulting in deterministic outputs and thus not requiring several runs. ## intrinsic evaluation of the correction for a first evaluation of the results, the character error rate (cer) was calculated on a gold standard of 15 manually normalized reports by the authors. a selection of reports were chosen randomly and 15 were selected to get a representation of all the different noise phenomenons. the correction process is not trivial as each word needs to be corrected, at least by putting the correct case back, decisions have to be made regarding abbreviations and punctuation, some words can be ambiguous due to the lack of accents... the examples in ## evaluation on a downstream task the second experiment conducted in this study consists in an automatic annotation of the 6 datasets (the original reports and the five datasets with a low cer) through a sequence labeling task. the original reports dataset was manually annotated based on a twelve-class typology of sequences. these classes are related to the main type of technical problem reported (i.e. "leakage", "malfunction", "missing component"...). the annotated text segments are lexical markers (cues) of the class ("leak", "leaking", "absence", "missing", "not present"...). the annotation was performed by three linguists. the inter-annotator agreement between the linguists and a field expert was measured with a gamma score @xcite of 0.63. in the first example in to reverberate the manual annotation from the original reports to the corrected reports, adjustments had to be done. to that effect, the corresponding offsets of the target sequence in the corrected versions were determined based on the output of the gnu wdiff utility 3 , with local homothety transformations for adapting to insertions and deletions. the sequence labeling task was then re-evaluated, using the five corrected versions of both the training and test sets, without any other preprocessing except using the lowercase for bertbase-multilingual-uncased. the results of the automatic sequence labeling shown in ## conclusion in this study, we have demonstrated that the automatic correction of a technical and noisy text with an llm produces mitigated results. the scores given by the cer seemed satisfactory enough to assume an efficient correction of the noise, at the condition that the llm is accurately prompted (context, goal of the normalization and list of phenomenons to correct). however, the results of the sequence labeling task do not confirm this hypothesis. some semantic information may be lost and lead to a negative impact on the sequence labeling task. in the example in figure 1 , we can see a case where the llm overcorrected and modified a critical word. the word "defectueux" (tr. "defective") was replaced by "défaut" (tr. "defect"). in the reports, "defect/défaut" is often found and used with the meaning of "problem" or "inadequacy". as such, it has been manually labeled most of the time with the label "out of specification". it differs from "defectueux/ defective" which means that a component is not functioning, thus labeled with "not working". in this example, by changing this particular word, the llm has modified the meaning of the sentence and even its correctness (in this case "est défaut" is nor grammatically accurate, nor attested in the corpus). the classifier applied on the rectified text thus incorrectly labels "est défaut" as "out of specification", while the original text get a correct label of "not working" for "de-fectueux". to conclude, we can say that the use of transformers models on noisy and technical data seems to be quite robust and able to cope with such a corpus, addressing the main types of noise. however, the noise itself does not seem to bear the difficulty of the sequence labeling task given the score obtained on the normalized dataset close to the score of the original reports dataset.