Handling Extreme Class Imbalance in Technical Logbook Datasets (2021)

Abstract: Technical logbooks are a challenging and under-explored text type in automated event identification. These texts are typically short and written in non-standard yet technical language, posing challenges to off-the-shelf NLP pipelines. The granularity of issue types described in these datasets additionally leads to class imbalance, making it challenging for models to accurately predict which issue each logbook entry describes. In this paper we focus on the problem of technical issue classification by considering logbook datasets from the automotive, aviation, and facilities maintenance domains. We adapt a feedback strategy from computer vision for handling extreme class imbalance, which resamples the training data based on its error in the prediction process. Our experiments show that with statistical significance this feedback strategy provides the best results for four different neural network models trained across a suite of seven different technical logbook datasets from distinct technical domains. The feedback strategy is also generic and could be applied to any learning problem with substantial class imbalances.

## introduction predictive maintenance techniques are applied to engineering systems to estimate when maintenance should be performed to reduce costs and improve operational efficiency @xcite , as well as mitigate risk and increase safety. maintenance records are an important source of information for predictive maintenance @xcite . these records are often stored in the form of technical logbooks in which each entry contains fields that identify and describe a maintenance issue @xcite . being able to classify these technical events is an important step in the development of predictive maintenance systems. in most technical logbooks, issues are manually labeled by domain experts (e.g., mechanics) in free text fields. this text can then be used to classify or cluster events by semantic similarity. classifying events in technical logbooks is a challenging problem for the nlp community for several reasons: (a) the technical logbooks are written by various domain experts and contain short text entries with nonstandard language including domain-specific abbreviated words (see original entry pre-processed entry fwd eng baff seeal needs resecured. we address the aforementioned challenges with a special focus on exploring strategies to address class imbalance. there is wide variation in the number of instances among the technical event classes examined in this work, as shown in figure 1 and this paper addresses these research questions: rq1: to which extent does the class granularity and class imbalance present in technical logbooks impact technical event classification performance, and can a feedback loop for training data selection effectively address this issue? rq2: which classification models are better suited to classify technical events for predictive maintenance across logbook datasets representing different technical domains? the main contributions of this work include: most expert-domain datasets containing events have focused on healthcare. for instance, altuncu et al. (2019) analyzed patient incidents in unstructured electronic health records provided by the u.k. national health service. they evaluated a deep artificial neural network model on the expertannotated textual dataset of a safety incident to identify similar events that occurred. deléger et al. (2010) proposed a method to deal with unstructured clinical records, using rule-based techniques to extract names of medicines and related information such as prescribed dosage. savova et al. (2010) considered free-text electronic medical records for information extraction purposes and developed a system to obtain clinical domain knowledge. @xcite proposed the cascade methods of extracting the medication records such as treatment duration or reason, obtained from patient's historical records. their approach for event extraction includes text normalization, tokenization, and context identification. a system using multiple features outperformed a baseline method using a bag of words model. @xcite proposed the lung disease phenotypes identification method to prevent the use of a handoperated identification strategy. they employed nlp pipelines including text pre-processing and further text classification on the textual reports to identify the patients with a positive diagnosis for the disease. based on the outcome, they achieve tech. event or issue label example instance of technical logbook entry abbr., misspelling, terminology substantial damage (1) aft on taxi, wing strueck fuel truck, chandler, az aft, wing, strueck, fuel baffle damage (2) r/h fwd upper baff seal needs to be resecured r/h, fwd, bafl minor damage (1) saw sml flock flying upon ldg flare, across rwy sml, ldg, rwy unknown (1) no dmg. bird remains on f/o windscreen dmg, f/o, windscreen pm service (3) pm services check tires for leaks check plow batt pm,tires, plow, batt driving issue (4) failure to yield right, ove correcting over steering ove, steering stop sign running (4) motorists regularly illegal u-turns in r/hour u-turns, r/hour building pm (5) the a/c unit in the kitchen on 3th floor dmg/leak a/c, dmg eng need repair (3) change oil & filter: l/h eng, check comp & plugs oil, eng, l/h, comp, plugs preventive maint (5) reset boiler #2 tmer, checked bldg. throughout boiler, bldg there is also relevant research on event classification in social media. for example, @xcite proposed an open-source event extraction and supervised tagger for noisy microblogs. @xcite applied word embedding-based modeling for information extraction on news-wire and tweets, comparing named entity taggers to improve their method. @xcite performed experimental work on instagram text using weakly supervised text classification to extracted clothing brand based on user descriptions in posts. the problem of class imbalance has been studied in recent years for numerous natural language processing tasks. tayyar @xcite studied automatic propaganda event detection from a news dataset using a pre-trained bert model. they recognized that the bert model had issues in generalizing. to overcome this issue, they proposed a cost-weighting method. al-azani and el-alfy (2017) analyzed polarity measurement in imbalanced tweet datasets utilizing features learned with word embeddings. @xcite studied the class imbalance problem in the task of discourse relation identification by comparing the accuracy of multiple classifiers. they showed that utilizing a unified method and further downsampling the negative instances can significantly enhance the performance of the prediction model on unbalanced binary and multi-classes. dealing with unbalance classes is also studied well in the sentiment classification task. @xcite introduced an active learning method that overcomes the problem of data class unbalance by choosing the significant sample of minority class for manual annotation and majority class for automatic annotation to lower the amount of human annotation required. furthermore, @xcite examined techniques to overcome the problem of dealing with high-class imbalance in classifying a collection of song lyrics. they employed neural network models including a multi-layer perceptron and a doc2vec model in their experiments where the finding was that undersampling the majority class can be a reasonable approach to remove the data sparsity and further improve the classification performance. @xcite also explored the problem of high data imbalance using cross-entropy criteria as well as standard performance metrics. they proposed a loss function called dice loss that assigns equal importance to the false negatives and the false positives. in computer vision, @xcite developed an automated feedback loop method to identify and classify wildlife species from unmanned aerial systems imagery, for training cnns to overcome the unbalanced class issue. on their expert imagery dataset, the error rate decreased substantially from 0.88 to 0.05. this work adapts this feedback loop strategy to the nlp problem of classifying technical events. ## technical event datasets in this work, we used a set of 7 logbook datasets from the aviation, automotive, and facility domains available at maintnet @xcite . maintnet is a collaborative open-source platform for predictive maintenance language resources featuring multiple technical logbook datasets and tools. these datasets include: 1) avi-main contains seven years of maintenance logbook reports collected by the university of north dakota aviation program on aircraft maintenance that were reported by the mechanic or pilot. 2) avi-acc contains four years of aviation accident and reported damages. 3) avi-safe contains eleven years of aviation safety and incident reports. accidents were caused by foreign objects/birds during the flights which led to safety inspection and maintenance, where safety crews indicated the damage (safety) level for further analysis. 4) auto-main is a single year report with maintenance records for cars. 5) auto-acc contains twelve years of car accidents and crash reports describing the related car maintenance issue and property damaged in the accident. 6) auto-safe contains four years of noted hazards and incidents on the roadway from the driver. 7) faci-main contains six years of logbook reports collected for building maintenance. these technical logbooks include short, compact, and descriptive domain-specific english texts single instances usually contain between 2 and 20 tokens on average including abbreviations and domain-specific words. an example instance from an instance in the logbook can be formed as a complete description of the technical event (such as a safety or maintenance inspection) like: #2 & #4 cyl rocker cover gsk are leaking, or it might contain an incomplete description by solely referring to the damaged part/section of machinery (hyd cap chck eng light on) using few domain words. in either form of the problem description, the given annotation (label) is at the issue type-level, e.g., baffle damage. further characteristics of these log entries include compound words (antifreeze, engine-holder, driftangle, dashboard). many of these words (e.g., a compound word: dashboard) essentially represent the items, or domain-specific parts used in the descriptions. additionally, function words (e.g., prepositions) are important and removing them could alter the meaning of the entry. the logbook datasets also have both the following shared and distinct characteristics: shared characteristics: each instance contains a descriptive observation of the issue and/or the suggested action that should be taken (eng inspection panel missing screw). each instance also refers to maintaining a single event, which means the recognized problem applies to the only single-issue type. as an example, the instance cyl #1 baff cracked at screw support & forward baff below #1 includes a combination of sequences that refers to the location and/or specific part of the machinery. ## results furthermore, we conducted the mann-whitney u-test of statistical significance by using the f1 scores of each of the 20 repeated experiments of the classification models, using the baseline and the feedback loop approach as the two different populations. the outcomes are shown in ## discussion to investigate the optimal strategies for dealing with these imbalanced technical datasets, we studied various methods on how to process the data, extract features, and classify the type of event. re- down over random feedback dataset model baseline (%) sampling (%) sampling (%) ds loop (%) loop (%) pre rec f1 pre rec f1 pre rec f1 pre rec f1 pre rec f1 avi-main the scarce data available in a dataset such as auto-main is certainly an issue for deep learning methods. examining the accuracy improvement by using the proposed feedback loop strategy, requires incorporating more instances to the event classes. similar to any supervised learning models, we noticed some limitations that could be addressed in future work. as shown in the previous sections (such as the pre-trained bert model provided a reasonable classification performance compared to the other deep learning models, however as bert is pre-trained on standard language, the performance when applying to logbook data was not optimal. training or fine-tunning bert to technical logbook data is likely to improve performance as observed in the legal and scientific domains @xcite . as training or finetuning bert requires large amounts of data, a limitation for fine-tuning a domain-specific bert is the amount of logbook data available. ## conclusion and future work this work focused on predictive maintenance and technical event/issue classification, with a special focus on addressing class imbalance. we acquired seven logbook datasets from three technical domains containing short instances with non-standard grammar and spelling, and many abbreviations. to address rq1, we evaluated multiple strategies to address the extreme class imbalance in these datasets and we showed that the feedback loop strategy performs best, almost entirely providing the best results for the 7 different datasets and 4 different models investigated. to address rq2, we empirically compared different classification algorithms (dnn, lstm, cnn, and pre-tuned bert). results show that the cnn model outperforms the in future work, we would like to fine-tune bert using logbook data, as described in section 6, and extend this work to datasets in other languages. the biggest challenge for these two research directions is the limited availability of logbook datasets. furthermore, we are exploring various methods of domain adaptation and transfer learning on these datasets to further improve the performance of classification models.