Usefulness of an artificial neural network to assess anaphylaxis severity
Authors: Magdalena Kraft, Johannes Gebauer, Sabine Dölle‐Bierke, Margitta Worm
Journal: Allergy (2020)

To the Editor, Anaphylaxis is an acute systemic allergic reaction, the clinical course of which may vary between some mild symptoms and life-threatening cardiovascular or respiratory involvement. The proper assessment of the reaction severity is important for patients, physicians, and researchers. Thus, multiple grading systems based on the most severe reaction symptom(s) have been developed.1-3 However, these grading systems bear several limitations: First, many of them were developed for a certain age group, a given elicitor of reaction, and/or clinical setting (eg, food challenges in children) what limits their applicability and comparability.4, 5 Second, they use a limited range of categories (eg, mild, moderate, and severe) to define a naturally continuous and not categorical phenomenon of reaction severity. For these reasons, alternative approaches to assess anaphylaxis severity are urgently needed. Numeric rating scale (NRS) is a commonly used one-dimensional tool to evaluate pain or other phenomena in clinical research.6 Thus, we assumed that NRS may also be a practical tool to assess anaphylaxis severity. To examine that, we used the data from the European Anaphylaxis Registry. Subjectivity and inter-rater variability are major problems while using NRS. The European Anaphylaxis Registry is a real-life database of anaphylactic reactions collected from more than hundred tertiary allergy centers from eleven countries.7 The data, including symptoms of a reaction, are captured in a standardized manner by trained health professionals. However, due to the variety of this data set (different elicitors, age groups, and country specific differences), we expect a high inter-rater variability of severity assessment at the level of participating centers. Thus, we decided to evaluate the cases from the registry using NRS centrally by one rater. An artificial neural network (ANN) was employed to enable the evaluation of 9719 reactions in this analysis (Figure S1). A total of 2059 anaphylaxis cases from the registry were evaluated by the human rater. We included 22 distinct symptoms (Table S1) in this evaluation; patient's age and elicitor of anaphylaxis were also considered. Every reaction was rated using NRS between 1 for mild reactions and 10 for near-fatal reactions. Afterward, ANN was trained and used to rate the remaining 7660 cases. Artificial neural network machine learning model was implemented with Java using the "Deep Learning for Java" library (https://deeplearning4j.org/). The network was constructed as a FNN (feed-forward neural network) with two dense hidden layers, each containing 33 neurons (Figure S2). The input layer size matched the number of variables resulting in 24 neurons (22 symptoms, elicitor, andage). The output layer consisted of 10 neurons, each representing a probability for the designated NRS score. This behavior was achieved by using softmax activation function. The number of neurons and hidden layers was determined by trial and error that achieved the best results. The source code is available on GitHub (https://github.com/jogeb/ann-anaphylaxis). During the ANN training, 75% of the 2059 manually evaluated cases were used to train and 25% to verify the quality of training. The training has been carried out over 175 000 epochs, starting with a learning rate of 0.6 which was decreased every 25 000 epochs by 0.05 which results in a learning rate of 0.25 for the last 25 000 epochs. The accuracy rate of ANN was 90.9% for complete overlay of values (exactly the same value of ANN as the human rater) and 98.7% if one-point deviation was allowed (Tables 1, S2 andS3 1). Thus, we assumed that ANN is able to learn an anaphylaxis severity assessment on NRS from a human rater and reproduce it. In the next step, we trained the ANN (with the same technical parameters as described above) with all 2059 manually evaluated cases and used it to determine NRS values for the remaining 7660 cases in our data set. The distribution of values calculated by ANN was alike to the distribution of the human rater values (Bhattacharyya coefficient = 0.996, Figure S3). To validate NRS, Spearman's rank correlation coefficients between NRS and other established grading systems were calculated. Spearman's rank correlation coefficients between NRS and grading systems described by Brown,1 Sampson,2 and Müller3 were 0.74, 0.69, and 0.68, respectively (Table 2, Figure S4). Thus, we propose that artificial intelligence (AI) can be used to reliably score anaphylaxis severity on a 1 to 10 scale, comparable to other established grading systems. Artificial intelligence has multiple potential applications in medicine. However, its use in allergology is still very limited. We were able to identify only very few publications, where AI was used for investigation of anaphylaxis.8, 9 Here, we demonstrate that an ANN can be used as a tool to evaluate anaphylaxis severity, as a physician would do. This instrument can not only evaluate thousands of cases in the standardized way within a second, but it can also do so in the same reliable way for all future cases. Our work has several limitations. Apart from the typical limitations of registry data (eg, missing values due to the retrospective nature), the arbitrary selection of variables is critical. The data on the clinical setting and treatment were not taken into consideration, which may restrict the capacity of the rater to recognize the true course of a reaction and to adequately evaluate it. The added value of NRS to evaluate anaphylaxis severity might be a subject of discussion but it is, in our opinion, substantial. NRS offers a more differentiated assessment of severity than just categorizing reactions into mild, moderate, or severe. This may be of particular interest in research and clinical studies. Unlike visual analog scale (VAS), NRS is an ordinal and not a continuous, metric scale. However, such kind of a 1 to 10 scale can be used as a quasi-metric scale, for example, to calculate and compare means. The main limitation of our project is that NRS values (both manually evaluated by a human rater and calculated by ANN) represent the subjective judgment of one rater. Although this judgment is based on years of clinical experience and research work in the field of anaphylaxis, we cannot claim that it is universal or "right." However, this pilot study demonstrates that devising this kind of severity scale and applying it on an unlimited number of cases are possible. AI trained with cases, evaluated in consensus by several experts, could be applied in the future as an universal tool to evaluate anaphylaxis caused by different elicitors in different age groups and thereby improve quality of anaphylaxis research. We thank all patients, parents, and their children for their support in providing data. We thank all centers for patient counseling and data entry into the European Anaphylaxis Registry. The participating centers are listed on https://www.anaphylaxie.net/en/centres. Open access funding enabled and organized by Projekt DEAL. M. Worm served as a consultant and received speaker honoraria from Mylan Germany, ALK-Abelló, and Allergopharma, outside the submitted work. The rest of the authors declare that they have no relevant conflicts of interest. M Kraft performed data analysis and wrote the manuscript. J Gebauer performed data analysis. S Dölle-Bierke managed data acquisition, contributed to the interpretation of data, and revised the manuscript critically for important intellectual content. M Worm managed data acquisition, contributed to the interpretation of data, and revised the manuscript critically for important intellectual content. All authors approved the final version of the manuscript for publication. The Anaphylaxis Registry is supported by The Network for Online Registration of Anaphylaxis NORA e.v. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.