SSNS heerin K avitha at S em E val-2023 Task 7: Semantic Rule Based Label Prediction Using TF - IDF and BM 25 Techniques (2023)

Abstract: The advancement in the healthcare sector assures improved diagnosis and supports appropriate decision making in medical domain. The medical domain data can be either radiology images or clinical data. The clinical data plays a major role in the healthcare sector by preventing and treating the health problem based on the evidence learned from the trials. This paper is related to multi-evidence natural language inference for clinical trial data analysis and its solution for the given subtasks (SemEval 2023 Task 7 - NLI4CT). In subtask 1 of NLI4CT, the inference relationship (entailment or contradiction) between the Clinical Trial Reports (CTRs) statement pairs with respect to the Clinical Trial Data (CTD) statement are determined. In subtask 2 of NLI4CT, predicted label (inference relationship) are defined and justified using set of supporting facts extracted from the premises. The objective of this work is to derive the conclusion from premises (CTRs statement pairs) and extracting the supporting premises using proposed Semantic Rule based Clinical Data Analysis (SRCDA) approach. From the results, the proposed model attained an highest F1-score of 0.667 and 0.716 for subtasks 1 and 2 respectively. The novelty of this proposed approach includes, creation of External Knowledge Base (EKB) along with its suitable semantic rules based on the input statements.

## introduction the natural language processing (nlp) is one of the fastest growing sub-domain of artificial intelligence (ai) for medical domain @xcite . the contributions of nlp for medical domain are transcribing medical documents, clinical entity resolver and end-to-end drug discovery. to improvise the existing applications, research forums are conducting tasks for current requirements. one among them is semeval 2023 and the task is multi-evidence natural language inference (nli) for clinical trial data @xcite and anal-ysis. the motivation of this task is to help the clinical practitioner to stay updated on all current literature by providing an opportunity to support the large-scale interpretation and retrieval of medical evidence. to achieve this, nli is applied on the clinical trial dataset (breast cancer clinical trial reports (ctrs) and clinical trial data (ctd)) to infer the medical evidence based on the relationship between premises and hypothesis. the clinical trial data analysis is divided into two subtasks namely, textual entailment and evidence retrieval. in textual entailment (subtask 1), the hypothesis is derived from ctrs premises of four sections namely, intervention, eligibility, results and adverse events with respect to the ctd statement. under each section there are three to eighty-two premises. in evidence retrieval (subtask 2), the anticipated premises which play a significant role in deriving the conclusion are retrieved. the remaining part spans across the following subsections. the background for the textual entailment and evidence retrieval subtasks are discussed in section 2. in section 3, the system overview of the proposed work is explained. the experimental setup and implementation are explained in section 4. a brief summary about the results, quantitative analysis and error analysis are given in section 5 and conclusion is summarized at the end. ## background the clinical trial dataset is common to both textual entailment and evidence retrieval subtasks of semeval 2023 (task 7). the dataset comprises clinical trial data (ctd) (training set -1700 samples, development set -200 samples, test set -500 samples ) and clinical trial reports (ctrs) (ct json -1000 samples). the training and development set of ctd consists of type (single or comparison), section_id (intervention, eligibility, results or adverse events), pri-mary_id (map to ctrs id), secondary_id (map to ctrs id), statement, label (entailment or contradiction), primary_evidence_index and sec-ondary_evidence_index. the test set consists of type, section_id, primary_id, secondary_id and statement. if type is single, then secondary_id and secondary_evidence_index are inappropriate. the dataset structure can be better understood through the following example. let the primary_id, secondary_id and section_id of ctd be "nct00856492", "nct00009945" and "eligibility", then the respective ctrs having the ctrs id as "nct00856492" and "nct00009945" are selected from ct json folder. within the selected files, the statements under the eligibility section are considered for the proposed work. from the dataset analysis, the following keypoints are observed, (i). every statement in the ctrs section need not be required to derive the conclusion. (ii). in case of comparison type, statements from both primary section and secondary section contribute equally in achieving the conclusion. (iii). for better result, @xcite stated that semantic level interpretation is required (for example, the clinical trial statement "most participants in the secondary trial and the primary trial did not suffer from enterocolitis" is equivalent to the statement "enterocolitis haemorrhagic 1/167 (0.60%)" and "enterocolitis haemorrhagic 0/167 (0.00%)" in the adverse event section (iv). external knowledge base @xcite which maintains synonyms is required to develop better models. for example, the statement "only patients with her2 positive breast carcinoma are eligible for the primary trial" is equivalent to "her-2 overexpressing breast cancer" in the eligibility section because "carcinoma" is synonymous to "cancer". after the analysis of the given clinical trial dataset, the semantic rule based clinical data analysis (srcda) approach is designed and explained in the following section. ## system overview the proposed srcda system for textual entailment and evidence retrieval is divided into three modules namely, srcda (common to subtasks 1 and 2), srcda -textual entailment and srcdaevidence retrieval. the srcda first module consists of five submodules and each of the second and third module consists of two task specific submodules which is shown in figure 1 for better understanding. the srcda module consists of prepro-cessing, external knowledge base (ekb) creation, semantic rules formulation, feature vector generation by term frequency inverse document frequency (tf-idf) technique @xcite and distance similarity metric @xcite by radial basis function kernel (rbf-kernel) modules. the two submodules specific to srcda -textual entailment are, clinical trial reports (ctrs) section statements filtration and label prediction. the two submodules specific to the srcda -evidence retrieval are ctrs section statements ranking by best match 25 (bm25) techniques @xcite and, primary and secondary indices retrieval. these modules are discussed in detail in the following subsections and the process is explained in algorithm 1. the proposed srcda model uses ekb, semantic rules, tf-idf techniques, rbf-kernal distance metrics, bm25 technique because of the following reasons. they are: (i). the ekb collects the knowledge from multiple external sources for model creation; (ii). the semantic rules are defied to understand and analyse the statements by considering the context; (iii). the tf-idf technique is used since it gives weights based on the uniqueness and relevance rather than frequency and hence more accurate representation among the overall polarity can be achieved; (iv). the rbf-kernel is selected since it computes the similarity score between two statements based on their distance in the higher dimensional space; (v). the bm25 technique, supports clinical trial report (ctr) statements length and term frequency saturation to rank the statements in the primary and/or secondary indices of ctr statements. ## experimental setup the system requirement for the proposed approach includes, (i). hardware requirement -intel i5 processor with nvidia graphics card, 4800m at 4.3ghz clock speed, 16gb ram, graphical processing unit and 2tb disk space (ii). software requirement -ubuntu 20.04 operating system, python 3.7 package with required libraries like tensorflow, torch, sklearn, json, numpy, nltk, pickle, pandas, rank_bm25, etc., the clinical trial data (ctd) in clinical trial dataset is divided into training (training and development) set and test set. both training and test set uses files in the ct json folder for utilizing statements in clinical trial reports (ctrs) section. the proposed model is developed as given in the system design for two subtasks for the training set. in the testing phase, test set is given as input to the proposed model to predict label (subtask 1) and, retrieve primary and secondary section indices (subtask 2). the effectiveness of the proposed model are evaluated by comparing the predicted output with the actual answer using performance metrics. the different performance metrics used for analysing the results are f1-score, recall and precision. in this, the f1-score measures the model accuracy with respect to the dataset, recall measures the number of correctly identified samples by the model to that of labels, and precision identifies the frequency in which a model is correct while predicting the positive class. ## results the quantitative analysis of the subtasks are, (i). some statements in the section are indirectly deriving the conclusion and it is resolved by incorporating semantic based rules like negation equivalence, double negation, deductive reasoning and condition based equivalence rules (ii). adopting tf-idf technique retrieves the suitable clinical trial report (ctr) statements efficiently among the ctr section statements (usually each section has 4 to 32 statements) (iii). the rbf-kernel distance similarity performs linear manipulation to map points to higher dimensional space and hence it is better than other distance similarity metrics. (iv). most of the hypothesis are entailment ("true") and hence computing primary and secondary index is feasible. the error analysis and the solution for the subtasks are, (i). the pairwise distance similarity (run 4 of subtask 1) measures distance between two points or matrices and hence it degrades the overall performance. incorporating the rbf-kernel distance similarity from run 10 improves the performance of the system (ii). in run 3 of subtask 1, the non-optimal critical score reduces nearly 30% of the f1-score as compared to run 1. this is addressed by fixing the suitable critical score in run 14 (iii). the ekb is used from 10th run and hence suitable critical score value changes and performance also increased. (iv). run 38 and run 1 (post competition) of subtask 2 shows that ekb and semantic based rules improved the overall performance of the system. ## conclusion and future work in this paper, the semantic rule based clinical data analysis (srcda) approach is developed for multi-evidence natural language inference (nli) for clinical trial data task. in srcda approach, negation equivalence, double negation, deductive reasoning and condition based equivalence rules were used along with other textual entailment and evidence retrieval techniques. hence, the f1-score is increased by 17.5% and 23.7% respectively for subtask 1 and subtask 2 respectively. but the overall performance of the system lacks by 18.9% and 14.0% as compared with the leading team in terms of f1-score. in the future work, the performance of the model can be improved by incorporating more semantic based rules like transitive and abductive rules, developing an improved external knowledge base (ekb) by including more synonyms words from different open sources, and choosing suitable critical score depending upon the contribution of premises towards deriving the conclusion are the ideas (plans) to be executed for improvement. . by incorporating the above discussed ideas, the researchers can develop an improved system which can be used in other domains like question answering system (chatbot), text summarization, dialogue to text conversion, etc.