The Influence of Background Data Size on the Performance of a Score-based Likelihood Ratio System: A Case of Forensic Text Comparison (2020)

Abstract: This study investigates the robustness and stability of a likelihood ratio–based (LR-based) forensic text comparison (FTC) system against the size of background population data. Focus is centred on a score-based approach for estimating authorship LRs. Each document is represented with a bag-of-words model, and the Cosine distance is used as the score-generating function. A set of population data that differed in the number of scores was synthesised 20 times using the Monte-Carol simulation technique. The FTC system’s performance with different population sizes was evaluated by a gradient metric of the log–LR cost (Cllr). The experimental results revealed two outcomes: 1) that the score-based approach is rather robust against a small population size—in that, with the scores obtained from the 40 60 authors in the database, the stability and the performance of the system become fairly comparable to the system with a maximum number of authors (720); and 2) that poor performance in terms of Cllr, which occurred because of limited background population data, is largely due to poor calibration. The results also indicated that the score-based approach is more robust against data scarcity than the feature-based approach; however, this finding obliges further study.

## introduction: the likelihood ratio framework and forensic text comparison the likelihood ratio (lr) conceptual framework has been studied for its effect on various types of forensic evidence; it was mathematically shown that, with some very reasonable assumptions, the lr is the only way of assessing the uncertainty inherited in evidential evaluation @xcite @xcite . it is becoming recognised as the logical and legally correct framework for both analysing forensic evidence and presenting it in court @xcite @xcite @xcite . yet, some argue that the lr is one possible tool for communicating to decision makers @xcite . although forensic text comparison (ftc) currently lags behind other forensic sciences, some studies have demonstrated that linguistic text evidence can be properly analysed using the lr framework @xcite @xcite . in the lr framework, instead of assessing the probabilities of two competing hypotheses given the evidence, the probabilities of observing the evidence (e) are assessed given the hypotheses: the prosecution hypothesis (hp) against the defence hypothesis (hd) @xcite @xcite . therefore, the lr can be defined as in equation (1). ## experiment design two sets of experiments were conducted, with the first set aiming to identify the conditions under which the ftc system optimally performs (see section 3.1). in the second set, with the best-performing conditions set, the ftc system's performance is assessed by altering the data number for training the score-to-lr conversion model (see section 3.2). the database, pre-processing of data, logistic-regression calibration and assessment metrics are also discussed in this section. ## results and discussions the boxplots presented in figure 4 reveal the degree of fluctuations in the cllr values of the 20 simulations; they also indicate how the cllr values converge as the number of authors increases. regardless of the word length, the ftc system's performance substantially fluctuates when the background database only comprises the text samples from 5~10 authors; that is, the performance is not stable. however, this instability quickly recovers if the text samples are collected from 20 or more authors. this is a positive finding in terms of ftc's practical application, as forensic scientists cannot afford the time and money required to collect a large number of data that are relevant to each case if they cannot find an already-existing dataset that is suitable to the case. sa scores da scores 750 weibull weibull 1500 weibull normal 2250 weibull normal it is evident from figure 4 (black circles) that the system's overall performance improves exponentially from @xmath0, resulting in the outcome in which the performance with @xmath1. to further investigate the reasons underlying the fluctuations in performance (especially with the small number of n), the cllr min and cllr cal values (discrimination loss and calibration loss, respectively) are plotted separately in figures 5 and 6 , respectively. they are presented in the same manner as figure 4 . as can be observed in figure 5 , being apart from the word length of 750, the system's discriminability is highly stable, even with small ns. specifically, regarding the word length of 2,250, figure 5c reveals that the cllr min values are constant and far less fluctuated, as they are not affected by the number of authors in the background database. that is, in terms of discrimination performance, when many words (e.g., 1,500 and 2,550 words) are available, the system is robust and stable against a small background population size. in contrast, figure 6 indicates that the cllr cal values exhibit a highly similar trend to that of the cllr values that are plotted in figure 4 -in that, a great variability in the cllr cal values is observed when the number of authors is small (e.g., @xmath2; however, this variability begins converging rapidly with more authors. this signifies that the cllr cal values also demonstrate a quick recovery with more authors. the observations drawn from figures 5 and 6 reveal that the poor performance associated with a small number of authors (@xmath3, as indicated by the cllr values from figure 4 , is not due to the system's poor discriminability, but due to poor calibration. following this interpretation, logistic-regression calibration was applied to all lrs, in which a gain in overall performance was expected. the cllr values of the calibrated lrs are again plotted as boxplots in figure 7 . it is apparent from figure 7 that the system's performance has noticeably improved in both stability and accuracy; the degree of fluctuations in the cllr values is lessened and the mean cllr values are lower, even with small ns. ishihara (2016) previously investigated how background population size affected the performance of an lr-based ftc system. in the experiments, the lrs were estimated using the multivariate kernel density (mvkd) lr formula @xcite , with two to eight stylometric features. texts collected from 140 authors were used to extract necessary statistical information for a monte-carlo simulation, for which a mixture gaussian model was used. the mvkd is a type of feature-based approach for estimating lrs. the population size was incremented by 10 from 10 authors to 140 authors. although a direct comparison between the current study and ishihara's (2016) study cannot be validly made, some noticeable differences can still be highlighted. the number of features (2~8) used in ishihara's study was far smaller than that of the current study (260), and ishihara reported a great improvement in cllr (from 10 to 50~60 authors), after which a small but continuous improvement could be observed with more authors. he also reported a relatively high variability in cllr, even with a large number of authors (e.g., 140). in light of these comparative observations, the ftc system's performance appears to reach its optimum with a smaller population size for the scorebased approach rather than for the feature-based approach. further, the fluctuation in performance also begins converging with a lesser number of background data for the score-based approach than for the feature-based approach. the relative robustness of the score-based approach that the current study revealed for linguistic text evidence aligns with the findings in previous studies regarding other types of evidence @xcite . however, the difference in performance between the score-and feature-based approaches must be further investigated under mutually comparable conditions. based on figure 7 , it can be concluded that logistic-regression calibration leads to an improvement in terms of the system's stability and validity. for training the logistic-regression weights, the development database that comprised the texts from 720 authors was employed. it is evident that the calibration performance also mainly relies on the quantity of the data in the development database. the positive outcome after applying the calibration is likely attributable to the amount of data in the development database. therefore, it is pertinent to analyse how the development database's size influences the ftc system's performance, as the application of calibration appears to be essential when the background database is substantially small in number (e.g., 5~10 authors). ## conclusion and further study the robustness and stability of a score-based lr ftc system with a bag-of-words model were investigated with different numbers of background population data, which were synthesised by a monte-carol simulation. the experiments' results revealed that the score-based ftc system is fairly robust and stable in performance against the limited number of background population data. for example, with 40~60 authors, the performance is both nearly compatible and as stable as with 720 authors. this is a beneficial finding for ftc practitioners. additionally, the instability and suboptimal performance observed in terms of cllr with a small number of data (e.g., 5~20 authors) were mainly attributed to poor calibration (i.e., the derived lrs were not calibrated) rather than to the poor discriminability potential. a comparison with the outcomes of previous studies indicates that the score-based approach may be more robust against a limited number of background population data than a feature-based approach; however, this point warrants further study.