Feature-Based Forensic Text Comparison Using a P oisson Model for Likelihood Ratio Estimation (2020)

Abstract: Score- and feature-based methods are the two main ones for estimating a forensic likelihood ratio (LR) quantifying the strength of evidence. In this forensic text comparison (FTC) study, a score-based method using the Cosine distance is compared with a feature-based method built on a Poisson model with texts collected from 2,157 authors. Distance measures (e.g. Burrows’s Delta, Cosine distance) are a standard tool in authorship attribution studies. Thus, the implementation of a score-based method using a distance measure is naturally the first step for estimating LRs for textual evidence. However, textual data often violates the statistical assumptions underlying distance-based models. Furthermore, such models only assess the similarity, not the typicality, of the objects (i.e. documents) under comparison. A Poisson model is theoretically more appropriate than distance-based measures for authorship attribution, but it has never been tested with linguistic text evidence within the LR framework. The log-LR cost (Cllr) was used to assess the performance of the two methods. This study demonstrates that: (1) the feature-based method outperforms the score-based method by a Cllr value of ca. 0.09 under the best-performing settings and; (2) the performance of the feature-based method can be further improved by feature selection.

## introduction the essential part of any source-detection task is to assess the similarity or difference between the objects or items under comparison. for this purpose, in stylometric studies too, various distance measures have been devised and tested, particularly in studies concerned with the authorship of text sources @xcite @xcite . burrows's delta @xcite is probably the most studied distance measure in stylometric studies, and its effectiveness and robustness have been demonstrated for a variety of texts from different genres and languages @xcite @xcite . since @xcite , several variants, including, for example, those based on euclidian distance, cosine similarity and mahalanobis distance have been proposed to better deal with the unique characteristics of linguistic texts, expecting to result in a better identification and discrimination performance @xcite @xcite . similarity-and distance-based measures make some assumptions about the distribution of the underlying data. for example, a laplace distribution is assumed by burrows's delta, which itself is based on manhattan distance, and a normal distribution by the euclidean and cosine distances. however, it is well known that stylometric features do not always conform to, for example, a normal distribution @xcite . moreover, a normal distribution is not theoretically appropriate for discrete count data (e.g. occurrences of function words) figure 1 shows the distributions of the counts of three words ('a', 'not' and 'they'), sampled from the database used in the current study. frequently-occurring words, such as 'a' (figure 1a ), tend to be normally distributed. however the distribution starts skewing positively for less-frequently-occurring words, such as 'not' (figure 1b ) and 'they' (figure 1c ). in order to fill this gap between the theoretical assumption arising from distance measures and the nature of textual data, a one-level poisson model is used in this study. in the 1990s, the success of dna analysis and some important united states court rulings, estab- ## likelihood ratio framework the lr framework has been proposed as a means of quantifying the weight of evidence for a variety of forensic evidence, including dna @xcite , voice @xcite , finger prints @xcite , handwriting @xcite , hair strands @xcite , mdma tablets @xcite , evaporated gasoline residual @xcite and earmarks @xcite . collected forensic items from known-(e.g. a suspect's known text samples) and questioned-source (e.g. text samples from the offender) can be evaluated by estimating the lr under two competing hypotheses. one specifying the prosecution (or the same-author) hypothesis (𝐻 𝑝 ) , and the other the defence (or the different-author) hypothesis (𝐻 𝑑 ). these are expressed as a ratio of condition probabilities as shown in equation 1 ). the relative strength of the evidence with respect to the competing hypotheses is reflected in the magnitude of the lr. the more the lr deviates from unity (@xmath0, the greater support for either the 𝐻 𝑝 (@xmath1. the lr is concerned with the probability of evidence, given the hypothesis (either prosecution or defence), which is in concordance with the role of an expert witness in court, leaving the trier-of-fact to be concerned with the probability of either hypothesis, given the evidence. ## experiments the two main approaches for estimating lrs, namely the score-and feature-based methods, will be implemented and their performance compared. after the database ( §3.1) and the pre-processing and modelling techniques ( §3.2) are introduced, the two methods are explained in §3.3 and §3.4, respectively, along with their pros and cons. fusion/calibration techniques and performance metrics are described in §3.5 and §3.6, respectively. ## results and discussion the cllr values are plotted as a function of the number of features, separately for the baseline model and the poisson model in figure 2 . the number of the features is incremented by 5 from @xmath2, and then by 20 from @xmath3. for example, @xmath4. whereas the scores, which are to be converted to the lrs, were measured based on the vector of the five most-frequent words for the score-based method. as can be observed from figure 2 , the performance of both models improves en masse as the n increases until a certain n, after which the performance remains relatively unchanged (or falls slightly). the baseline model's performance stays relatively stable for a higher number of n, while the performance of the poisson model begins to decline after 180 features. due to the deterioration with a large number of feature numbers, although the poisson model outperforms the baseline model overall, the baseline model does better with @xmath5. the best performance, however, was observed for the poisson with a lower number of features (@xmath6.26439; @xmath7.35682; @xmath8. the superior performance of the feature-based method (poisson model) relative to the score-based method (baseline model) conforms to the reports of previous studies on other types of evidence @xcite . as described earlier, the baseline and the poisson models exhibit different performance characteristics in terms of the number of features required for optimal cllr and the effect of increasing n. the performance of the baseline model stays relatively unchanged with more features, while the performance of the poisson model continuously declines with more features. in order to further investigate this performance difference, the cllr, cllr min and cllr cal values are plotted separately for the two models in figure 3 . for the baseline model, it can be seen from figure 3a that 1) the cllr cal values consistently remain close to 0, meaning the lrs are very well calibrated regardless of the number of features, and also that 2) the cllr min values display an almost identical trend as the cllr values, meaning that like the cllr values, the discriminability potential remains relatively constant even with an increase in the feature number after the best-performing point. in contrast, the three metrics plotted in figure 3b reveal some notably different characteristics of the poisson model. the cllr cal values stay low only until @xmath9, after which the cllr cal values start increasing at a constant rate with an increase in the feature number; that is, the lrs become less well calibrated as n increases beyond 140~160 features. unlike the calibration loss (and the baseline model), the discriminability potential, quantified by cllr min , continues to improve at a small but constant rate, even after @xmath10, where the best cllr was observed. thus, it is clear from figure 3 that the deterioration of the poisson model in performance after @xmath11. as explained in §3.5, logistic-regression fusion/calibration should theoretically yield well calibrated lrs. the poor calibration performance observed for the poisson model for large feature numbers may be due to the interaction between the dimensions of the lrs to be fused and the amount of the training data for the fusion/calibration weights. this seems to be a typical example of the phenomenon known as the 'curse of dimensionality' @xcite ), but further analysis is warranted. nevertheless, it is clear that the use of a poisson-based model, which theoretically better suits the distributional pattern of textual data and allows the rarity/typicality of evidence to be considered for lr estimation, can offer performance gains. ## feature selection for the poisson model, lrs were first estimated separately for each of the 400 feature words. the resulting lrs were fused by gradually increasing the number of lrs included in the fusion set. lrs were arranged according to word frequency in the experiments reported in §4. yet, the performance of a given feature (i.e. word) did not always correspond to the frequency of its occurrence. this is illustrated in thus, in this section, the words were first sorted according to their performance in terms of the cllr min values, and then the lrs were fused/calibrated based on the sorted words. the cllr values of the experiments are plotted in figure 4 including the results presented in figure 2 for comparison. it is clear from figure 4 that selecting the features according to their cllr min values contributes to an improvement in performance for all numbers of features. as a result, the cllr is lower (0.21664) with less features (@xmath12. this feature selection approach was only possible because the lrs are estimated separately for the each of the 400 different words. this is possibly an advantage for the poisson model. the magnitude of the lrs with the best-performing settings are shown on tippett plots, separately for the baseline model, the original poisson model, and the poisson model with cllr min -sorted features in figure 5 . tippet plots show the cumulative proportion of lrs from the sa comparisons (salrs), which are plotted rising from the left, as well as of the lrs of the da comparisons (dalrs), plotted rising from the right. for all tippett plots, the cumulative proportion of trails is plotted on the y-axis against the log10 lrs on the xaxis. the intersection of the two curves is the equal error rate (eer) which indicates the operating point at which the miss and false alarm rates are equal. as the low cllr cal values indicate, it can also be observed from figure 5 that the lrs are very well calibrated. however, comparing figure 5a and figure 5bc we see that the magnitude of the lrs are weaker overall in the baseline model compared to the two poisson models; the tippet lines are further from unity (log10 @xmath13. although the overall magnitude of lrs is greater for the poisson models, unlike the baseline model, they evince some very strong contrary-to-fact dalrs (which are indicated by arrows in figure 5 ). this is a concern, and the reason for this needs to be further investigated. ## conclusions and future studies a feature-based approach for estimating forensic lrs was implemented with a poisson model for the first time in lr-based ftc. the results of the experiments showed that the feature-based ftc system outperforms the score-based ftc system with the cosine distance. it has also been demonstrated that the performance of the feature-based system can be further improved by selecting the sets of lrs to be fused according to their cllr min values. it was observed that the discrimination loss in the feature-based ftc system reduces as the number of features increases, but becomes less well calibrated with a large number of features. it has been argued that this is a typical case of the 'curse of dimensionality' @xcite ), but further investigation is required. a simple one-level poisson lr model shows good performance. however, it has been reported that word counts are often modelled poorly by standard parametric models such as the binomial and poisson models, and some alternatives have been proposed, such as the negative binomial and the zero-inflated poisson @xcite . alternatively, a two-level poisson model might be implemented based if the prior distributions of 𝜆 is assumed @xcite . these alternatives should be tested to see if any improvements in performance are achievable. the set of features tested in the current study is only one type of many potential authorship attribution features (according to @xcite , over 1,000 different feature types have so far been proposed in the literature). while the purpose of the present study was to compare modelling approaches, rather than the relative performance of different feature types, an interesting future task would be to explore a richer feature set and the effect of different pre-processing techniques (e.g. stop word removal). the lrs derived using the score-based method were well-calibrated, and therefore logistic-regression calibration was not necessary). this was not the case for lrs using the feature-based method where logistic-regression fusion/calibration was required. this procedure necessitates an extra set of data, namely a development database, and is another shortcoming of the feature-based method applied in this study