Estimating the Strength of Authorship Evidence with a Deep-Learning-Based Approach (2022)

Abstract: This study is the first likelihood ratio (LR)-based forensic text comparison study in which each text is mapped onto an embedding vector using RoBERTa as the pre-trained model. The scores obtained with Cosine distance and probabilistic linear discriminant analysis (PLDA) were calibrated to LRs with logistic regression; the quality of the LRs was assessed by log LR cost (Cllr). Although the documents in the experiments were very short (maximum 100 words), the systems reached the Cllr values of 0.55595 and 0.71591 for the Cosine and PLDA systems, respectively. The effectiveness of deep-learning-based text representation is discussed by comparing the results of the current study to those of the previous studies of systems based on conventional feature engineering tested with longer documents.

## introduction in forensic science, the likelihood ratio (lr) framework has long been considered the logically and legally correct approach to interpreting the analysis of forensic evidence @xcite @xcite . the lr framework is standard in dna typing. the community of forensic text comparison (ftc), commonly known as forensic authorship verification, recently recognised the importance of this framework @xcite . despite the importance of the lr framework in forensic science, lr-based studies on textual evidence are still conspicuously rare @xcite @xcite . many studies claim to be forensic but treat the problem as a usual authorship verification problem. however, there are important differences between conventional and forensic authorship verification. conventional authorship verification aims to answer a verification problem. forensic authorship verification aims to assist the fact finder in concluding the case, not answering the problem. legally, giving an answer to a verification problem (even in a probabilistic term) equates to referring to the ultimate question of 'guilty vs. not guilty', which is only permitted for the fact finder. logically, forensic scientists without all evidential information of the case cannot estimate the probability of a hypothesis from incomplete evidence. thus, they cannot logically refer to the ultimate question. however, forensic scientists can logically and legally estimate the strength of evidence via lr @xcite @xcite . lr is given in equation (1). lr is the ratio of two conditional probabilities; one is the probability of evidence (𝐸) given the prosecution hypothesis (𝐻 𝑝 ) and the other is the probability of the same evidence given the defence hypothesis (𝐻 𝑑 ). the relative strength of the given evidence with respect to the competing hypotheses is reflected in the magnitude of the lr. the greater the lr value is than 1, the stronger support the evidence is considered to provide for the prosecution; the smaller the lr value is than 1, mutatis mutandis, for the defence hypothesis. it is very important to note that the lr is not a binary expression of truth. with an lr estimated as the strength of evidence, the fact finder's belief regarding the hypotheses (quantified as prior odds) is raised to the posterior odds through the bayesian theorem, as shown in equation (2). 𝑃(𝐻 𝑝 |𝐸) 𝑃(𝐻 𝑑 |𝐸) ⏟ posterior @xmath0 the posterior odds are equivalent to the fact finder's belief regarding the hypotheses given the evidence despite the success of deep learning in many natural language processing tasks, a conventional machine learning approach with traditional feature engineering remains effective in authorship verification, particularly for small datasets @xcite . nonetheless, deep-learning-based systems gradually started achieving better verification accuracy than conventional approaches, in particular with a large volume of data @xcite @xcite . despite of its clear presence, deep learning has no yet made inroads into the lr-based ftc. this preliminary study looks in the effectiveness of a deep-learning approach in lrbased ftc. ## https://github.com/lingjzhu/idiolect the text length did not exceed 100 tokens; i.e. @xmath1. ## results the experimental results for the 𝐶 𝑙𝑙𝑟 -based metrics are shown in ishihara (2021) conducted lr-based ftc experiments by measuring the cosine distance of documents modelled via word unigrams. the target documents were also product reviews for amazon. each document was approximately 4 kb in data (approximately 800 words in length)considerably longer than the current study's (maximum 100 words). ishihara reported a 𝐶 𝑙𝑙𝑟 of 0.70640 as the optimal result. ishihara's experiments were carried out with the test, reference and calibration datasets, each of which had 720 authors. despite the very short documents, the systems tested in this study achieved nearly the same level of performance (cosine: 𝐶 𝑙𝑙𝑟 = 0.55595; plda: 𝐶 𝑙𝑙𝑟 = 0.71591) as ishihara's (2021) system based on documents of approximately 800 words (𝐶 𝑙𝑙𝑟 = 0.70640). although the experiments are not directly comparable, the effectiveness of the deeplearning-based text representation for estimating lrs can be conjectured. ## conclusions in this study, the lrs were estimated by logistic regression calibrating the scores obtained through two systems: one based on cosine distance and the other on the plda model. the documents were mapped on embedding vectors using roberta as the pre-trained model, and the derived lrs were assessed with 𝐶 𝑙𝑙𝑟 . albeit the documents being very short, the systems reached the 𝐶 𝑙𝑙𝑟 values of 0.55595 and 0.71591, respectively for the cosine and plda systems. the effectiveness of the deeplearning-based text representation was discussed in comparison to the results of a previous study which was based on the system with conventional feature engineering and longer documents.