Scaling Systematic Literature Reviews with Machine Learning Pipelines (2020)

Abstract: Systematic reviews, which entail the extraction of data from large numbers of scientific documents, are an ideal avenue for the application of machine learning. They are vital to many fields of science and philanthropy, but are very time-consuming and require experts. Yet the three main stages of a systematic review are easily done automatically: searching for documents can be done via APIs and scrapers, selection of relevant documents can be done via binary classification, and extraction of data can be done via sequence-labelling classification. Despite the promise of automation for this field, little research exists that examines the various ways to automate each of these tasks. We construct a pipeline that automates each of these aspects, and experiment with many human-time vs. system quality trade-offs. We test the ability of classifiers to work well on small amounts of data and to generalise to data from countries not represented in the training data. We test different types of data extraction with varying difficulty in annotation, and five different neural architectures to do the extraction. We find that we can get surprising accuracy and generalisability of the whole pipeline system with only 2 weeks of human-expert annotation, which is only 15% of the time it takes to do the whole review manually and can be repeated and extended to new data with no additional effort.

## introduction systematic reviews are part of the field of evidencebased analysis, and are a methodology for conducting literature surveys, where the focus is on comprehensively summarising and synthesising existing research for the purpose of answering research questions @xcite . the aim of this process is to be very broad coverage to avoid unknown bias creeping into results via the alternative of cherry-picking scientific results @xcite . conducting systematic reviews requires trained researchers with domain knowledge. the stages of the process are time-consuming, but vary in how much physical and mental labour they require @xcite . as a result, systematic reviews suffer from three primary challenges @xcite : 1. they are very expensive, as they require many months of expert human labour; 2. they easily become out of date, for the same reason; so though systematic reviews have been shown to be very effective and less prone to human biases @xcite , these issues often prove prohibitive. however, these challenges are well suited to machine learning solutions, and there has recently been an increase in interest in applying nlp to this process @xcite . in this paper, we investigate the feasibility of implementing the multi-stage human process of a systematic review as a machine learning pipeline. we construct a systematic review pipeline which aims to assist researchers and organisations focusing on livestock health in various african countries who previously performed reviews manually (via a process visualised in figure 1 ). the pipeline begins with scraping for articles, then classifies them into whether or not to include in the review, then identifies data to extract and outputs a spreadsheet. we discuss the technical options we evaluated at each steps. pipeline components are evaluated with intrinsic metrics as well as more pragmatic, extrinsic, considerations such as time and effort saved. while previous work exists surveying the applicability of various machine learning methods and toolkits to the systematic review process (section 6) and a few apply them, there are no extant studies that implement a full system and analyse the trade-offs between different methods of training data creation, different annotation schemas, human expert hours needed to build a system, and final accuracy. we experiment with all of these factors, as well as with a few different architectures, with the aim of informing the planning and implementation of systematic review automation more broadly. to further this goal, we particularly experiment with low resource scenarios and with generalisability. we investigate different thresholds for training data for the document classifier and different annotation schemas for the data extraction. we additionally test the ability of the system to generalise to documents from new countries. key research questions are as follows: extraction which techniques are best for identifying and extracting the desired information? data requirements how much labelled training data is needed? can existing resources be leveraged? re-usability how generalisable is a pipeline to new diseases and countries? performance what is the trade-off between pipeline accuracy and human time savings? architecture & pre-training how important is model architecture as applied to extraction tasks? how important is embedding pre-training, and how important is pre-training on scientific literature vs. general content (domain match)? we find that surprisingly little training data (and few human hours) are necessary to get an accurate document classifier, and that it generalises well to unseen african countries (section 5), which enables systematic reviews to be expanded to new areas with essentially constant time. in our text extraction experiments, we find that both sentence and phrase level extraction models can each play a role in such a pipeline, but that phrase extraction, which has not previously been done for this task, performed better than expected both with baseline cnn models @xcite and with bert-based transformers @xcite , with transformers based on scientific pre-training @xcite performing best. we demonstrate how the creation of labelled training data can be sped up through annotation tools, and that consideration should be given to the balance of training examples present within this data, since doing so may require less data overall while still maintaining good performance. furthermore, besides automatic information extraction, much labour in constructing systematic reviews can be saved through simply automating the process of searching and downloading documents. we empirically demonstrate that most of the three month pipeline of a systematic review can be automated to require very little human intervention, with acceptable accuracy of results. we release our code, annotation schema, and labelled data to assist in the expansion of systematic reviews via automation. while we demonstrate this system on one domain, the framework is domain independent and could be applied to other kinds of systematic reviews. new training data and annotation schemes would be necessary to switch to medical or other domains, but our findings on time saving processes for annotation would apply, and confidence thresholds that we implement are adjustable to customise to different levels of accuracy to human time tradeoffs that are appropriate to different fields. our exploration into necessary amounts of training data for accuracy and generalisability are broadly applicable. ## background and motivation as a case study, we work with the supporting evidence based interventions team at the royal (dick) school of veterinary studies at the university of edinburgh, focusing on putting data and evidence at the centre of livestock decision-making in low and middle-income countries, predominantly in africa. in these countries, livestock offer a path out of poverty for millions of smallholders, as well as providing vital nutrition for families and communities. while the veterinary technology and techniques required to improve livestock outcomes al-ready exist (and are readily available to large scale commercial concerns worldwide), there is a lack of reliable information on animal health and productivity in these countries, at this scale. this data is needed not only in order to best target interventions, but to select the most efficient intervention in any particular context. there is very little data in this area for these countries, and it is often out of date. one proxy for direct measurement of livestock health in all herds in a country is the evidence found in veterinary science research publications, which have conducted prevalence studies. individually, these studies give an indication of the prevalence of a specific disease in a specific region of a country at a specific time, affecting a specific breed of animal. but collectively, they give a much broader understanding of livestock health. four strands of data are of key importance: general livestock statistics (herd size and characteristics), health (mortality and disease), production (yields and growth rates) and economics (breeding costs, feeding costs, produce sale values). here, we focus on health, specifically the prevalence of a wide range of diseases (e.g. brucellosis, foot and mouth disease) that effect ruminants (sheep, goats and cattle), with a focus on countries such as nigeria, ethiopia and tanzania. however, collecting and summarising the findings of these studies is time-consuming manual work. for example, searching databases such as google scholar, pubmed and web of science for prevalence studies, conducted between 2010 and 2018 on 28 diseases in small ruminants in ethiopia, returned many thousands of results. of these, 403 papers were considered relevant and information was extracted by experienced veterinarian researchers. the completely manual process, outlined in figure 1 , produced high quality data but took approximately three months. the target information consists of dates (the start/end of the study), numerical data (sample size, direct/percentage prevalence numbers, animal age) and small lexical terms. these terms can be veterinary (diagnostic tests used, production systems, study design, statistical analysis performed, species) or geographic (region, ecosystem). 2 since this summarisation process isn't abstractive, readers find it easy, if time-consuming, to identify the relevant information in research papers. the current project aimed to automate as much of the process as possible, to allow it to scale to a wider range of diseases (approximately 50) and countries. all but the initial component of figure 1 can be entirely automated -humans are still required to define search terms. ## pipeline details & experimental setup in the following we detail each stage of the constructed pipeline, how it compares to the human version and the human time saved. in the succeeding section 4 we detail metrics and evaluation for our targeted experiments with the classification and extraction pipeline components. ## methodology we performed detailed evaluation of the different classification and extraction components. ## related work the application of nlp to systematic reviews is relatively new, but has been recently receiving more attention. there is a growing body of work that assesses the potential for automation in systematic reviews, but little that builds systems for the purpose and tests them empirically. @xcite review available tools that can be used to automate each element of the systematic review pipeline. marshall et al. (2020) further review opportunities for semi-automation and assess opportunities and risks. @xcite ; @xcite conduct systematic reviews of automation for systematic reviews. @xcite analyse the systematic review pipeline to find ways that human-machine collaboration can be applied and improve the speed. @xcite create a pdf viewer that humans can use to make the systematic review process easier and faster, by training a cnn to assess risk of bias in a document (an important part of evidence-based analysis in the medical domain, though not for our particular task) and identifies and displays sentences to the user that contain a subset of the information necessary for a systematic review. @xcite create an extraction system that identifies sentences and then post-processes them to extract data, but operate only on structured html & xml. @xcite apply fine-tuned bert-based transformers to the task of to sentence classification for semi-automated systematic review. @xcite build a pdf retrieval system for systematic reviews for psychology and use a random forest classifier to identify sentences for extraction. as far as we are aware, no other work builds a phrase-based system, tests data volume and generalisability, or applies a diverse set of modern architectures to the task. ## conclusion & future work we investigated the application of automation to all stages of the systematic review pipeline for our veterinary research case study. we found that with two weeks ( 80 hours) of human expert annotation we can automate a systematic review that previously took 3 months, and still maintain high levels of accuracy. our classification system generalises well, enabling it to be applied to new countries for additional systematic reviews with no additional human annotation cost. sentence-based and phasebased data extraction both perform well, and the creation of phrase-based training data can still fit within a small amount of human annotation hours and avoids the need for extensive post-processing. fine-tuned bert-based transformers perform best at data extraction, with bert pre-trained on scien-tific data giving the largest boost in performance, though a baseline cnn still performs surprisingly well. in future work, we plan to test generalisability cross-lingually, expand the generalisability tests to extraction as well as classification, and study the performance improvements of continuous training of classifiers on human corrections of low-confidence output.