The Global Burden of Disease (GBD) study has estimated the leading causes of death and disability worldwide. [1] Health inequalities have been defined as differences in health which are avoidable, remediable, and considered unjust. [2] A key strategy to reduce death and morbidity is to conduct randomized controlled trials (RCTs) to determine the effects of interventions for improving health. [3] However, RCTs are expensive and time-consuming, and there is necessarily a limit on the number and size of these types of study that can be carried out. There are very . CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020. 10 .08.20209353 doi: medRxiv preprint many factors which could reasonably affect which health problems are researched, not limited to: the existence of highly effective treatments for certain conditions (where poor implementation or availability is the problem); national governments prioritising conditions which affect their own country's population; and the unpredictable nature of scientific breakthroughs in therapy.
Pharmaceutical companies are likely to develop drugs which have the potential to be sold at profit; this can lead to neglect of conditions (e.g. infectious diseases) which disproportionately affect low income countries. [4] Nonetheless, a rational approach to funding research -reduce health inequalities and maximise global benefit -would take account of the global disease burden.
Studies from the US and the UK have reported that disparities exist in national health research funding for certain conditions. Some conditions, such as diabetes and cancer, have attracted a comparatively generous amount of research funding in these countries relative to their national and global burden, whereas other conditions, such as stroke and respiratory diseases, have been relatively neglected. [5] [6] [7] [8] In a later study, Evans proportionate to disease burden in high-income countries, but not in non-high income countries. [10] To our knowledge, there has been no comprehensive, global study comparing published research (both in terms of trial and participant numbers) with the relative burden of each condition. We address this gap by evaluating the total volume of evidence (numbers of RCTs and trial participants) published for clinical conditions via an automated analysis of all research articles indexed in PubMed (http://pubmed.ncbi.nlm.nih.gov) from 1990 onwards. Our system uses natural language processing to automatically identify text within articles that describes the conditions being studied, and maps these to GBD-defined categories. This permits comparison between the global disease burden and the amount of published evidence that exists for each condition, and could help prioritise research funding to relatively understudied conditions with high burden. Uniquely, as our method is fully automatic, we are able to analyse articles at large . CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020.10.08.20209353 doi: medRxiv preprint scale (classification of 100's of thousands of documents), and with continuous, live updatingneither of which have been feasible so far with manual methods.
We developed a machine learning system which maintains a live database of annotated RCT reports, named Trialstreamer. We have described the computational methods and accuracy of the system components in detail elsewhere, [11] and summarize the key points relevant to the current study below.
We first downloaded the full PubMed database (annual baseline data, from 11th December 2018) and then subsequently monitored and added all newly published articles daily. This comprises >30 million articles. We automatically identify the subset of articles that describe RCTs using a machine learning system, which operates over the text of the title and abstract, and the Publication Type database index. [12] Briefly, we ensemble a Support Vector Machine (SVM) [13] and a Convolutional Neural Network (CNN). [14] These machine learning models are trained 1 on 280,000 abstracts manually labelled as being RCTs or not by Cochrane Crowd, 2 a collaborative citizen science project. We do not rely on the Publication Type index alone, as we have previously found it to miss a substantial proportion of the 5-7 most recent years of articles [11] (due to delay in manual indexing after publication). We next removed any RCTs that were not conducted in humans (e.g., animal or agricultural studies) using an SVM model. This model was trained using labelled abstracts from PubMed (labels derived as a function of whether the MeSH 3 term 'Humans' had been applied or not).
To characterise the trial participants and outcomes, we first automatically extract 'snippets' of text (typically a few consecutive words within the abstract) describing these components. For this, we use a sequence tagging machine learning model (namely a Long Short-Term Memory network [15] 1 Model 'training' describes the process of adjusting model parameters to align with (typically large amounts of) data. Here, the models consume text documents (abstracts), with manual labels designating whether these are RCTs or not. Once trained (i.e., once the parameters are estimated) the model can be applied to new, unseen abstracts to provide predictions. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020.10.08.20209353 doi: medRxiv preprint followed by a conditional random field, [16] usually abbreviated to LSTM-CRF [17] ). [18] This model was trained using a publicly available dataset called EBM-NLP, which comprises 5,000 RCT abstracts in which text spans relevant to the respective PICO characteristics (participants, interventions, comparators, and outcomes) were manually annotated.
These text descriptions are then standardised to MeSH terms. This allows us to infer, for example, that 'myocardial infarction' and 'heart attack' refer to the same condition, and that both are subtypes of cardiovascular disease. To achieve this step, we use an approach similar to that described by Demner-Fushman and colleagues, in their MetaMap software. [19] This approach involves generating a large database of text synonyms for MeSH terms, done by finding alternative descriptions for each term in linked vocabularies. These synonyms are then sought in the extracted population and outcomes text spans.
In this study, we use the 22 level 2 categories defined by the GBD collaboration. These categories describe broad disease areas such as "Cardiovascular diseases", and "Mental disorders" (the full list is provided in Table 1 ). The GBD collaboration makes available International Classification of Diseases (ICD-10) codes which define the scope of each category. We automatically link these ICD-10 codes to the equivalent MeSH terms using a tool and dataset named Metathesaurus, from the Unified Medical Language System (UMLS). [20] In brief, Metathesaurus provides a database of synonyms between different health science vocabularies, and thus lists of synonymous terms between ICD-10 and MeSH can be retrieved. One clinical author (IJM) manually checked all of the automatically generated mappings from MeSH to Global Burden of Disease category (both removing any errors from the automatic mapping, and adding links where they had been missed). This resulted in a final mapping of 5,256 parent MeSH terms to one of the 22 GBD categories.
We report descriptive statistics for RCT counts and participant numbers. We use the country of the first author's affiliation as a proxy for the country organising or funding the study. In order to capture trials addressing prevention, we also consider RCTs as relevant to the GBD category where rates of the health condition in question are identified as an outcome, even where participants did not have the condition at trial onset. For example, trials that recruited people with diabetes but measured the effects of an intervention on subsequent coronary heart disease were considered as relevant to both the 'diabetes' and 'cardiovascular diseases' categories. We also . CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020.10.08.20209353 doi: medRxiv preprint present separate estimates based on this distinction, that is, separating instances where a given GBD condition was found in the trial population ('treatment' trials) from those where it was inferred to describe an outcome ('prevention' trials).
We randomly sampled articles (titles and abstracts) from the finalised dataset, and two authors independently manually labelled these with respect to study design (RCT in humans vs. not; 500 abstracts, done by IJM and BCW), Global Burden of Disease category (250 abstracts, done by IJM and VL), and number of participants randomized (500 abstracts, done by IJM and BCW). Authors then met to resolve discrepancies by consensus. We then used the bootstrap method to estimate 95% prediction intervals for final counts, using the method described by Fox and colleagues. [21, 22] In short, we use our manually labeled data to calculate 10,000 bootstrap estimates of the precision and recall of each labeling task. [23] These precision and recall estimates are then used to adjust the automated results from the full dataset (generating 10,000 adjusted count estimates for each classification task), which can be used to construct a 95% prediction interval. To evaluate uncertainty around extracted sample sizes we use a similar approach, except we used estimates of the absolute error taken from the manually labeled abstracts. We then simulated the effect of this error on the full dataset using the bootstrap method with 10,000
iterations. These intervals permit assessment of the degree to which model misclassification and bias has affected the final results.
To examine associations between global disease burden and volume of evidence (both numbers of RCTs and participants), we used a series of linear regression models with log-transformed global DALYs as the predictor, and the log-transformed numbers of global RCTs as the dependent variable. We evaluate annual data on DALYs for each clinical condition for the full period available from the Global Burden of Disease study , and adjust for trends over time by incorporating the year as a categorical predictive variable. A detailed description alongside statistical code and data for these analyses is available at the Open Science Framework (DOI: 10.17605/osf.io/3db76).
In our first regression analysis, we examined the association of global DALYs on RCT publications in the same year. We explored incorporating the year (as categorical variable) both as a fixed and a random effect, evaluating using the Hausman test. We compare observed RCT counts for each GBD condition with those predicted by the model. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . In our second regression analysis, we examine the effect of lag (given that RCTs will publish reports some time after measures of disease burden are known). Here we use finite length distributed lag models, and evaluate the association of global DALYs with up to 1, 2, 3, 4, and 5 years lag on published RCTs in the index year. We compare models using the adjusted R 2 statistic, the Akaike information criterion (AIC), and Bayesian information criterion (BIC).
In our third regression analysis, we add a variable to account for whether a particular condition tends to impose a disproportionate burden in high-income, rather than low and middle-income countries.
We examine associations between socioeconomic status and evidence volume, using the Socio-Demographic Index (SDI) of the country from the Global Burden of Disease study. As a metric of disparity in disease burden, we used the ratio of DALYs occurring in high SDI countries (the top fifth) to DALYs occurring in low and medium SDI countries (i.e., the bottom four fifths).
Finally, we conducted a sensitivity analysis aiming to discover whether our results are sensitive to 'salami slicing' (the practice of publishing results from a single trial across multiple publications, which could lead us to overestimate the evidence base). We first estimated the mean number of publications per RCT for each GBD condition, utilising the subset of articles which contained a unique trial registry identifier. Then we use these values to discount our estimate of RCT numbers for each GBD condition. We repeated our regression analyses with these discounted RCT numbers.
We analysed records of 31,367,011 articles indexed in PubMed on July 27th 2020. We estimate that 459,000 (1.5%) of these articles were reports of RCTs conducted in humans (95% prediction interval 435,000 to 480,000). We show how these trials are distributed globally (by location of the first author)
in is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020.10.08.20209353 doi: medRxiv preprint prevention or diagnosis of the condition) in Table 1 and Figure 2 . In all cases except for Mental disorders, the numbers of trials and participants in the 'treatment' trials exceeded those in the 'prevention' trials.
. CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020.10.08.20209353 doi: medRxiv preprint Our primary analysis found that, on average, increasing DALYs were associated with a statistically significant increase in RCT publication in the same year, but that variance was not well explained by the model (β=0.52 for log DALYs, 95% confidence interval 0.33 to 0.70; R 2 =0.13). This translates to an average increase in the number of RCTs published of 5% (3.2% to 6.9%) for each 10% increase in DALYs in the same year. We report differences between the expected versus observed number of RCTs and trial participants based on this model in Table 2 . is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . https://doi.org/10.1101/2020.10.08.20209353 doi: medRxiv preprint
We found that models incorporating between 3-5 years of lag had improved fit compared with the equivalent model with no lag effects (demonstrated by reduction in Akaike information criterion
[AIC], Bayesian information criterion [BIC] , and increase in adjusted R 2 ). The model incorporating lag up to 3 years found an average 7.2% increase in published RCTs for every 10% increase in DALYs (adjusted R 2 =0.29). The models with lag up to 4, and up to 5 years gave near identical estimates and fit (full analysis results provided in the statistical appendix).
We provide full results for our sensitivity analysis (where we adjust for the differential impact of RCTs producing multiple publications in each disease in our statistical appendix (https://osf.io/zf97n/). We found that RCTs with a registry identifier produced a mean of 1.3 articles, but that there was variation between GBD categories (from 1.2 articles/RCT in nutritional deficiencies to 1.5 articles/RCT in cardiovascular diseases). Compared with our primary results, our sensitivity analysis found a near identical relationship between DALYs and published trials (β=0.48, 95% CI 0.30 to 0.66; R 2 =0.13). Adjusting for multiple publications did not affect which conditions were found to have relatively high or low associated RCT publications. In our final model, which examined the effect of the DALY location (ratio of DALYs in the top fifth of countries by socio-demographic index vs. the bottom four fifths), we found that DALYs, and the DALY location were both associated with increased RCT publications (βDALY=0.75, 95% CI 0.56 to 0.91, βlocation=0.44, 95% CI 0.38 to 0.51; adjusted R 2 =0.33). This may be interpreted that for every 10% shift in DALYs towards high-income countries, the number of RCTs increased by 4% (3.7 to 4.9), and for every 10% increase in DALYs globally, RCTs increased by 7% (5.7 to 9.1).
We present the results from a natural language processing system that we used to perform a large-scale, comprehensive, automated analysis of all published RCTs indexed in the PubMed database. We find the number of published RCTs covering health conditions correlates only weakly with the burden that they inflict globally. Some conditions (particularly respiratory infections and tuberculosis, enteric infections, and transport injuries) attract substantially fewer published RCTs than expected based on the global burden.
Conditions with fewer RCTs and participants than expected tend to disproportionately affect countries with low-and medium socio-economic status. For instance, respiratory infections and tuberculosis, which had the largest disparity between global burden and number of published is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . RCTs disproportionately affects populations in lower income countries. [24] By contrast, cancers and diabetes, for which there appears to be a relative abundance of evidence, are comparatively common in high-income countries, although this picture is changing over time. [25, 26] In other words, the top fifth of countries by socio-demographic index, who conduct the vast majority of RCTs, are more closely matching RCT conduct with disease burden within their own countries (albeit still weakly). These findings suggest that a research funding strategy based on individual countries priorities does not optimally meet global needs. However, the picture is complex: global development has meant that the most important diseases affecting high-income countries (i.e. cardiovascular disease and cancers) have also become the top health priorities in low-and middleincome countries a number of years later.
The median number of participants per RCTs was 72 (i.e., a typical trial is likely to have around 36 participants per arm, or fewer trials with >2 arms). Smaller trials can be conducted faster and at lower cost, and thus can be more responsive to need, but have important limitations. [27] Small trials are more susceptible to the effects of publication bias, and have a high likelihood of generating false positive, and false negative findings. [28] Meta-epidemiological studies have shown that trials with fewer than 100 participants per arm tend to exaggerate treatment effects, compared with larger ones. [29, 30] Twenty-five years after Yusuf and colleagues called for larger, simpler RCTs, [31] unfortunately we find that a majority of published trials are likely to be too small to provide useful or definitive information.
Recently published related work has used machine learning to extract sex data automatically from a large set of articles describing RCTs, finding evidence of systematic underrepresentation of women in trials. [32] Here, we broaden this approach by applying machine learning to infer the conditions and outcomes under study in trials and extracting corresponding sample sizes. By using our previously validated model [12] to automatically recognize reports of RCTs, we are able to continuously surveil the literature to maintain an up-to-date, comprehensive view of the evidence, which we make publicly available to facilitate further research.
Prior work has investigated the relationship between burden of disease and NIH funding. [8, 5] These efforts relied on manual compilation of data, which necessarily limited the scope of analysis. Additionally, these efforts could only analyze conditions for which funding data was available. The authors of the landmark 1999 NEJM study noted that data was not available for conditions making up 38% of the total DALY estimate at that time. [5] Conducting this analysis manually additionally precludes the realization of a "living" view of the evidence base. By is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. . contrast, we have developed a fully automated approach that facilitates comprehensive, real-time analysis of the published RCT literature. For illustration, at the time of writing (September 2020), our system has indexed 158 RCTs examining treatments for covid-19. This contrasts with PubMed, which has indexed (largely manually) 58 with the equivalent MeSH terms.
Because our coverage of diseases is broader than prior work on the relationship between funding and burden of disease, the results are not directly comparable. However, some findings do qualitatively align with findings from prior analyses of research funding. For example, we also find that diabetes generated a relatively generous number of RCTs, as do cancers (in general).
Future analyses using the methods and data could examine specific conditions within the categories presented here, investigating, for example, whether stroke is adequately researched compared with coronary artery disease, or the distribution of trials relevant to different subtypes of cancer.
Our analysis has some limitations. First, our primary analysis examines publications and not RCTs themselves. Our sensitivity analysis confirms previous findings that RCTs may generate more than one publication, [33] and so our estimates of publication numbers are likely to be an overestimate of the number of trials conducted. However, we found adjusting for estimates of duplicate publication made little difference to estimates of research done as a function of global burden. Second, the automated data extraction methods used here are only able to count explicit mentions of a disease or disorder, and explicit descriptions of participant numbers from the abstract, but abstracts do not always contain all the necessary information (in this study, for example, 34% of articles publications did not report the number of study participants). There is evidence of improvement over time after the publication of the CONSORT recommendations in 1996. [34] Last, we restrict our enquiry to PubMed and therefore have not counted RCTs which have produced publications in journals not indexed in PubMed. PubMed tends to under-represent articles published in non-English languages (though includes some where an English-language translations abstracts exist). PubMed also does not index conference presentations or pharmaceutical company data repositories. By focusing on published articles, this analysis will by construction miss RCTs whose results are never published (although unpublished trial results are arguably not effectively contributing to reducing the global disease burden). Nonetheless, PubMed is reasonably comprehensive; and those RCT publications that it misses have been found to comprise a relatively small fraction of the available health research. [35, 36] . CC-BY 4.0 International license It is made available under a perpetuity.
is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. .
Disparities in funding may not be the sole explanation: organisational and cultural differences within medical subdisciplines could plausibly play a role. National strategies exist for increasing participation in oncology research (e.g., 2008 guidance from the American Society of Clinical Oncology which set a target of >10% of patients participating in clinical trials [37] ); a large proportion of research looking at how best to increase trial recruitment has arisen from cancer RCTs. [38] Further investigation of how the better performing medical disciplines achieve their high recruitment rates might uncover strategies which are more widely applicable.
Clinical trials are one small part of tackling the global disease burden and the UN sustainable development goals, and this wider landscape needs to be considered when determining research priorities. For example, the provision of clean water is likely to be more important for preventing enteric infections than any specific health intervention, and probably does not require an RCT to prove it. [39] There may be other conditions (e.g., HIV [40] ) where established treatments can enable those with the condition to have a normal lifespan without significant disability. This contrasts with some types of cancer (e.g. pancreatic cancer[41]), where we still lack reliably effective treatments. For pharmacological research, the ability to run new trials depends on the availability of new candidate drugs to test.[42] Such new candidate drugs depend upon unpredictable scientific breakthroughs, which will not neatly tally with global disease burden rankings.
Nonetheless, this data on the relative attention attracted by different health conditions may provide a useful metric to consider alongside other indicators of research need. Given that the vast majority of trials are being conducted by the wealthiest 20% of countries, targeting national health priorities is not sufficient, and a global approach is needed. We publish daily updates of the figures in this paper at www.robotreviewer.net/stateoftheevidence, and make the source code and dataset freely available via the Open Science Framework (DOI: 10.17605/osf.io/3db76). is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint
The copyright holder for this this version posted October 11, 2020. .