Academic output of 9 years of EU investment into health research
Authors: Michael J Galsworthy, Dimitar Hristovski, Lara Lusa, Kelly Ernst, Rachel Irwin, Kate Charlesworth, Matthias Wismar, Martin McKee
Journal: The Lancet (2012)

The European Union's (EU's) Framework Programmes (FP) for Research and Technological Development are major funders of research in Europe. The European Commission clearly values its science programme: the budget rose from €19·1 billion under FP61Community Research and Development Information ServiceFP6 budget.http://cordis.europa.eu/fp6/budget.htmGoogle Scholar (2002–06) to €53 billion2Community Research and Development Information ServiceBudget breakdown of the Seventh Framework Programme of the European Community (EC) (2007-2013) and Euratom (2007-2011) (in EUR million).http://cordis.europa.eu/fp7/budget_en.htmlGoogle Scholar in FP7 (2007–13). With billions of taxpayers' Euros invested, it is desirable to know where the money is being spent and what is produced. We attempted to use an informatics approach to identify academic publications resulting from health-related research funded under FP5 and FP6. Although the Research Portfolio Online Reporting Tools (RePORT) of the US National Institutes of Health (NIH) dynamically link to PubMed/Medline to return papers associated with NIH grant codes, there is no similar system for the online record of EU-funded projects (Community Research and Development Information Service [CORDIS]). We therefore wrote a program to data-mine Google Scholar with EU grant codes, counting papers returned. We then wrote another mining program using paper titles extracted from Google Scholar (which subsumes PubMed; appendix) to link to PubMed. There were 26 946 projects in the full CORDIS database. Manual classification identified 4756 of these projects as relating to human health. However, since only half the FP6 projects had unique alphanumeric grant codes (appendix), we were only able to search for academic output for 3668 projects. Of these projects 50·1% produced no identifiable articles according to Google Scholar. The rate was 56·0% according to PubMed. The table breaks this finding down into funding bands (our divisions). Division of the entire investment by the total number of Google Scholar papers produced a mean of €140 244 per paper. With PubMed data (upper limit of 100 papers/project, appendix), a figure of €224 679 was reached. €570·2 million (€755·4 million by PubMed data) of health-related research resulted in no detectable academic output.TableComparison of Google Scholar (GS) and PubMed outputs by funding bandNumber of projectsTotal funds (€millions)Number of papers in GS (PubMed)Money/paper in GS (PubMed; €1000s)Number of projects with no papers in GSNumber of projects with no papers in PubMedFunds allocated to projects with no papers in GS (PubMed; €millions)€1–80K57326·3249 (139)105·6 (189·1)505 (88·1%)520 (90·8%)21·8 (22·7)>€80–115K35237·7278 (170)135·7 (221·9)268 (76·1%)286 (81·3%)28·7 (30·6)>€115–150K45861·9364 (243)170·0 (254·6)325 (71·0%)351 (76·6%)44·2 (47·6)>€150–240K45781·9341 (230)240·1 (356·0)350 (76·6%)369 (80·7%)61·6 (65·2)>€240–900K483263·52728 (1436)96·6 (183·5)214 (44·3%)268 (55·5%)107·0 (137·7)>€900K–1·45 million421496·53881 (2712)127·9 (183·1)75 (17·8%)108 (25·7%)85·0 (122·7)>€1·45–2 million441754·85924 (4057)127·4 (186·0)56 (12·7%)81 (18·4%)97·0 (139·1)>€2–2·7 million241555·73152 (1939)176·3 (286·6)30 (12·4%)44 (18·3%)68·4 (99·9)>€2·7–5 million140454·22955 (1789)148·6 (247·0)14 (10·0%)23 (16·4%)45·0 (73·0)>€5–17 million1021047·57081 (4109)153·7 (253·9)2 (2·0%)3 (2·9%)11·5 (16·9)Total36683780·026 953 (16824)140·2 (224·7)1839 (50·1%)2053 (56·0%)570·2 (755·4) Open table in a new tab Furthermore, €4·163 billion was un-accounted for in CORDIS. The 26 946 projects in the full database accounted for €29·91 billion (€13·41 billion for FP5; €16·50 billion for FP6), compared with the expected €34·073 billion (€14·96 billion for FP53Community Research and Development Information ServiceGlobal FP5 budget (EC & Euratom sections) (in million euro).http://cordis.europa.eu/fp5/src/budget.htmGoogle Scholar and €19·113 billion for FP6.1Community Research and Development Information ServiceFP6 budget.http://cordis.europa.eu/fp6/budget.htmGoogle Scholar) The difference was not explained by projects with zero or missing financial values (123 in total), and is far too large for administrative costs. However, it might include project top-ups (unrecorded in this database), projects not in the database, or underspends. The Directorate-General for Research and Innovation should provide clarification. Finally, €2·2 billion was simply untraceable owing to a change of policy on grant codes from FP5 to FP6. The unique FP5 alphanumeric codes (which include contract type and year) are ideal for both web mining and meta-analysis. We strongly recommend a return to such structured alphanumeric codes for the EU's next programme (beginning in 2014). Limited space precludes us from listing all caveats; however, we must state that more papers do not equate to "better" output. Additionally we cannot presume that projects do, as required, cite their grants in their papers and do so accurately. A well developed Medline-linked (and author-checked) online database such as NIH's RePORT should address that issue. Also, an unknown proportion of projects had industry involvement and so might not be expected to publish findings in the public domain. We recommend development of practical project-type and subject-area classification systems within the CORDIS database in future. This is also necessary should the EU develop an online graphical interface to allow open analysis of their CORDIS database. How should the EU secure more scientific output from its financial input? Beyond encouraging papers, it could, where relevant, demand annotated databases to be submitted to a central EU repository as a final deliverable. At some point after project termination, access to the database would become permanently open for meta-analyses and data reuse. The potential benefit is colossal. Such databases could be cited like publications, complete with author lists, year, and reference number. If the USA can provide the global authoritative resource for academic papers (PubMed), why should the EU—the world's most prolific body for academic output—not take the initiative to establish the leading scientific data repository? This letter was written as part of the Health Research for Europe (HR4E) project, funded by the European Commission's Directorate General for Research, within the 6th Framework Programme. Grant number: LSSM-CT-2007-037397. We declare that we have no conflicts of interest. Download .pdf (.19 MB) Help with pdf files Supplementary appendix Academic output from EU-funded health research projectsMichael Galsworthy and colleagues1 examine the academic output of investment in health research by European Union (EU) Framework Programmes (FPs) 5 and 6. Notwithstanding the limitations of their approach (such as the fact that only half of FP6 projects had unique alphanumeric codes), their statement that €570·2 million resulted in no detectable academic output omits other key dissemination areas. Full-Text PDF Academic output from EU-funded health research projectsMichael Galsworthy and colleagues' Correspondence "Academic output of 9 years of EU investment into health research" (Sept 15, p 971)1 leaves the impression that health research projects funded by the European Union (EU) have no or extremely limited academic output. Full-Text PDF