CAFA and the Open World of protein function predictions
Authors: Christophe Dessimoz, Nives Škunca, Paul D. Thomas
Journal: Trends in Genetics (2013)

The plummeting cost of DNA sequencing means that vast amounts of gene sequences are becoming available across all domains of life. Translating this information into useful biological and biomedical knowledge requires an understanding of the biological functions of these genes. Unlike sequencing, however, discovering the function(s) of a gene remains painstaking work that is largely restricted to a handful of model species. Thus, biological analyses, particularly of non-model organisms, increasingly rely on computational inference, or ‘prediction’, of function. As such, it is critical to understand the strengths and weaknesses of different prediction methods. Towards this end, the first ‘critical assessment of protein function annotation’ (CAFA) experiment was recently reported, a community-wide effort to evaluate computational function prediction methods [1Radivojac P. et al.A large-scale evaluation of computational protein function prediction.Nat. Methods. 2013; 10: 221-227Crossref PubMed Scopus (566) Google Scholar]. Following an open call, 23 participating teams submitted 54 algorithms for assessment that predicted gene ontology (GO) functional terms [2Gene Ontology ConsortiumThe Gene Ontology in 2010: extensions and refinements.Nucleic Acids Res. 2010; 38: 331-335Crossref Scopus (369) Google Scholar] for a common set of ∼50 000 proteins that then lacked experimentally corroborated annotations. In the 11 months following the submission deadline, GO curators – continuing to work independently from the CAFA organisers – examined relevant literature and assigned functional annotations to 866 of those proteins. These 866 proteins became the gold standard reference set for evaluating the performance of all CAFA submissions. We applaud this effort and fully support the aims of CAFA, but we are concerned that the primary CAFA evaluation metric fails to account for the ‘Open World’ assumption underlying GO annotations [3Thomas P.D. et al.On the use of gene ontology annotations to assess functional similarity among orthologs and paralogs: a short report.PLoS Comput. Biol. 2012; 8: e1002386Crossref PubMed Scopus (69) Google Scholar]: the functional annotations of most proteins are incomplete and, consequently, absence of evidence of function does not amount to evidence of absence of function. This omission leads to a systematic overestimation of false-positive prediction rates, which may significantly affect the results and conclusions reported in the CAFA study. We agree that the gold standard reference set can confirm predictions (i.e., count true positives) but, because it does not exhaustively represent all functions of the target sequences, we argue that the reference set cannot falsify predictions (i.e., count false positives). To illustrate this point, consider the first target in the reference dataset: CLC4E_MOUSE, which was assigned the molecular functions ‘receptor activity’ and ‘protein binding’ based on GO annotations accrued in the Swiss-Prot database [4UniProt ConsortiumReorganizing the protein space at the Universal Protein Resource (UniProt).Nucleic Acids Res. 2012; 40: 71-75Crossref Scopus (1091) Google Scholar] in the 11 months following the close of the competition (Supplementary Table 1 in [1Radivojac P. et al.A large-scale evaluation of computational protein function prediction.Nat. Methods. 2013; 10: 221-227Crossref PubMed Scopus (566) Google Scholar]). InterProScan [5Hunter S. et al.InterPro in 2011: new developments in the family and domain prediction database.Nucleic Acids Res. 2012; 40: 306-312Crossref Scopus (847) Google Scholar] also predicts a function of ‘carbohydrate binding’, which in CAFA would be considered a false positive because it does not appear in the gold standard set. However, this prediction is actually correct, based on experimental evidence of alpha-mannose binding [6Yamasaki S. et al.C-type lectin Mincle is an activating receptor for pathogenic fungus, Malassezia.Proc. Natl. Acad. Sci. U.S.A. 2009; 106: 1897-1902Crossref PubMed Scopus (317) Google Scholar] that is not yet recorded in Swiss-Prot GO annotations. This example is not atypical, because the Swiss-Prot database is maintained by expert curators who, owing to resource limitations, process entries according to defined priorities. Even with vastly more resources, the database would remain incomplete because most functional information has yet to be discovered through direct experiments in the first place. To quantify the extent of spurious false positives that results from disregarding the Open World assumption, we simulated the CAFA experiment by considering a different, older set of data for which we now have the benefit of hindsight: successive releases of the UniProt-GOA database [7Dimmer E.C. et al.The UniProt-GO annotation database in 2011.Nucleic Acids Res. 2011; 40: 565-570Crossref Scopus (267) Google Scholar] dating back to 2007. Analogous to predictions submitted to CAFA, we first retrieved all gene products with computational (predicted) annotation but no experimental annotation in the 2007-01-19 release. Analogous to the way the CAFA gold standard set was built, we then established which of these gene products accumulated new experimental annotations between the 19 January 2007 and 16 January 2008 releases. For these targets, and following the CAFA protocol, we counted as false positives all electronic annotations in the 19 January 2007 release that were not confirmed by an experimental annotation in the 16 January 2008 release. However, a considerable proportion of these purported false-positive predictions were in fact confirmed by experimental annotations in subsequent UniProt-GOA releases (22 January 2012, 11 January 2011, 7 February 2012, 7 January 2013), thereby contradicting the initial assessment (Figure 1): in our analysis, ∼14.7% of the predictions initially deemed as false positives were later confirmed to be correct. This is necessarily an underestimate of the error rate in the CAFA definition of false-positive predictions, and questions the ranking of methods reported by CAFA. To compare prediction methods meaningfully, future CAFA sequels *The next CAFA challenge has just been announced (http://biofunctionprediction.org/node/20).*The next CAFA challenge has just been announced (http://biofunctionprediction.org/node/20). should consider the Open World assumption and tackle head-on the complications associated with it. For instance, explicit annotations of absence of function – identified by the keyword ‘NOT’ in the qualifier field of GO annotations – should be required to falsify predictions. Currently, however, only about 2500 of the 530 000 (0.48%) experimentally confirmed molecular function and biological process annotations in UniProt-GOA are negative ones – in part because they have often been perceived as less useful than their positive counterparts. But, to improve and evaluate function prediction, negative annotations are invaluable and more of them are needed [8Skunca N. et al.Quality of computationally inferred gene ontology annotations.PLoS Comput. Biol. 2012; 8: e1002533Crossref PubMed Scopus (88) Google Scholar]. Another way of addressing the problem would be to limit the scope of function prediction to specific aspects of function that can be thoroughly assessed in experiments after the submission deadline (e.g., particular enzymatic activities) [9Huttenhower C. et al.The impact of incomplete knowledge on evaluation: an experimental benchmark for protein function prediction.Bioinformatics. 2009; 25: 2404-2410Crossref PubMed Scopus (26) Google Scholar]. For these restricted functional aspects, the more straightforward ‘Closed World’ assumption would apply, but the conclusions drawn might not hold in general. Neither solution constitutes a ‘quick fix’. Indeed, progress in assessing protein function prediction is likely to require a substantial coordinated effort and broad support from the community. In that sense, the CAFA group is already well positioned to help drive the field forward. N.Š. participated in the CAFA experiment and was a co-author on the CAFA paper [1Radivojac P. et al.A large-scale evaluation of computational protein function prediction.Nat. Methods. 2013; 10: 221-227Crossref PubMed Scopus (566) Google Scholar].