Bioinformatics Advance Access originally published online on June 26, 2009
Bioinformatics 2009 25(18):2404-2410; doi:10.1093/bioinformatics/btp397
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The impact of incomplete knowledge on evaluation: an experimental benchmark for protein function prediction
1Department of Computer Science, Princeton University, 35 Olden Street, Princeton, NJ 08540-5233, 2Lewis-Sigler Institute for Integrative Genomics, Carl Icahn Laboratory, Princeton University, Princeton, NJ 08544, 3Jackson Laboratory, 600 Main Street, Bar Harbor, ME 04609 and 4Department of Computer Science, University of Minnesota, 200 Union Street SE, Minneapolis, MN 55455, USA
*To whom correspondence should be addressed.
 |
Abstract |
|---|
Motivation: Rapidly expanding repositories of highly informative genomic data have generated increasing interest in methods for protein function prediction and inference of biological networks. The successful application of supervised machine learning to these tasks requires a gold standard for protein function: a trusted set of correct examples, which can be used to assess performance through cross-validation or other statistical approaches. Since gene annotation is incomplete for even the best studied model organisms, the biological reliability of such evaluations may be called into question.
Results: We address this concern by constructing and analyzing an experimentally based gold standard through comprehensive validation of protein function predictions for mitochondrion biogenesis in Saccharomyces cerevisiae. Specifically, we determine that (i) current machine learning approaches are able to generalize and predict novel biology from an incomplete gold standard and (ii) incomplete functional annotations adversely affect the evaluation of machine learning performance. While computational approaches performed better than predicted in the face of incomplete data, relative comparison of competing approaches—even those employing the same training data—is problematic with a sparse gold standard. Incomplete knowledge causes individual methods' performances to be differentially underestimated, resulting in misleading performance evaluations. We provide a benchmark gold standard for yeast mitochondria to complement current databases and an analysis of our experimental results in the hopes of mitigating these effects in future comparative evaluations.
Availability: The mitochondrial benchmark gold standard, as well as experimental results and additional data, is available at http://function.princeton.edu/mitochondria
Contact: ogt{at}cs.princeton.edu
Supplementary information: Supplementary data are available at Bioinformatics online.
The authors wish it to be known that, in their opinion, the first three authors should be regarded as joint First Authors.
Associate Editor: Jonathan Wren
Received on April 10, 2009; revised on June 5, 2009; accepted on June 23, 2009
CiteULike 
Connotea 
Del.icio.us What's this?