Bioinformatics Advance Access originally published online on March 28, 2007
Bioinformatics 2007 23(11):1363-1370; doi:10.1093/bioinformatics/btm117
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Classification based upon gene expression data: bias and precision of error rates
1School of Mathematical Sciences, Queensland University of Technology, Gardens Point, GPO Box 2434, Brisbane, QLD 4001, Australia and 2Queensland Institute of Medical Research, Post Office, Royal Brisbane Hospital, 300 Herston Rd., Herston, QLD 4029, Australia
*To whom correspondence should be addressed.
 |
Abstract |
|---|
Motivation: Gene expression data offer a large number of potentially useful predictors for the classification of tissue samples into classes, such as diseased and non-diseased. The predictive error rate of classifiers can be estimated using methods such as cross-validation. We have investigated issues of interpretation and potential bias in the reporting of error rate estimates. The issues considered here are optimization and selection biases, sampling effects, measures of misclassification rate, baseline error rates, two-level external cross-validation and a novel proposal for detection of bias using the permutation mean.
Results: Reporting an optimal estimated error rate incurs an optimization bias. Downward bias of 3–5% was found in an existing study of classification based on gene expression data and may be endemic in similar studies. Using a simulated non-informative dataset and two example datasets from existing studies, we show how bias can be detected through the use of label permutations and avoided using two-level external cross-validation. Some studies avoid optimization bias by using single-level cross-validation and a test set, but error rates can be more accurately estimated via two-level cross-validation. In addition to estimating the simple overall error rate, we recommend reporting class error rates plus where possible the conditional risk incorporating prior class probabilities and a misclassification cost matrix. We also describe baseline error rates derived from three trivial classifiers which ignore the predictors.
Availability: R code which implements two-level external cross-validation with the PAMR package, experiment code, dataset details and additional figures are freely available for non-commercial use from http://www.maths.qut.edu.au/profiles/wood/permr.jsp
Contact: i.wood@qut.edu.au
Supplementary information: Supplementary data are available at Bioinformatics online.
Associate Editor: John Quackenbush
Received on November 27, 2006; revised on March 12, 2007; accepted on March 15, 2007
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  S. Lee Mistakes in validating the accuracy of a prediction classifier in high-dimensional but small-sample microarray data Statistical Methods in Medical Research, December 1, 2008; 17(6): 635 - 642. [Abstract] [PDF]
|
|