Bioinformatics Advance Access originally published online on October 28, 2004
Bioinformatics 2005 21(7):1046-1054; doi:10.1093/bioinformatics/bti081
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Superior feature-set ranking for small samples using bolstered error estimation
1Department of Electrical Engineering, Texas A&M University College Station, TX, USA
2Section of Clinical Cancer Genetics, University of Texas M. D. Anderson Cancer Center Houston, TX, USA
3Department of Pathology, University of Texas M. D. Anderson Cancer Center Houston, TX, USA
*To whom correspondence should be addressed.
Motivation: Ranking feature sets is a key issue for classification, for instance, phenotype classification based on gene expression. Since ranking is often based on error estimation, and error estimators suffer to differing degrees of imprecision in small-sample settings, it is important to choose a computationally feasible error estimator that yields good feature-set ranking.
Results: This paper examines the feature-ranking performance of several kinds of error estimators: resubstitution, cross-validation, bootstrap and bolstered error estimation. It does so for three classification rules: linear discriminant analysis, three-nearest-neighbor classification and classification trees. Two measures of performance are considered. One counts the number of the truly best feature sets appearing among the best feature sets discovered by the error estimator and the other computes the mean absolute error between the top ranks of the truly best feature sets and their ranks as given by the error estimator. Our results indicate that bolstering is superior to bootstrap, and bootstrap is better than cross-validation, for discovering top-performing feature sets for classification when using small samples. A key issue is that bolstered error estimation is tens of times faster than bootstrap, and faster than cross-validation, and is therefore feasible for feature-set ranking when the number of feature sets is extremely large.
Availability: We provide a companion website, which contains the complete set of tables and plots regarding the simulation study, and a compilation of references on feature-set ranking with applications in Genomics. The companion website can be accessed at the URL http://ee.tamu.edu/~edward/bolster_ranking
Contact: edward{at}ee.tamu.edu
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  Y. Saeys, I. Inza, and P. Larranaga A review of feature selection techniques in bioinformatics Bioinformatics, October 1, 2007; 23(19): 2507 - 2517. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. D. Onken, L. A. Worley, R. M. Davila, D. H. Char, and J. W. Harbour Prognostic Testing in Uveal Melanoma by Transcriptomic Profiling of Fine Needle Biopsy Specimens J. Mol. Diagn., November 1, 2006; 8(5): 567 - 573. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
X. Zhou and K. Z. Mao The ties problem resulting from counting-based error estimators and its impact on gene selection algorithms Bioinformatics, October 15, 2006; 22(20): 2507 - 2515. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Choudhary, M. Brun, J. Hua, J. Lowey, E. Suh, and E. R. Dougherty Genetic test bed for feature selection Bioinformatics, April 1, 2006; 22(7): 837 - 842. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Larranaga, B. Calvo, R. Santana, C. Bielza, J. Galdiano, I. Inza, J. A. Lozano, R. Armananzas, G. Santafe, A. Perez, et al. Machine learning in bioinformatics Brief Bioinform, March 1, 2006; 7(1): 86 - 112. [Abstract] [Full Text] [PDF]
|
|