Bioinformatics Advance Access originally published online on August 11, 2005
Bioinformatics 2005 21(20):3940-3941; doi:10.1093/bioinformatics/bti623
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ROCR: visualizing classifier performance in R
1Department of Computational Biology and Applied Algorithmics, Max-Planck-Institute for Informatics Stuhlsatzenhausweg 85, 66123 Saarbrücken, Germany
2Department of Mathematics, University of California Berkeley, CA 94720-3840, USA
*To whom correspondence should be addressed.
 |
Abstract |
|---|
 TOP Abstract REFERENCES |
|---|
Summary: ROCR is a package for evaluating and visualizing the performance of scoring classifiers in the statistical language R. It features over 25 performance measures that can be freely combined to create two-dimensional performance curves. Standard methods for investigating trade-offs between specific performance measures are available within a uniform framework, including receiver operating characteristic (ROC) graphs, precision/recall plots, lift charts and cost curves. ROCR integrates tightly with R's powerful graphics capabilities, thus allowing for highly adjustable plots. Being equipped with only three commands and reasonable default values for optional parameters, ROCR combines flexibility with ease of usage.
Availability: http://rocr.bioinf.mpi-sb.mpg.de. ROCR can be used under the terms of the GNU General Public License. Running within R, it is platform-independent.
Contact: tobias.sing{at}mpi-sb.mpg.de
Pattern classification has become a central tool in bioinformatics, offering rapid insights into large data sets (Baldi and Brunak, 2001). While one area of our work involves predicting phenotypic properties of HIV-1 from genotypic information (Beerenwinkel et al., 2002, 2003; Sing et al., 2004), scoring or ranking predictors are also vital in a wide range of other biological problems. Examples include microarray analysis (e.g. prediction of tissue condition based on gene expression), protein structural and functional characterization (remote homology detection, prediction of post-translational modifications and molecular function annotation based on sequence or structural motifs), genome annotation (gene finding and splice site identification), protein–ligand interactions (virtual screening and molecular docking) and structure–activity relationships (predicting bioavailability or toxicity of drug compounds). In many of these cases, considerable class skew, class-specific misclassification costs, and extensive noise due to variability in experimental assays complicate predictive modelling. Thus, careful predictor validation is compulsory.
The real-valued output of scoring classifiers is turned into a binary class decision by choosing a cutoff. As no cutoff is optimal according to all possible performance criteria, cutoff choice involves a trade-off among different measures. Typically, a trade-off between a pair of criteria (e.g. sensitivity versus specificity) is visualized as a cutoff-parametrized curve in the plane spanned by the two measures. Popular examples of such trade-off visualizations include receiver operating characteristic (ROC) graphs, sensitivity/specificity curves, lift charts and precision/recall plots. Fawcett (2004) provides a general introduction into evaluating scoring classifiers with a focus on ROC graphs.
Although functions for drawing ROC graphs are provided by the Bioconductor project (http://www.bioconductor.org) or by the machine learning package Weka (http://www.cs.waikato.ac.nz/~ml), for example, no comprehensive evaluation suite is available to date. ROCR is a flexible evaluation package for R (http://www.r-project.org), a statistical language that is widely used in biomedical data analysis. Our tool allows for creating cutoff-parametrized performance curves by freely combining two out of more than 25 performance measures (Table 1). Curves from different cross-validation or bootstrapping runs can be averaged by various methods. Standard deviations, standard errors and box plots are available to summarize the variability across the runs. The parametrization can be visualized by printing cutoff values at the corresponding curve positions, or by coloring the curve according to the cutoff. All components of a performance plot are adjustable using a flexible mechanism for dispatching optional arguments. Despite this flexibility, ROCR is easy to use, with only three commands and reasonable default values for all optional parameters.
|
In the example below, we will briefly introduce ROCR's three commands—prediction, performance and plot—applied to a 10-fold cross-validation set of predictions and corresponding class labels from a study on predicting HIV coreceptor usage from the sequence of the viral envelope protein. After loading the dataset, a prediction object is created from the raw predictions and class labels.
data(ROCR.hiv)pred <- prediction(
ROCR.hiv$hiv.svm$predictions,
ROCR.hiv$hiv.svm$labels)
Performance measures or combinations thereof are computed by invoking the performance method on this prediction object. The resulting performance object can be visualized using the method plot. For example, an ROC curve that trades off the rate of true positives against the rate of false positives is obtained as follows:
perf <- performance(pred, "tpr", "fpr")plot(perf, avg="threshold",
spread.estimate="boxplot")
The optional parameter avg selects a particular form of performance curve averaging across the validation runs; the visualization of curve variability is determined with the parameter spread.estimate.
Issuing demo(ROCR) starts a demonstration of further graphical capabilities of ROCR. The command help(package=ROCR) points to the available help pages. In particular, a complete list of available performance measures can be obtained via help(performance). A reference manual can be downloaded from the ROCR website.
In conclusion, ROCR is a comprehensive tool for evaluating scoring classifiers and producing publication-quality figures. It allows for studying the intricacies inherent to many biological datasets and their implications on classifier performance.
|
|
Acknowledgments |
|---|
Work at MPI supported by EU NoE BioSapiens (LSHG-CT-2003-503265).
Conflict of Interest: none declared.
Received on March 10, 2005; revised on June 1, 2005; accepted on August 9, 2005
|
REFERENCES |
|---|
|
TOPAbstractREFERENCES |
|---|
Baldi, P. and Brunak, S. Bioinformatics: The Machine Learning Approach, (2001) , Cambridge, MA MIT Press.
Beerenwinkel, N., et al. (2003) Geno2pheno: estimating phenotypic drug resistance from HIV-1 genotypes. Nucleic Acids Res., 31, 3850–3855
Beerenwinkel, N., et al. (2002) Diversity and complexity of HIV-1 drug resistance: a bioinformatics approach to predicting phenotype from genotype. Proc. Natl Acad. Sci. USA, 99, 8271–8276
Technical Report HPL-2003-4 Fawcett, T. (2004) ROC graphs: notes and practical considerations for researchers. , Palo Alto, CA HP Labs.
Sing, T., Beerenwinkel, N., Lengauer, T. (2004) Learning mixtures of localized rules by maximizing the area under the ROC curve. Proceedings of the 1st International Workshop on ROC Analysis in Artificial Intelligence , Valencia, Spain , pp. 89–96.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  S. S. F. Lee, L. Sun, R. Kustra, and S. B. Bull EM-random forest and new measures of variable importance for multi-locus quantitative trait linkage analysis Bioinformatics, July 15, 2008; 24(14): 1603 - 1610. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. D. Ward and H. J. Bussemaker Predicting functional transcription factor binding through alignment-free and affinity-based analysis of orthologous promoter sequences Bioinformatics, July 1, 2008; 24(13): i165 - i171. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. T. Dittrich, G. W. Klau, A. Rosenwald, T. Dandekar, and T. Muller Identifying functional modules in protein-protein interaction networks: an integrated exact approach Bioinformatics, July 1, 2008; 24(13): i223 - i231. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. F. W. Saunders and B. Kobe The Predikin webserver: improved prediction of protein kinase peptide specificity using structural information Nucleic Acids Res., July 1, 2008; 36(suppl_2): W286 - W290. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 X.-J. Ma, R. Salunga, S. Dahiya, W. Wang, E. Carney, V. Durbecq, A. Harris, P. Goss, C. Sotiriou, M. Erlander, et al. A Five-Gene Molecular Grade Index and HOXB13:IL17BR Are Complementary Prognostic Factors in Early Stage Breast Cancer Clin. Cancer Res., May 1, 2008; 14(9): 2601 - 2608. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Kathiresan, O. Melander, D. Anevski, C. Guiducci, N. P. Burtt, C. Roos, J. N. Hirschhorn, G. Berglund, B. Hedblad, L. Groop, et al. Polymorphisms Associated with Cholesterol and Risk of Cardiovascular Events N. Engl. J. Med., March 20, 2008; 358(12): 1240 - 1249. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Bernauer, R. P. Bahadur, F. Rodier, J. Janin, and A. Poupon DiMoVo: a Voronoi tessellation-based method for discriminating crystallographic and biological protein-protein interactions Bioinformatics, March 1, 2008; 24(5): 652 - 658. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Holyoake, P. O'Sullivan, R. Pollock, T. Best, J. Watanabe, Y. Kajita, Y. Matsui, M. Ito, H. Nishiyama, N. Kerr, et al. Development of a Multiplex RNA Urine Test for the Detection and Stratification of Transitional Cell Carcinoma of the Bladder Clin. Cancer Res., February 1, 2008; 14(3): 742 - 749. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
I. Sommer, O. Muller, F. S. Domingues, O. Sander, J. Weickert, and T. Lengauer Moment invariants as shape recognition technique for comparing protein binding sites Bioinformatics, December 1, 2007; 23(23): 3139 - 3146. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Robbins, A. K. Aragaki, C. Kooperberg, N. Watts, J. Wactawski-Wende, R. D. Jackson, M. S. LeBoff, C. E. Lewis, Z. Chen, M. L. Stefanick, et al. Factors Associated With 5-Year Risk of Hip Fracture in Postmenopausal Women JAMA, November 28, 2007; 298(20): 2389 - 2398. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. J. Vogel, D. P. Atchley, J. Erlichman, K. R. Broglio, K. J. Ready, V. Valero, C. I. Amos, G. N. Hortobagyi, K. H. Lu, and B. Arun BRCA1 and BRCA2 Genetic Testing in Hispanic Patients: Mutation Prevalence and Evaluation of the BRCAPRO Risk Assessment Model J. Clin. Oncol., October 10, 2007; 25(29): 4635 - 4641. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Pandini, G. Mauri, A. Bordogna, and L. Bonati Detecting similarities among distant homologous proteins by comparison of domain flexibilities Protein Eng. Des. Sel., June 30, 2007; (2007) gzm021v2. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. D. Fjell, R. E.W. Hancock, and A. Cherkasov AMPer: a database and an automated discovery tool for antimicrobial peptides Bioinformatics, May 1, 2007; 23(9): 1148 - 1155. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
X. Guo, R. Liu, C. D. Shriver, H. Hu, and M. N. Liebman Assessing semantic similarity measures for the characterization of human regulatory pathways Bioinformatics, April 15, 2006; 22(8): 967 - 973. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||