Bioinformatics Vol. 19 no. 10 2003
Pages 1183-1193
© 2003 Oxford University Press
Statistical evaluation of the Predictive Toxicology Challenge 2000–2001
1 Department of Computer Science,
PO Box 26 (Teollisuuskatu 23), FIN-00014 University of Helsinki,
Finland
2 Oxford University Computing Laboratory,
Wolfson Building, Parks Road, Oxford OX1 3QD, UK
3 Department of Computer Science,
Penglais, Aberystwyth, Ceredigion SY23 3DB, Wales, UK
4 Institut Für Informatik,
Albert-Ludwigs-Universität Freiburg, Georges-Köhler-Allee, Gebäude 079,
D-79110 Freiburg i. Br. Germany
5 Institute for Computer Science,
Technical University of Munich, Boltzmannstr. 3, D-85748 Garching b. Muenchen,
Germany
Received on July 30, 2002
; revised on November 7, 2002
; accepted on November 11, 2002
Motivation: The development of in silico models to predict chemical carcinogenesis from molecular structure would help greatly to prevent environmentally caused cancers. The Predictive Toxicology Challenge (PTC) competition was organized to test the state-of-the-art in applying machine learning to form such predictive models.
Results: Fourteen machine learning groups generated 111 models. The use of Receiver Operating Characteristic (ROC) space allowed the models to be uniformly compared regardless of the error cost function. We developed a statistical method to test if a model performs significantly better than random in ROC space. Using this test as criteria five models performed better than random guessing at a significance level p of 0.05 (not corrected for multiple testing). Statistically the best predictor was the Viniti model for female mice, with p value below 0.002. The toxicologically most interesting models were Leuven2 for male mice, and Kwansei for female rats. These models performed well in the statistical analysis and they are in the middle of ROC space, i.e. distant from extreme cost assumptions. These predictive models were also independently judged by domain experts to be among the three most interesting, and are believed to include a small but significant amount of empirically learned toxicological knowledge.
Availability: PTC details and data can be found at: http://www.predictive-toxicology.org/ptc/
Contact: hannu.toivonen{at}cs.helsinki.fi