Bioinformatics Advance Access originally published online on September 1, 2008
Bioinformatics 2008 24(20):2420; doi:10.1093/bioinformatics/btn456
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Comments on ‘On correcting the overestimation of the permutation-based false discovery rate estimator’
Department of Clinical Sciences, The Simmons Comprehensive Cancer Center, University of Texas Southwestern Medical Center, Dallas, TX 75390, USA
 |
ABSTRACT |
|---|
 TOP ABSTRACT REFERENCES |
|---|
Contact: Yang.xie{at}utsouthwestern.edu
We thank Jiao and Zhang for pointing out a typo in the FDR formula, Equation (2) in our original paper (Xie et al., 2005). We have sent the correction to the journal. However, we found that key statements and results in the manuscript by Jiao and Zhang are misleading at best. We would like to address them in this letter.
- Jiao and Zhang's statement ‘This mistake [the typo for true FDR in the formula (2)] will affect the evaluation of their (Xie et al., 2005) proposed FDR estimator’ is false. All the results in Xie et al. were obtained by using the correct formula. Therefore, despite the key proclamation by Jiao and Zhang, all the results are valid. Jiao et al. did not provide any evidence that the results in the 2005 paper in question were based on the incorrect true FDR formula. On the contrary, their results are consistent with Xie et al.'s results. The authors' statement ‘With the correct FDR formula, we show the method of (Xie et al., 2005) always gives biased estimation of FDR’ is misleading. The performance and the estimation bias of the method presented in Xie et al. have been explored in detail and discussed in their original paper; please refer to Table 2 and Figures 1–4, results and discussion in Xie et al. (2005). Those are not new findings that can be attributed to the application of the corrected formula by Jiao and Zhang.
- The purpose of Xie et al. (2005) FDR estimation method is to serve as a better criterion (compared to the standard method) for evaluating various statistical methods. It was not proposed to be an unbiased estimator of true FDR. From the results of both Xie et al. (2005) and Jiao and Zhang (2008), we can clearly see that Xie et al.'s method could serve as a fair criterion for comparing mean statistics, SAM statistics and t-statistics. Thus, the conclusions in the original paper by Xie et al. (2005) are valid. However, Figure 2 in Jiao and Zhang's paper clearly demonstrates that Jiao and Zhang's method underestimates FDR for mean statistics, but overestimate it for t-statistics. In this case, Jiao and Zhang's FDR estimators favor mean statistic, and thus the results directly contradict to their own statement in the introduction, ‘Hence, the estimated FDR will serve as the criterion to compare different methods when controlling the error rates’.
- Jiao and Zhang's statement ‘Xie et al. (2005) implicitly uses
’ is false. Xie et al. (2005) explicitly stated this point in the last sentence of Section 2.3.
- In Jiao and Zhang's paper, the comparisons between Xie et al. method and Jiao and Zhang's method were incomplete and misleading. Figures 1 and 2 present the results from the same simulation study, and the only difference is the range of the x-axis (number of significant genes). By splitting one figure into two figures, the authors excluded from the comparisons a large number of genes, from 200th to 500th, arguably the most important range needed to be compared, since there are 400 true DE genes in this simulation. Also notice that the FDR ranges in Figures 1 and 2 are small.
- Except for the biased inference from the results illustrated in Figures 1 and 2, the authors do not offer any strong evidence in support of their conclusion that ‘...our (Jiao and Zhang's) estimator gives more accurate estimation’, even in a simulation study. In real data example, 63 genes have been confirmed by real-time PCR (Xie et al., 2005), but Jiao and Zhang (2008) used only 38 genes and gave no explanation on how these 38 genes were selected.
- The method 1 ‘proposed’ by Jiao and Zhang is not new; it has been used and compared in Xie's original Bioinformatics paper (2005) for both simulated and real data examples.
Conflict of Interest: none declared.
|
FOOTNOTES |
|---|
Associate Editor: Alex Bateman
Received on July 10, 2008; revised on August 20, 2008; accepted on August 21, 2008
|
REFERENCES |
|---|
|
TOPABSTRACTREFERENCES |
|---|
Jiao S, Zhang S. On correcting the overestimation of the permutation based false discovery rate estimator. Bioinformatics (2008) 24:1655–1661.
Xie Y, et al. A note on using permutation-based false discovery rate estimates to compare different analysis methods for microarray data. Bioinformatics (2005) 21:4280–4288.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||