Bioinformatics

Bioinformatics Advance Access originally published online on January 5, 2008
Bioinformatics 2008 24(3):412-419; doi:10.1093/bioinformatics/btm579

This Article Abstract FREE Full Text (Print PDF) All Versions of this Article: 24/3/412    most recent btm579v1 Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to citation manager Search for citing articles in: ISI Web of Science (2) Request Permissions Google Scholar Articles by Wang, L. Articles by Zou, H. Search for Related Content PubMed PubMed Citation Articles by Wang, L. Articles by Zou, H. Social Bookmarking          What's this?

© The Author 2008. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org

Hybrid huberized support vector machines for microarray classification and gene selection

Li Wang ¹, Ji Zhu ^2,* and Hui Zou ³

¹Ross School of Business, ²Department of Statistics, University of Michigan, Ann Arbor, MI 48109 and ³School of Statistics, University of Minnesota, Minneapolis, MN 55455, USA

*To whom correspondence should be addressed.

	ABSTRACT

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 

Motivation: The standard L₂-norm support vector machine (SVM) is a widely used tool for microarray classification. Previous studies have demonstrated its superior performance in terms of classification accuracy. However, a major limitation of the SVM is that it cannot automatically select relevant genes for the classification. The L₁-norm SVM is a variant of the standard L₂-norm SVM, that constrains the L₁-norm of the fitted coefficients. Due to the singularity of the L₁-norm, the L₁-norm SVM has the property of automatically selecting relevant genes. On the other hand, the L₁-norm SVM has two drawbacks: (1) the number of selected genes is upper bounded by the size of the training data; (2) when there are several highly correlated genes, the L₁-norm SVM tends to pick only a few of them, and remove the rest.

Results: We propose a hybrid huberized support vector machine (HHSVM). The HHSVM combines the huberized hinge loss function and the elastic-net penalty. By doing so, the HHSVM performs automatic gene selection in a way similar to the L₁-norm SVM. In addition, the HHSVM encourages highly correlated genes to be selected (or removed) together. We also develop an efficient algorithm to compute the entire solution path of the HHSVM. Numerical results indicate that the HHSVM tends to provide better variable selection results than the L₁-norm SVM, especially when variables are highly correlated.

Availability: R code are available at http://www.stat.lsa.umich.edu/~jizhu/code/hhsvm/

Contact: jizhu{at}umich.edu

Supplementary information: Supplementary data are available at Bioinformatics online.

	1 INTRODUCTION

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 
The DNA microarray technology is a powerful tool for biological and medical research. It can detect thousands of gene expression levels simultaneously, providing a wealth of information. On the other hand, however, microarray datasets usually contain only a small number of samples. These characteristics pose great challenges for sample classification and gene selection. The support vector machine (SVM) is one of the most effective methods for microarray classification (Guyon et al., 2002; Mukherjee et al., 2000; Ramaswamy et al., 2001); however, a major limitation of the SVM is that it cannot perform automatic gene selection. Since in microarray analysis, researchers are often interested in identifying informative genes, it is desirable to have a tool that can achieve both classification and gene selection simultaneously.

Guyon et al. (2002) proposed therecursive feature elimination (RFE) method for the SVM. The method, called the SVM-RFE, recursively eliminates irrelevant genes. At each step, the SVM-RFE trains for a SVM classifier, ranks the genes according to some score function and eliminates one or more genes with the lowest ranking scores. This process is repeated until classification accuracy starts to degrade. The SVM-RFE method is computationally intensive, especially for microarray datasets, which usually has a large number of genes. Bradley and Mangasarian (1998) proposed the L₁-norm SVM, which can automatically select genes via the L₁-norm regularization. The L₁-norm SVM, however, has two limitations:

1. The number of selected genes is upper bounded by the sample size. Therefore, when the number of relevant genes exceeds the sample size, the L₁-norm SVM can only discover a portion of them.
   
2. For the highly correlated and relevant genes, the L₁-norm SVM tends to pick only one or a few of them.
   

In this article, we propose an HHSVM for microarray classification. The HHSVM has the form of ‘loss’ + ‘penalty’, where it uses the huberized hinge function to measure the loss and the elastic-net penalty (Zou and Hastie, 2005) to control the complexity of the model. The HHSVM has several major benefits:

• Similar to the L₁-norm SVM, it automatically selects genes.
  
• The number of selected genes is no longer upper bounded by the sample size.
  
• When genes are highly correlated, they tend to be selected or removed together, i.e. the grouping effect.
  

Furthermore, inspired by the LAR/LASSO method (Efron et al., 2004) and the general piecewise linear solution path strategy (Rosset and Zhu, 2007), we develop an efficient algorithm, which solves the entire solution path for every possible value of the regularization parameter.

The rest of the article is organized as follows. In Section 2, we describe the HHSVM model. In Section 3, we develop the solution path algorithm. In Section 4, we apply the HHSVM method to simulation and real microarray datasets. We conclude the article with Section 5.

	2 MODEL

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 
2.1 The support vector machine
We briefly introduce the SVM and refer interested readers to (Burges, 1998; Evgeniou et al., 1999; Vapnik, 1995) for detailed tutorials. Let {x₁, x₂, ... , x_n} represent the n input vectors, where x_i ^p, and let {y₁, y₂, ... , y_n} be the corresponding output labels, where y_i {1, – 1}. The SVM finds a hyperplane that separates the two classes of data points by the largest distance:

(1)

(2)

(3)

(4)

where _i (i = 1, ... , n) are slack variables, and C 0 is a tuning parameter that controls the overlap. Denote the solution as and , a new point with input vector x is then assigned with the label .

Many researchers have recognized that the SVM can be equivalently transformed into the ‘loss’ + ‘penalty’ format, i.e.

(5)

where the loss function (1 – ·)₊ = max(1 – ·, 0) is called the hinge loss and the penalty is the L₂-norm of the fitted coefficients. 0 is a regularization parameter, which controls the balance between the ‘loss’ and the ‘penalty’. The same L₂-norm penalty has also been used in the ridge regression (Hoerl and Kennard, 1970) and neural networks. By shrinking the magnitude of the coefficients, the L₂-norm penalty reduces the variance of the estimated coefficients and may result in better prediction accuracy.

2.2 The L₁-norm SVM
Bradley and Mangasarian (1998) proposed to replace the L₂-norm penalty in the standard SVM with the L₁-norm penalty (Tibshirani, 1996):

(6)

Similar to the L₂-norm penalty, the L₁-norm penalty also reduces the variance of the estimates and improves the prediction accuracy. Furthermore, due to the singularity of the L₁-norm function, when ₁ is large enough, it tends to shrink the coefficients of irrelevant variables to exactly zero. When changes, different coefficients are set to zero, hence the L₁-norm penalty allows a kind of automatic continuous variable selection.

2.3 The hybrid huberized SVM
Zou and Hastie (2005) argued that the L₁-norm penalty has two major limitations:

1. The number of variables selected by the L₁-norm penalty is upper bounded by the sample size n. In microarray analysis, we nearly always have the case p >> n, but the L₁-norm SVM can identify at most n relevant genes.
   
2. For highly correlated and relevant variables, the L₁-norm penalty tends to select only one or a few of them. In microarray analysis, genes sharing the same biological pathway tend to have highly correlated expression levels. It is often desirable to identify all, rather than a few, of them if they are related to the underlying biological process.
   

To overcome these limitations, Zou and Hastie (2005) proposed the elastic-net penalty:

which is a hybrid of the L₁-norm and the L₂-norm penalties. The elastic-net penalty retains the variable selection feature of the L₁-norm penalty, and the number of selected variables is no longer bounded by n. Furthermore, the elastic-net penalty tends to provide similar estimated coefficients for highly correlated variables, i.e. the grouping effect; hence, highly correlated variables tend to be selected or removed together.

In this article, we apply the elastic-net penalty to the SVM and propose the HHSVM:

(7)

where ₁, ₂ 0 are regularization parameters. Increasing ₁ tends to eliminate more irrelevant variables; and increasing ₂ makes the ‘grouping effect’ more prominent, which we will illustrate by a theorem at the end of this section.

Notice that instead of using the standard hinge loss function of the SVM, we use the huberized hinge loss function (Rosset and Zhu, 2007) to measure ‘badness-of-fit’:

where 0 is a pre-specified constant.

Figure 1 compares the standard hinge loss function and the huberized hinge loss function. Notice that they have similar shapes: when yf decreases (misclassification), they both increase linearly; when yf is bigger than 1, they are both equal to zero. Therefore, we expect the classification performance of these two functions should be similar to each other. Also notice that, unlike the hinge loss, the huberized hinge loss function is differentiable everywhere. As we will see in the next section, this differentiability can significantly reduce the computational cost for the HHSVM algorithm.

View larger version (10K): [in this window] [in a new window] [Download PowerPoint slide]   Fig. 1. The hinge and the huberized hinge loss functions (with = 2). Note that the Elbow corresponds to the region (1 –; 1); the Left and the Right correspond to the regions (– , 1 – ) and (1,), respectively.

 
Before delving into details for the HHSVM algorithm, we illustrate the ‘grouping effect’ with the following theorem:

THEOREM 1
Let and denote the solution for problem (7). For any pair (j, j'), we have

(8)

If the input vector x_j and x_j' are centered and normalized, then

(9)

where is the sample correlation between x_j and x_{j '}.

Details of the proof are in the Supplementary Material. Theorem 1 suggests that highly correlated variables tend to have similar estimated coefficients; hence, they tend to be selected or removed together when ₂ is sufficiently large.

	3 ALGORITHM

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 
The HHSVM involves two tuning parameters ₁ and ₂. According to our experience, the prediction performance is often more sensitive to ₁, since it has more impact on selecting variables; therefore, the value of ₁ must be chosen more carefully. For parameter tuning, one usually specifies a number of candidates, tests each of them and chooses the best one according to some criterion. However, this trial-and-error approach is computationally expensive and the optimal parameter setting can be easily missed. In this section, we propose an efficient algorithm, which solves the entire solution path for every possible value of ₁ (when ₂ is fixed). The algorithm is based on the fact that the solution () is a piecewise linear function (in a multi-dimensional space) with respect to ₁.

3.1 Problem setup
Since the huberized hinge loss function has different definitions in different regions, for simplicity we define each region as:

• = {i: y_i f (x_i) > 1} (Right)
  
• = {i: 1 – < y_i f (x_i) 1} (Elbow)
  
• = {i: y_i f (x_i) 1 – } (Left)
  

where . We also define the indices for non-zero β_j as the active set :

• = { j : β_j 0, j = 1,2, ... , p} (Active)
  

Since problem (7) is an unconstrained convex optimization problem, for any optimal solution the derivatives of its objective function must be zero. Setting the derivative with respect to β₀ to zero, we get:

(10)

Setting the derivative with respect to β_j (j ) to zero, we get:

(11)

In the linear system (10)–(11), there are || + 1 unknowns and || + 1 equations, where || represents the number of elements in set . Therefore, the solution β₀ and β_j (j ) can be uniquely determined, given that the system is non-singular.

When sets , , and are fixed, the structure of the linear system (10)–(11) is also fixed. Under this condition, β₀ and β_j (j ) are linear functions of ₁, which can be seen from (11). However, as ₁ decreases, sooner or later some of the sets , , and will change. We call this an event. After an ‘event’ occurs the new β₀ and β_j (j ) are still linear functions of ₁, but their derivatives with respect to ₁ will change. Therefore, the entire solution path is piecewise linear in ₁, and between any two consecutive events, β₀ and β_j (j ) change linearly with ₁. Each ‘event’ corresponds to a kink on the piecewise linear solution path.

Our algorithm starts from ₁ = , continuously decreases ₁, solves the solutions along this path, and terminates if ₁ reaches 0. The algorithm provides the solutions on each kink. For any ₁ between two consecutive kinks, the solution can be precisely obtained using linear interpolation. Table 1 shows the outline of the our algorithm.

View this table: [in this window] [in a new window]   Table 1. Outline of the HHSVM algorithm

 
3.2 Initial solution
The algorithm starts from ₁ = . From the objective function of problem (7), we can see that β = 0 at this stage. Therefore, the problem reduces to:

(12)

which involves only one parameter β₀. Since the huberized hinge loss function is convex and differentiable everywhere, this one-variable optimization problem can be easily solved. In fact, one can easily develop an analytical solution for the initial β₀, but due to the lack of space, we skip the details.

Let represent the optimal solution when ₁ = . Hence = (β = 0) and sets , and can be determined using the values of (i = 1, ... , n). Reducing ₁ tends to increase the magnitude of β, and we can find a critical point, denoted as , where exactly one β_j (j = 1, ... , p) joins , i.e. the estimate β_j will become non-zero if ₁ is further reduced.

Since Equation (11) must hold for any j , we can determine the critical point by:

The corresponding variable that joins first can be identified as:

and the sign for is:

Let the superscript k indicate the iteration number and k = 0 for the initial stage. Now, we have ^k = { j*}, , (j = 1, ... , p), , and the ^k, ^k, ^k are determined by (i = 1, ... , n).

We emphasize again that the initial solution can be easily determined here because of the differentiability of the huberized hinge loss function; however, this is not the case for the hinge loss function. Since the hinge loss is not everywhere differentiable, it is difficult to determine the critical point and the corresponding ⁰, ⁰, ⁰ and ⁰ at the initial stage. Zhu et al. (2004) proposed an approach for solving the initial solution of the L₁-norm SVM. The approach was based on linear programming, but it can be computationally intensive, especially when p is large.

3.3 Solution path
The algorithm continuously decreases ₁ until it reaches 0. Let , where ₁ < 0. When ₁ is reduced by a small enough amount, sets , , and do not change, because of the continuity of f(x) with respect to ₁. Therefore, based on the systems (10)–(11), the derivatives of β₀ and β_j (j ) with respect to ₁ can be solved from the following equations:

(13)

(14)

Since there are || + 1 unknowns and || + 1 equations, and (j ) are uniquely determined, given that the system is non-singular. Then when |₁| is sufficiently small, the solution and fitted values are linear in ₁:

If we keep reducing ₁, some of the sets , , and will change. We call this an event, and four types of ‘events’ may occur:

1. A point i reaches the boundary between and .
   
2. A point i reaches the boundary between and .
   
3. A parameter β_j becomes zero (j leaves ).
   
4. A zero-valued parameter β_j becomes non-zero (j joins ).
   

The boundary between and is 1 – and the boundary between and is 1. Therefore, when y_i f(x_i) for a point crosses 1 – or 1, one of the first two ‘events ’ occurs. To determine the step size ₁ for the first ‘event’, for each i or we calculate:

Let _1,1 represent the step size for the first ‘event’. Since ₁ only decreases, _1,1 0 and its value should be determined by:

Similarly, for the second ‘event’ we calculate:

for each i or . And the step size for the second ‘event’ is:

When a non-zero β_j reduces to 0, the third ‘event’ occurs. Therefore, we calculate for each j :

The step size for the third ‘event’ is:

The fourth ‘event’ (a zero-valued β_j becomes non-zero) is a little complicated to determine. For j = 1, ... , p, we define:

which is part of the left-hand side of Equation (11). From (11) and the initial conditions, we know that:

• |C_j| = ₁, sign(C_j) = – sign(β_j), for j ;
  
• |C_j| < ₁, for j .
  

Notice that when | ₁| is sufficiently small, C_j is also a linear function of ₁:

As ₁ decreases, the value for a |C_j| (j ) will first meet the decreasing ₁, after which the corresponding β_j will become non-zero if we further reduce ₁.

Therefore, to determine the step size for the fourth ‘event’, for each j we calculate:

The step size _1,4 for the fourth ‘event’ is:

After solving for _1,1, _1,2, _1,3 and _1,4, the overall step size ₁ can be obtained:

We then update , , and according to the corresponding ‘event’, and also update ₁, β₀, β_j, C_j and the fitted value f(x_i). Finally, let k = k + 1 and the algorithm goes to the next iteration: solving the linear systems (13)–(14) and calculating the step size ₁. This entire process is repeated, until ₁ reaches 0.

Between any two consecutive ‘events’, the solutions are linear in ₁, and after an ‘event’ occurs, the derivative of the solution with respect to ₁ is changed. Therefore, the solution path is piecewise linear in ₁, where each ‘event’ corresponds to a kink on the path. The algorithm provides solutions at these kinks, and for any ₁ between two consecutive kinks the solutions can be calculated precisely via linear interpolation.

3.4 Computational cost
The major computational cost in each iteration comes from solving the linear system (13)–(14). Since this system has || + 1 unknowns, the computational cost seems to be O(||³) for each iteration. However, between any two iterations, only one element is changed for sets , , and , so using inverse updating and downdating the computational cost can be reduced to O(||²). It is difficult to predict the number of iterations. According to our experience, O(min(n,p)) is a reasonable estimate. The heuristic is that the algorithm needs O(n) to move all the points to and O(p) steps to add all the parameters into . Since p is the upper bound for ||, the overall computational cost is upper bounded by O(min(n,p)p²).

	4 NUMERICAL RESULTS

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 
In this section, we use both simulation data and real microarray datasets to illustrate the HHSVM.

4.1 Simulation
The main purpose of the simulation is to demonstrate that when input variables are independent, the L₁-norm SVM and the HHSVM perform similarly, but the HHSVM can have an advantage when the variables are highly correlated, especially at identifying the relevant variables.

We first consider the scenario where all input variables are independent. The ‘+’ class has a normal distribution with mean

and covariance = I_{p x p}. The ‘–’ class has a similar distribution except that

So the Bayes optimal classification rule only depends on x₁, ... , x₁₀, and the Bayes error is about 0.06, independent of the dimension p.

We generated 50 = 25 + 25 training data, each input x_i is a p = 300-dimensional vector. We compared the standard L₂-norm SVM, the L₁-norm SVM, and the HHSVM. We used 50 validation data to select the tuning parameters for each method, then applied the selected models to a separate 10 000 testing dataset. Each experiment was repeated 100 times. The means of the prediction errors and the corresponding standard errors (in parentheses) are summarized in Table 2. As we can see, the prediction errors of the L₁-norm SVM and the HHSVM are similar and both are better than that of the L₂-norm SVM. This is due to the fact that the L₂-norm SVM uses all input variables, and its prediction accuracy is ‘polluted’ by the noise variables.

View this table: [in this window] [in a new window]   Table 2. Comparison of test errors

 
Besides the prediction error, we also compared the selected variables of the L₁-norm SVM and the HHSVM (the L₂-norm SVM keeps all input variables). In particular, we consider q_signal = number of selected relevant variables, and q_noise = number of selected noise variables. The results are in Table 3. Again, we see that the L₁-norm SVM and the HHSVM perform similarly; both are able to identify the relevant variables and remove most of the irrelevant variables.

View this table: [in this window] [in a new window]   Table 3. Comparison of variable selection

 
Now we consider the scenario when the relevant variables are correlated. Similar as the independent scenario, the ‘+’ class has a normal distribution, with mean

and covariance

where the diagonal elements of * are 1 and the off-diagonal elements are all equal to = 0.8. The ‘–’ class has a similar distribution except that

So the Bayes optimal classification rule depends on x₁, ... , x₁₀, which are highly correlated. The Bayes error is 0.13, independent of the dimension p.

Again, we considered n = 25 + 25 and p = 300. Each experiment was repeated 100 times. The results for the prediction errors are shown in Table 2. Now the performance of the HHSVM is slightly better than that of the L₁-norm SVM, after taking into account the standard error. The right part of Table 3 compares the variables selected by the L₁-norm SVM and the HHSVM, which sheds some light on what happened. Both the L₁-norm SVM and the HHSVM are able to identify relevant variables. However, when the relevant variables are highly correlated, the L₁-norm SVM tends to keep only a small subset of the relevant variables, and overlook the others, while the HHSVM tends to identify all of them, due to the grouping effect. Both methods seem to work well in removing irrelevant variables.

4.2 Real Data Analysis
The first dataset we considered is the colon cancer dataset in Alon and Barkai (1999). This dataset consists of 62 samples (40 colon cancer tumors and 22 normal tissues). Each sample consists of p = 2000 genes. We randomly split the samples into the training and the test sets for 100 times; for each split, 42 samples (27 cancer samples and 15 normal tissues) were used for training and the rest 20 samples (13 cancer samples and 7 normal tissues) were for testing. For each split, we applied four methods, the standard L₂-norm SVM, the L₁-norm SVM, the SVM-RFE and the HHSVM, to the training data. Tuning parameters were chosen using 10-fold cross-validation on the training data, and the chosen models were evaluated on the test data. For the standard L₂-norm SVM, we tried different kernels and found the linear kernel has the best performance. For the SVM-RFE, we used the same strategy as in Guyon et al. (2002), i.e. eliminating 10% ofthe remaining genes in each iteration. The results are summarized in the upper part of Table 4. We can see that the HHSVM seemed to have a slightly better classification accuracy than other methods. However, we note that this is not necessarily conclusive, for the sample size is small.

View this table: [in this window] [in a new window]   Table 4. Results on 100 randomsplits of the original datasets: the upper part is for the colon cancer dataset, and the lower part is for the lung cancer dataset

 
Tables 5–6 summarize the genes that were ‘frequently’ selected by the L₁-norm SVM and the HHSVM. As we can see, only 5 genes were selected for more than 50 times by the L₁-norm SVM, while 40 genes were selected for more than 50 times by the HHSVM. The selection frequency implies that these genes may have certain power in differentiating the colon cancer samples from the normal tissues, however, the L₁-norm SVM was not always able to identify them. Figure 2 shows the number of selected genes for each selection frequency out of the 100 random splits. Again, we can see that the HHSVM is more stable than the L₁-norm SVM in terms of selecting genes, i.e. over the 100 random splits, for important genes, the HHSVM selects them more frequently than the L₁-norm SVM (lower part of Fig. 2), while for unimportant genes, the HHSVM selects them less frequently (upper part of Fig. 2).

View larger version (14K): [in this window] [in a new window] [Download PowerPoint slide]   Fig. 2. The upper part shows the number of selected genes versus the selection frequency for the L1-norm SVM (left) and the HHSVM (right) on 100 random splits of the colon cancer dataset. The lower part ‘zooms in’ to the region where the ‘selection frequency’ is bigger than 50 (the left panel is for the L1-norm SVM and the right panel is for the HHSVM).

 

View this table: [in this window] [in a new window]   Table 5. The 32 most frequently selected genes by the L1-SVM from the colon cancer dataset

 

View this table: [in this window] [in a new window]   Table 6. The 100 most frequently selected genes by the HHSVM from the colon cancer dataset

 
To assess the stability of prediction, we also recorded the frequency that each sample, as a test observation, was correctly classified. For example, if a sample appears in 70 test sets among the 100 random splits, and out of the 70 predictions, the sample was correctly classified for 60 times, then we recorded 60/70 for this sample. The results are shown in Figure 3. As we can see, for most samples, both the L₁-norm SVM and the HHSVM classified them correctly for most of the random splits, but there are also some samples where both methods tended to misclassify. Overall, the HHSVM seems to be slightly more stable than the L₁-norm SVM in terms of prediction.

View larger version (6K): [in this window] [in a new window] [Download PowerPoint slide]   Fig. 3. Number of samples vs the frequency that a sample (as a test observation) was correctly classified on 100 random splits of the colon cancer dataset. The left panel is for the L1-norm SVM, and the right panel is for the HHSVM.

 
The second dataset we considered is a more recent lung cancer dataset (Potti et al., 2006), which consists of 198 samples (54 cancer tumors and 144 normal tissues). Each sample consists of expression measurements on 22 215 genes. We randomly splitted the data into training and test sets, with sample sizes 137 (37 cancer samples and 100 normal tissues) and 61, respectively. We repeated it 100 times. The lower part of Table 4 summarizes the results. The gene selection behavior of the L₁-norm SVM and the HHSVM for the lung cancer dataset are similar to those for the colon cancer dataset. Due to the lack of space, we skip the results.

	5 CONCLUSION

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 
In this article, we have proposed the HHSVM for microarray classification and gene selection. The HHSVM uses the huberized hinge loss function to measure the ‘badness-of-fit’ and the elastic-net penalty to control the model complexity. The huberized hinge loss function allows efficient computation for calculating the entire solution path. The elastic-net penalty allows automatic flexible variable selection. We have presented some evidence that the new method tends to select more relevant variables (than the L₁-norm SVM method), especially when variables are highly correlated.

	ACKNOWLEDGEMENTS

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 
We thank the Associate Editor, David Rocke, and two referees for their feedback which led us to improve the article, particularly the numerical result section. We also thank Saharon Rosset for helpful comments. L.W. and J.Z. are partially supported by grant DMS-0505432 and DMS-0705532 from the National Science Foundation.

Conflict of Interest: none declared.

	FOOTNOTES

 
Associate Editor: David Rocke

Received on June 15, 2007; revised on October 9, 2007; accepted on November 18, 2007

	REFERENCES

TOP ABSTRACT 1 INTRODUCTION 2 MODEL 3 ALGORITHM 4 NUMERICAL RESULTS 5 CONCLUSION ACKNOWLEDGEMENTS REFERENCES

 

Alon U, Barkai N. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon cancer tissues probed by oligonucleotide arrays. Proc. Natl Acad. Sci. USA (1999) 96:6745–6750.[Abstract/Free Full Text]

Bradley P, Mangasarian O. Feature selection via concave minimization and support vector machines. (1998) Proceedings of the 15th International Conference on Machine Learning.

Burges C. A tutorial on support vector machines for pattern recognition. Data Mining Knowl. Discov (1998) 2:121–167.[CrossRef]

Efron B, et al. Least angle regression. Ann. Stat (2004) 32:407–499.[CrossRef]

Evgeniou T, et al. Regularization networks and support vector machines. In: Advances in Large Margin Classifiers.—Smola A, et al, eds. (1999) MIT Press.

Guyon I, et al. Gene selection for cancer classification using support vector machines. Mach. Learn (2002) 46:389–422.[CrossRef]

Hoerl A, Kennard R. Ridge regression: biased estimation for nonorthogonal problems. Technometrics (1970) 12:55–67.[CrossRef][Web of Science]

Mukherjee S, et al. Support vector machine classification of microarray data. Technical report (2000) Massachusetts Institute of Technology. Artificial Intelligence Laboratory.

Potti A, et al. A genomic strategy to refine prognosis in early-stage non-small-cell lung cancer. N. Eng. J. Med (2006) 355:570–580.[Abstract/Free Full Text]

Ramaswamy S, et al. Multiclass cancer diagnosis using tumor gene expression signatures. Proc. Natl Acad. Sci. USA (2001) 98:15149–15154.[Abstract/Free Full Text]

Rosset S, Zhu J. Piecewise linear regularized solution paths. Ann. Stat (2007) 35:1012–1030.[CrossRef]

Tibshirani R. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B (1996) 58:267–288.

Vapnik V. The Nature of Statistical Learning Theory. (1995) New York: Springer-Verlag.

Zhu J, et al. 1-norm SVMs. (2004) Proceedings of the Neural Information Processing Systems.

Zou H, Hastie T. Regularization and variable selection via the elastic net. J. R. Stat. Soc. B (2005) 67:301–320.[CrossRef]

CiteULike    Connotea    Del.icio.us    What's this?

This Article Abstract FREE Full Text (Print PDF) All Versions of this Article: 24/3/412    most recent btm579v1 Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to citation manager Search for citing articles in: ISI Web of Science (2) Request Permissions Google Scholar Articles by Wang, L. Articles by Zou, H. Search for Related Content PubMed PubMed Citation Articles by Wang, L. Articles by Zou, H. Social Bookmarking          What's this?

Online ISSN 1460-2059 - Print ISSN 1367-4803

Copyright © 2009 Oxford University Press

Oxford Journals Oxford University Press

• Site Map
• Privacy Policy
• Frequently Asked Questions

Other Oxford University Press sites: Oxford University Press Oxford Journals China Oxford Journals Japan American National Biography Booksellers' Information Service Children's Fiction and Poetry Children's Reference Corporate & Special Sales Dictionaries Dictionary of National Biography Digital Reference English Language Teaching Higher Education Textbooks Humanities International Education Unit Law Medicine Music Online Products Oxford English Dictionary Reference Rights and Permissions Science School Books Social Sciences Very Short Introductions World's Classics