A Geometric Approach to Support Vector Regression
Jinbo Bi and Kristin Bennett
Department of Mathematical Sciences
Rensselaer Polytechnic Institute
We develop an intuitive geometric framework for support vector
regression (SVR). By examining when epsilon-tubes exist, we show that
SVR can be regarded as a classification problem in the dual space.
Hard and soft epsilon-tubes are constructed by separating the convex or
reduced convex hulls respectively of the training data with the
response variable shifted up and down by epsilon. A novel SVR model is
proposed based on choosing the max-margin plane between the two
shifted datasets. Maximizing the margin corresponds to shrinking
the effective epsilon-tube. In the proposed approach, the
effects of the choices of all parameters become clear
geometrically. The kernelized model corresponds to separating the
convex or reduced convex hulls in feature space. Generalization
bounds for classification can be extended to characterize the
generalization performance of the proposed approach. We propose a
simple iterative nearest-point algorithm that can be directly
applied to the reduced convex hull case in order to construct soft
epsilon-tubes. Computational comparisons with other SVR formulations are
paper has been accepted by Neurocomputing, special issue on support vector machines.
- CPLEX programs
In this paper, we compared the three support vector regression algorithms, RH-SVR (the proposed approach), C-SVR (the classic SVR algorithm) and nu-SVR. They
were implemented in Unix C++ using the commercial optimization software CPLEX 6.6. The
codes can be available under request (firstname.lastname@example.org). An
appropriate version of CPLEX is required to run the programs.
- NPA package
The proposed geometric approach can be solved using a
faster iterative algorithm called the nearest point algorithm (NPA). A preliminary solver has
been written using Visual C++ 6.0 under Windows 2000. This implementation was generated as
one of the course projects for Operating Systems. The project report can be found here. The
package can be downloaded here
for non-commercial uses and the author will not take any
responsibility for problems encountered when using the package. An updated and improved
version of the NPA program may be available under request.
Contact Jinbo Bi (email@example.com) for information about this page.