DM: Re: Why is Singular Vector Decomposition for OLS?

[Wray Buntine]

>  > Since SVD is used widely for OLS I guess it has clear advantages 
>over
>  > plain Inverse(X'X) X'Y.  Can anyone tell me about it?  Any good 
>books
>  > or references on the motivation for SVD and application of other 
>matrix
>  > decomposition in statistics?
>  
>  The main reason to use SVD for OLS is for diagnostics on
>  ill-conditioning.  SVD is less accurate and much slower than a QR
>  decomposition.

Warren Seale sums up the reasons.  Here's some additional detail.

Techniques for overfitting (a common one used is cross validation) can
be used so that you can be more creative in fitting higher-order
polynomials or fitting multivariate data.  The really small
eigenvectors in the SVD are "smoothed up" to higher values, or simply
removed, so that when you solve the matrix equation, things don't blow
up.  The math. here gets a bit tricky but it lets you fit a 50 degree
polynomial to a mere 20 data points, something every introductory
statistics class tells you is impossible!  With a usual inverse, it is
impossible, but with an SVD and judicious smoothing it works fine.  It
also lets you fit multivariate data more creatively to avoid known
traps such as "all subsets regression," and means that linear
regression handles complex non-linear curves just fine.  I gave a few
examples at a tutorial for MaxEnt 1995:
        http://www.ultimode.com/~wray/MaxEnt95/
but unfortunately the slides for this component aren't online.

Similar math. can be used in smoothing wavelets, or any other complex
basis function where the smoothing is done for "noise reduction" (see
recent work by Donoho of Stanford).  Online work can be found at:
        
http://playfair.stanford.edu/research/subjects/wave_sig_pro.html

Wray Buntine                                   
VP R&D
Ultimode Systems, LLC                   Phone:  (510) 548 8978
2560 Bancroft Way #213,                 Email:  wray@ultimode.com
Berkeley, CA 94704                      http://WWW.Ultimode.com/~wray/