A Survey of Ridge Regression for Improvement Over Ordinary Least Squares

Multicollinearity may be a possible cause in case of study with two or more explanatory variables. In the presence of multicollinearity, the design matrix becomes nearly singular and hence X and the corresponding XX are not of full rank. In this situation, the Ordinary Least Square (OLS) estimate cannot be obtained. Thus, adequate attention is required to be given on the presence of multicollinearity in the data. In this survey only ridge regression is discussed as a solution to the problem of multicollinearity. Hoerl and Kennard proposed the technique of ridge regression that has become a popular tool for data analysis faced with the problem of a high degree of multicollinearity. They have suggested adding a small positive quantity in the diagonal elements of the design matrix, XX before inverting it. In other words, they proposed R X X k I X Y     1 ) ( ˆ in place of ˆ  (XX)1 XY,where R ˆ and ˆ are the ridge and OLS estimates of the parameter vector  respectively. Though the ridge estimate is biased, it has a smaller mean squared error than OLS estimator. A critical appraisal is also given on the choice of biasing parameter, in addition to its properties, its relation with other estimators and Bayesian and non-Bayesian interpretations