Sparse estimation based on a validation criterion
Abstract:
A sparse estimator with close ties with the LASSO (least absolute shrinkage and selection operator) is analysed. The basic idea of the estimator is to relax the least-squares cost function to what the least-squares method would achieve on validation data and then use this as a constraint in the minimization of the $\ell_1$-norm of the parameter vector. In a linear regression framework, exact conditions are established for when the estimator is consistent in probability and when it possesses sparseness. By adding a re-estimation step, where least-squares is used to re-estimate the non-zero elements of the parameter vector, the so called Oracle property can be obtained, i.e. the estimator achieves the asymptotic Cramér-Rao lower bound corresponding to when it is known which regressors are active. The method is shown to perform favorably compared to other methods on a simulation example.