More Cross Validation (14.0)

Here are the cross validation results for 18 models which various combinations of 3 algorithms, 2 time periods (1 to 12 months and 13 to 24 months), 2 Y variables (1 month and 12 month) and whether PCA was used in pre-processing.  The main point is to illustrate lack of consistency from one year to the next.  

Variation	Values
Model type	GBM (Stochastic Gradient Boosting), SVM, MARS (Multivariate Adaptive Regression Spline)
Time Period	Months 1-12 (essentially 2003) cross-validated or 13-24 (2004)
Y Variable	1 Month return or 12 month return
PreProcess	With or without PCA

Since PCA did not help much, I only performed it for the 1 month Y variable. Below there are 5 panels: LongEx is the average return of the 50 stocks with highest predicted return minus the average of all stocks. ShortEX is the average return of the 50 stocks with the lowest predicted return minus the average of all stocks. Hedge is the LongEx-ShortEX. R2 and RMSE are the r-square and root mean square error. I think of LongEx, ShortEX and Hedge as measures of how well the models predict the right, left and both tails. R2 and RMSE measure how well all observations are predicted.

Most importantly it appears the models produce good LongExresults for the first 12 month period and poor for the second. That is reversed for ShortEx. GBM produces ok results accross the board.

## $LongEx
##          Y1M_Mos1_12 Y1M_Mos13_24 Y12M_Mos1_12 Y12M_Mos13_24
## GBM             2.51         0.29         2.26          0.53
## MARS            3.18        -0.67         3.29         -1.19
## SVM             1.19         0.07         1.57          0.85
## GBMwPCA         1.48         0.99           NA            NA
## MARSwPCA        1.36         0.69           NA            NA
## SVMwPCA         1.21         0.14           NA            NA
## 
## $ShortEX
##          Y1M_Mos1_12 Y1M_Mos13_24 Y12M_Mos1_12 Y12M_Mos13_24
## GBM            -0.58         1.88        -0.84          1.89
## MARS           -0.36         1.70         9.57          4.10
## SVM             0.63         0.02        -0.15         -0.43
## GBMwPCA        -1.22         0.64           NA            NA
## MARSwPCA       -0.56        -0.30           NA            NA
## SVMwPCA         0.27        -0.26           NA            NA
## 
## $Hedge
##          Y1M_Mos1_12 Y1M_Mos13_24 Y12M_Mos1_12 Y12M_Mos13_24
## GBM             1.32         2.06         1.20          1.58
## MARS            2.82         0.19         8.59          0.11
## SVM             1.39        -0.37         1.04          0.13
## GBMwPCA        -0.02         1.27           NA            NA
## MARSwPCA        0.64         0.40           NA            NA
## SVMwPCA         0.49        -0.12           NA            NA
## 
## $R2
##          Y1M_Mos1_12 Y1M_Mos13_24 Y12M_Mos1_12 Y12M_Mos13_24
## GBM             0.40         0.16         0.31          0.18
## MARS            0.14         0.04         0.13          0.41
## SVM             0.25         0.13         0.15          0.12
## GBMwPCA         0.23         0.08           NA            NA
## MARSwPCA        0.00         0.00           NA            NA
## SVMwPCA         0.18         0.09           NA            NA
## 
## $RMSE
##          Y1M_Mos1_12 Y1M_Mos13_24 Y12M_Mos1_12 Y12M_Mos13_24
## GBM            11.81        11.04        20.07         12.85
## MARS           52.18        11.46        63.97         32.60
## SVM            11.56        10.87        19.90         12.75
## GBMwPCA        11.73        10.98           NA            NA
## MARSwPCA       36.65        92.58           NA            NA
## SVMwPCA        11.48        10.80           NA            NA