In this post, we look at out-of-sample performance for some gradient boosted machine (GBM) models. The purpose here is not an exhaustive analysis. We are only hoping to get an indication that our efforts might bear fruit.
Each model we build uses predictor (x) data from one month (e.g. Dec 2002) and response (y) data from the subsequent month (e.g. Jan 2003). We refer to a model here by xmonth/ymonth (e.g, Dec02/Jan03) to indicate the months of the x and y data. The y variable is a company’s excess return defined as the return of company for the month less the average return of all companies for that month.
The first model is Dec02/Jan03. The first set of predictions was for Jan 2004 created by averaging the 12 predicted returns for each company by feeding in the Dec03 x data into the Dec02/Jan03 through the Nov03/Dec03 models. The last prediction is for Oct 2015. We have 142 months of predicted return, a bit shy of 12 years.
For each month with a predicted return, we calculate three values: LongEx, ShortEx and Hedge. Respectively, these roughly represent buying the 50 stocks with the top predicted returns, shorting the bottom 50, and doing both which can be thought of as long, short and hedged portfolios. However, since our y variable is an excess return over the average stock, the values for LongEx is the return over the average. The values for ShortEx are the returns under the index. Hedge represents the long minus the short. In all cases, positive values are desirable.
Below is a summary of the 142 observations for each.
## LongEx ShortEX Hedge
## Min. :-21.8452 Min. :-24.769 Min. :-33.005
## 1st Qu.: -2.8444 1st Qu.: -1.654 1st Qu.: -2.607
## Median : 0.2659 Median : 1.209 Median : 1.648
## Mean : 0.2940 Mean : 1.421 Mean : 1.715
## 3rd Qu.: 2.8928 3rd Qu.: 5.286 3rd Qu.: 6.773
## Max. : 22.8778 Max. : 23.424 Max. : 46.302
The means and medians are all positive, but there is substantial variation. Below we look to see if the means are significantly different from zero. The standard error is the standard deviation divided by the square root of 142. That is that standard deviation of the sample mean used to determine the signicance of a sample mean. The z score is the mean divided by the standard error.
## LongEx ShortEX Hedge
## Mean 0.2939676 1.4212453 1.7152129
## SD 5.4530391 6.9189390 10.2775924
## StdErr 0.4576089 0.5806245 0.8624764
## z 0.6423993 2.4477875 1.9887071
It appears that the model is better at finding short candidates than long. For fun, the average annual return for the hedged portfolio (without transaction costs or taxes) would have been 15.17% which sounds nice but it had an annualized standard deviation of 35.6 which is much higher than a stock index.
The data in graphical and tabular forms follows.
## LongEx ShortEX Hedge R2 RMSE
## 20040102 4.4804562 1.57539047 6.05584666 2.051306e-02 11.531041
## 20040130 -3.1004470 -1.65188009 -4.75232710 1.750013e-02 9.979721
## 20040227 0.1136727 0.19691053 0.31058323 8.836536e-04 10.200688
## 20040402 -6.8675942 -1.37402545 -8.24161965 4.501892e-02 11.933072
## 20040430 -6.5257645 -6.57552158 -13.10128604 1.370959e-02 11.772875
## 20040604 5.8045868 1.02379413 6.82838097 1.164503e-02 9.544704
## 20040702 1.1833060 7.79095590 8.97426189 4.452512e-03 10.566717
## 20040730 -0.8397685 1.87906727 1.03929881 1.290337e-02 10.118168
## 20040903 5.6043349 -7.24342919 -1.63909426 1.372271e-03 9.702592
## 20040930 -2.2496437 1.34523990 -0.90440375 1.323665e-02 10.682788
## 20041029 -4.8729916 -2.53875630 -7.41174787 1.153225e-01 4.572047
## 20041203 -4.9016458 -3.00009469 -7.90174051 4.306946e-03 10.093777
## 20041231 5.1859436 9.48016279 14.66610640 6.338233e-02 9.439808
## 20050131 8.3554239 3.17228764 11.52771154 3.560239e-02 9.888518
## 20050228 1.1398614 -1.65429137 -0.51443001 6.942819e-04 9.512741
## 20050331 -2.9022806 5.51431793 2.61203732 1.557547e-02 9.531778
## 20050429 -2.9317298 -6.98733369 -9.91906352 2.923010e-02 11.295694
## 20050531 6.6079376 -3.23712871 3.37080888 5.108261e-03 9.130988
## 20050630 4.0869711 -3.54998242 0.53698869 4.182929e-03 9.765563
## 20050729 4.4467331 -2.81801575 1.62871731 3.740829e-03 9.869402
## 20050831 5.9119463 -1.99713549 3.91481085 2.911387e-02 10.014786
## 20050930 -4.3744453 2.80701964 -1.56742571 1.410876e-02 9.707607
## 20051031 0.4936483 -1.91087025 -1.41722193 3.964903e-06 9.534197
## 20051130 1.3152118 3.31415404 4.62936585 4.459394e-03 8.197680
## 20051230 10.4466053 -1.86030232 8.58630295 2.581204e-02 11.349427
## 20060131 -10.7935793 1.08769631 -9.70588296 3.504893e-02 9.868935
## 20060228 0.6297246 0.61394818 1.24367280 7.498170e-03 9.779378
## 20060331 -0.6400524 3.37151326 2.73146090 6.756148e-03 9.633340
## 20060428 -3.6849981 -0.41999246 -4.10499055 2.803800e-02 9.848106
## 20060531 -1.2282337 4.76475783 3.53652412 2.992966e-03 8.893213
## 20060630 2.0611674 4.96603277 7.02720019 9.006035e-06 9.866460
## 20060731 -3.7489384 0.42759653 -3.32134190 8.361648e-03 10.332663
## 20060831 -3.9105259 2.95843656 -0.95208939 5.893924e-05 8.543617
## 20060929 1.3318449 -2.95841884 -1.62657396 4.971125e-03 10.070357
## 20061031 0.6743648 -4.44497331 -3.77060848 9.707332e-04 9.469400
## 20061130 -0.2912007 6.82119372 6.52999305 2.123290e-02 8.075700
## 20061229 -1.1050421 6.72039079 5.61534868 1.220891e-02 8.535747
## 20070131 0.2813190 1.38671605 1.66803509 3.801413e-04 8.637763
## 20070228 -1.4166892 5.34846860 3.93177939 1.002809e-02 8.755420
## 20070330 -1.8134228 -1.61399920 -3.42742196 1.526751e-07 8.352992
## 20070430 2.1966099 5.09304581 7.28965569 3.527596e-03 9.947826
## 20070531 -0.3484850 3.45312623 3.10464119 5.236355e-03 8.290130
## 20070629 -0.6902493 5.65858171 4.96833240 8.614327e-03 10.288178
## 20070731 -1.7350226 3.60422634 1.86920371 2.552182e-05 10.840216
## 20070831 6.6274811 0.70090681 7.32838792 1.817729e-02 10.015743
## 20070928 12.1550218 8.67353742 20.82855921 6.751654e-02 12.192783
## 20071031 -1.5577721 8.04080024 6.48302816 9.717738e-03 11.030412
## 20071130 1.0707887 -0.47628167 0.59450699 1.059295e-02 11.444802
## 20071231 -4.1298808 -16.11316912 -20.24304989 8.802494e-02 13.334368
## 20080131 7.1115468 10.59594501 17.70749184 5.729564e-02 12.426358
## 20080229 -2.7388912 12.99850664 10.25961545 2.594665e-03 11.719874
## 20080331 4.0338069 4.54339825 8.57720515 7.415183e-03 12.818374
## 20080430 7.7046779 12.86765563 20.57233350 4.926652e-02 11.665036
## 20080530 22.8777690 23.42405619 46.30182515 2.185188e-01 12.460203
## 20080630 -21.8452053 -7.14260982 -28.98781510 9.976686e-02 15.854723
## 20080731 -6.3124261 -9.83763975 -16.15006584 4.455588e-02 12.160854
## 20080829 -11.1003861 -5.71285630 -16.81324240 2.105534e-02 15.094456
## 20080930 6.3087282 6.35733551 12.66606375 4.373598e-02 16.619119
## 20081031 6.9581293 11.99049505 18.94862438 6.495681e-02 16.787016
## 20081128 -0.9585375 -12.60407976 -13.56261729 3.513564e-02 16.485555
## 20081231 -8.2017433 4.50898824 -3.69275507 1.318709e-03 15.939062
## 20090130 0.4607981 15.80131193 16.26211005 7.545809e-02 13.364931
## 20090227 0.2504831 -3.98860509 -3.73812198 9.366807e-03 13.900018
## 20090331 -8.2352165 -24.76935176 -33.00456824 1.654495e-01 21.181400
## 20090430 -6.6755156 -12.97698002 -19.65249560 2.408758e-02 15.729201
## 20090529 -4.9018633 -0.85202016 -5.75388345 1.215713e-03 12.016162
## 20090630 -0.6200505 -3.92020524 -4.54025572 1.408301e-04 12.230451
## 20090731 8.8289690 -4.88941648 3.93955256 7.934652e-04 12.265497
## 20090831 6.8330282 -1.66075608 5.17227210 1.014569e-02 10.668416
## 20090930 -6.5173888 5.54391933 -0.97346943 8.227045e-04 9.930213
## 20091030 -3.3226677 -1.15454973 -4.47721747 2.390326e-03 10.923646
## 20091130 5.9842343 0.28345390 6.26768818 2.099230e-02 9.577300
## 20091231 0.4246416 -0.76824734 -0.34360573 1.474345e-05 9.493154
## 20100129 1.9497804 1.34379621 3.29357661 5.861863e-03 9.249832
## 20100226 5.8241792 0.78403315 6.60821237 2.003099e-02 10.470405
## 20100331 2.1724846 -0.01338648 2.15909817 9.830716e-03 9.978537
## 20100430 -4.0222670 -0.75430582 -4.77657279 2.159198e-02 8.488533
## 20100528 -10.3988513 -1.37558122 -11.77443255 6.669126e-02 9.593447
## 20100630 7.3897421 1.59886651 8.98860858 7.464327e-03 9.796380
## 20100730 -3.1515735 0.39769321 -2.75388026 7.631506e-03 10.614657
## 20100831 3.5541220 -4.68430002 -1.13017802 3.949245e-04 9.528127
## 20100930 2.5430752 2.28489812 4.82797331 2.966763e-03 8.926510
## 20101029 -0.3747559 2.47588034 2.10112448 5.030222e-03 9.617739
## 20101130 7.1592502 -0.93817542 6.22107477 6.459225e-03 9.558991
## 20101231 0.8093349 6.57756802 7.38690288 4.947966e-03 9.058666
## 20110131 5.4958556 2.83249943 8.32835507 2.547845e-02 9.326603
## 20110228 3.6982796 5.32997404 9.02825366 1.427021e-02 9.161439
## 20110331 -1.5056150 1.54687332 0.04125832 1.506031e-03 9.241698
## 20110429 -0.2284042 1.60536486 1.37696065 5.219065e-05 8.505485
## 20110531 -0.4956597 1.66295489 1.16729518 2.456909e-03 7.685961
## 20110630 -1.4866051 2.38164003 0.89503493 9.434753e-05 9.038818
## 20110729 -3.3768675 7.26962723 3.89275977 3.528697e-05 10.830363
## 20110831 -1.8263393 10.12701630 8.30067698 7.419489e-03 10.525453
## 20110930 -4.8773408 -8.44679081 -13.32413156 1.076053e-01 12.785670
## 20111031 6.0658434 9.99291772 16.05876116 4.857237e-02 10.307887
## 20111130 0.4086228 3.18809485 3.59671767 8.388918e-03 8.401387
## 20111230 -5.2805620 -7.81151194 -13.09207395 9.642607e-02 11.538577
## 20120131 2.6704043 0.09519766 2.76560200 4.044009e-05 9.006189
## 20120229 2.5201274 1.40957042 3.92969781 2.075017e-02 8.198825
## 20120330 0.4768408 -0.56528695 -0.08844614 1.102996e-02 8.532742
## 20120430 4.7082126 7.42593015 12.13414276 7.476680e-02 9.645694
## 20120531 0.2088106 1.03475503 1.24356561 1.411428e-03 8.514347
## 20120629 -0.7890062 5.15371649 4.36471031 1.098969e-02 10.104601
## 20120731 1.3627650 -1.56265771 -0.19989275 2.151484e-03 9.844206
## 20120831 1.2228883 -3.17089554 -1.94800727 6.180135e-03 7.324758
## 20120928 2.2014011 -0.76737442 1.43402664 4.712458e-04 9.054782
## 20121031 0.7662031 5.48627504 6.25247810 1.083755e-02 8.325804
## 20121130 -0.5426604 -0.58835749 -1.13101786 4.419362e-04 6.920809
## 20121231 0.1320228 1.56948250 1.70150529 3.832838e-03 8.079075
## 20130131 4.5509483 8.34282238 12.89377071 2.345268e-02 8.133354
## 20130228 0.2227061 -0.29514265 -0.07243656 1.481332e-02 7.293513
## 20130329 2.0880405 1.85345985 3.94150039 2.764153e-03 8.570177
## 20130430 2.8641943 3.14529514 6.00948944 2.053273e-03 10.492903
## 20130531 -2.2466517 9.49288056 7.24622887 5.608706e-03 8.488379
## 20130628 0.7401412 -1.51166180 -0.77152060 3.211673e-03 9.660531
## 20130731 -1.2150014 -3.14794863 -4.36295004 4.672797e-04 8.802981
## 20130830 1.6582574 10.33943056 11.99768795 3.862723e-02 9.100338
## 20130930 -2.8375587 8.21986907 5.38231039 8.821435e-05 10.140551
## 20131031 2.8991279 7.11798627 10.01711417 2.395812e-02 9.572929
## 20131129 -0.2203424 1.00449063 0.78414821 6.256081e-03 9.434207
## 20131231 8.7929624 -8.51994247 0.27301988 9.756253e-05 12.261133
## 20140131 2.8486403 0.51828385 3.36692411 1.009918e-02 9.763306
## 20140228 -6.9509677 0.55038918 -6.40057848 5.333648e-02 9.201548
## 20140331 -6.1162032 0.65732288 -5.45888033 3.506616e-02 9.591483
## 20140430 -2.8467217 3.68465637 0.83793471 2.831008e-05 8.601572
## 20140530 6.0681905 -7.80803288 -1.73984236 1.610949e-03 9.892745
## 20140630 -2.7482977 1.33120773 -1.41709000 4.559307e-04 9.343817
## 20140731 0.9038226 -3.07087883 -2.16705620 1.215868e-04 8.763249
## 20140829 -3.6656411 -1.09496768 -4.76060881 2.013618e-03 9.127370
## 20140930 -2.3057577 3.64365706 1.33789939 1.021884e-03 11.660196
## 20141031 2.2538410 4.60841098 6.86225196 2.276797e-02 9.990055
## 20141128 2.8736441 5.72015638 8.59380050 1.959293e-02 10.201338
## 20141231 4.8432917 6.30486373 11.14815546 2.524939e-02 10.461886
## 20150130 -0.4686961 -6.90220626 -7.37090240 1.170588e-02 10.204333
## 20150227 1.8615128 12.37942370 14.24093650 3.272233e-02 9.176222
## 20150331 -8.1689649 -24.59796908 -32.76693398 1.988255e-01 11.102598
## 20150430 15.5061029 13.35092502 28.85702793 1.135935e-01 9.943807
## 20150529 2.3726033 6.62318450 8.99578775 4.649454e-02 8.805501
## 20150630 6.2049437 22.99736412 29.20230784 1.456164e-01 10.637548
## 20150731 -8.0587545 0.81176157 -7.24699292 1.179361e-02 10.206703
## 20150831 -9.6637881 17.69056709 8.02677896 5.339833e-02 10.642361
## 20150930 -7.1714385 -2.52125216 -9.69269066 7.668640e-03 12.903508
No comments:
Post a Comment