In this post, we look at out-of-sample performance for some random forests. The purpose here is not an exhaustive analysis. We are only hoping to get an indication that our efforts might bear fruit.
Each random forest model we build uses predictor (x) data from one month (e.g. Dec 2002) and response (y) data from the subsequent month (e.g. Jan 2003). We refer to a model here by xmonth/ymonth (e.g, Dec02/Jan03) to indicate the months of the x and y data. The y variable is a company’s excess return defined as the return of company for the month less the average return of all companies for that month.
The first forest is Dec02/Jan03. The first set of predictions was for Jan 2004 created by averaging the 12 predicted returns for each company by feeding in the Dec03 x data into the Dec02/Jan03 through the Nov03/Dec03 models. The last prediction is for Oct 2015. We have 142 months of predicted return, a bit shy of 12 years.
For each month with a predicted return, we calculate three values: LongEx, ShortEx and Hedge. Respectively, these roughly represent buying the 50 stocks with the top predicted returns, shorting the bottom 50, and doing both which can be thought of as long, short and hedged portfolios. However, since our y variable is an excess return over the average stock, the values for LongEx is the return over the average. The values for ShortEx are the returns under the index. Hedge represents the long minus the short. In all cases, positive values are desirable.
Below is a summary of the 142 observations for each.
## LongEx ShortEX Hedge
## Min. :-22.5162 Min. :-24.762 Min. :-40.723
## 1st Qu.: -2.2450 1st Qu.: -1.205 1st Qu.: -2.135
## Median : 0.6743 Median : 1.375 Median : 2.676
## Mean : 0.5615 Mean : 1.750 Mean : 2.311
## 3rd Qu.: 3.7627 3rd Qu.: 5.406 3rd Qu.: 7.610
## Max. : 20.7098 Max. : 26.804 Max. : 43.397
The means and medians are all positive, but there is substantial variation. Below we look to see if the means are significantly different from zero. The standard error is the standard deviation divided by the square root of 142. That is that standard deviation of the sample mean used to determine the signicance of a sample mean. The z score is the mean divided by the standard error.
## LongEx ShortEX Hedge
## Mean 0.5615240 1.749783 2.3113067
## SD 5.8138698 7.068079 10.9250335
## StdErr 0.4878891 0.593140 0.9168084
## z 1.1509254 2.950033 2.5210355
It appears that the model is better at finding short candidates than long. For fun, the average annual return for the hedged portfolio (without transaction costs or taxes) would have been 22.42% which sounds nice but it had an annualized standard deviation of 37.85 which is much higher than a stock index.
The data in graphical and tabular forms follows.
## LongEx ShortEX Hedge R2 RMSE
## 20040102 5.40387192 1.49061552 6.8944874 1.834194e-02 11.543642
## 20040130 -2.90584266 -1.20667423 -4.1125169 1.569290e-02 9.995714
## 20040227 0.13278078 0.98072400 1.1135048 7.969597e-05 10.180885
## 20040402 -9.13640289 -5.63940335 -14.7758062 6.364674e-02 12.002930
## 20040430 -2.62634212 -4.01278530 -6.6391274 9.988053e-03 11.767189
## 20040604 5.90600405 0.43858870 6.3445928 1.747279e-02 9.522831
## 20040702 6.20691840 6.41741864 12.6243370 5.745495e-03 10.558267
## 20040730 -2.46220377 5.16432893 2.7021252 9.654185e-04 10.214583
## 20040903 8.95935876 -3.82297670 5.1363821 1.086204e-05 9.665120
## 20040930 -2.25621028 -4.98650710 -7.2427174 1.922212e-02 10.730126
## 20041029 -4.71568260 -4.40054214 -9.1162247 1.233034e-01 4.586698
## 20041203 -5.28801760 0.52139770 -4.7666199 4.370066e-04 10.039059
## 20041231 2.99957129 8.19629732 11.1958686 5.723372e-02 9.451499
## 20050131 5.52407287 5.46746389 10.9915368 3.020293e-02 9.907540
## 20050228 -0.30067131 3.79045522 3.4897839 1.649513e-03 9.499445
## 20050331 -1.67862247 2.83974979 1.1611273 1.628908e-02 9.529142
## 20050429 0.99275765 -6.97082462 -5.9780670 2.219801e-02 11.282139
## 20050531 2.72670839 -2.16575073 0.5609577 1.069156e-03 9.180009
## 20050630 1.84195759 -0.69437808 1.1475795 3.426903e-03 9.782919
## 20050729 1.79737890 -2.47887147 -0.6814926 6.166191e-03 9.857890
## 20050831 6.17286384 2.24980071 8.4226646 2.554721e-02 10.014044
## 20050930 -5.98986592 4.41408379 -1.5757821 1.051827e-02 9.738044
## 20051031 2.85395424 -0.08926807 2.7646862 3.864877e-04 9.538442
## 20051130 2.02526217 3.97892719 6.0041894 6.130005e-03 8.198109
## 20051230 10.57817177 -1.16175125 9.4164205 3.635384e-02 11.298988
## 20060131 -12.31409780 1.11100556 -11.2030922 3.765661e-02 9.937796
## 20060228 -1.23882121 0.02364098 -1.2151802 5.932061e-03 9.790312
## 20060331 2.33144960 3.83521890 6.1666685 1.126032e-02 9.610501
## 20060428 -2.16512505 -0.25432596 -2.4194510 2.215926e-02 9.868287
## 20060531 0.57065155 3.44502314 4.0156747 5.118772e-03 8.882741
## 20060630 -0.73614571 6.70541092 5.9692652 1.451162e-03 9.924111
## 20060731 -4.82746900 1.06078641 -3.7666826 1.820947e-02 10.411808
## 20060831 -1.53685366 2.36372882 0.8268752 4.551253e-08 8.550465
## 20060929 1.34401623 -5.22733741 -3.8833212 4.963447e-03 10.091191
## 20061031 0.33179334 -1.79494054 -1.4631472 2.071015e-03 9.493776
## 20061130 3.23719076 7.11509833 10.3522891 1.899938e-02 8.077665
## 20061229 0.62651354 5.53317368 6.1596872 1.917711e-02 8.510653
## 20070131 1.79883810 1.91282218 3.7116603 8.551330e-05 8.662607
## 20070228 2.36532269 8.12193065 10.4872533 1.870687e-02 8.724430
## 20070330 0.66013174 -1.44985453 -0.7897228 4.948791e-05 8.340518
## 20070430 4.20283840 5.72909214 9.9319305 5.325003e-03 9.935763
## 20070531 2.55541558 4.78139656 7.3368121 9.077696e-03 8.271241
## 20070629 4.45754882 8.88819376 13.3457426 4.694815e-02 10.177925
## 20070731 0.35362342 4.90751993 5.2611434 3.905772e-04 10.864827
## 20070831 9.35609986 5.22171948 14.5778193 4.011274e-02 9.938294
## 20070928 13.02347206 9.71712419 22.7405962 7.581077e-02 12.143863
## 20071031 -3.29591884 6.85961644 3.5636976 1.028862e-02 11.033317
## 20071130 4.27899793 0.50451265 4.7835106 1.897805e-02 11.390148
## 20071231 -6.81048107 -16.58904093 -23.3995220 1.019013e-01 13.510965
## 20080131 9.62473902 6.67259437 16.2973334 6.698388e-02 12.362566
## 20080229 -1.30283115 10.77690007 9.4740689 1.629201e-03 11.750796
## 20080331 5.23295558 5.13420383 10.3671594 6.292338e-03 12.836256
## 20080430 9.51447558 15.06175678 24.5762324 4.709161e-02 11.662685
## 20080530 20.70976327 22.68674048 43.3965037 2.454185e-01 12.378729
## 20080630 -22.51623306 -9.16655632 -31.6827894 9.108577e-02 15.812902
## 20080731 -6.71746970 -6.45012430 -13.1675940 4.224404e-02 12.228179
## 20080829 -12.11853852 -6.15555256 -18.2740911 7.854400e-03 15.045762
## 20080930 7.70343500 2.98878497 10.6922200 6.351603e-02 16.550795
## 20081031 7.77065580 16.33398338 24.1046392 6.723788e-02 16.789974
## 20081128 -4.74044557 -8.85837340 -13.5988190 2.856071e-02 16.384498
## 20081231 -6.38976733 0.76137445 -5.6283929 1.109729e-03 15.911839
## 20090130 2.93786172 14.11290523 17.0507670 6.924414e-02 13.395856
## 20090227 0.28782440 1.44022007 1.7280445 8.413000e-03 13.855778
## 20090331 -19.96418207 -20.75861716 -40.7227992 1.752411e-01 21.165795
## 20090430 -1.22754181 -12.17448192 -13.4020237 3.708760e-02 15.749353
## 20090529 -5.97528124 4.49151171 -1.4837695 1.406668e-02 11.896696
## 20090630 0.99788090 -0.57775222 0.4201287 3.801494e-06 12.187126
## 20090731 7.49204705 -4.62934670 2.8627004 1.542391e-02 12.130585
## 20090831 6.02854870 -0.06153462 5.9670141 1.724183e-02 10.617830
## 20090930 -5.13539674 1.13381662 -4.0015801 1.873951e-02 10.037458
## 20091030 0.57892472 -0.02746684 0.5514579 6.325462e-05 10.795407
## 20091130 5.78601764 1.30962862 7.0956463 1.452356e-02 9.556327
## 20091231 -4.89740870 -5.58256908 -10.4799778 6.613241e-03 9.547761
## 20100129 0.68854332 4.15288100 4.8414243 5.792483e-03 9.218921
## 20100226 3.56453273 3.48810664 7.0526394 2.171415e-02 10.425906
## 20100331 3.76850032 -1.11862379 2.6498765 1.043408e-02 9.955274
## 20100430 -3.57297455 1.03427232 -2.5387022 1.200480e-02 8.437923
## 20100528 -9.19090511 1.02365515 -8.1672500 6.393829e-02 9.551787
## 20100630 3.30652374 -3.85553371 -0.5490100 5.207713e-04 9.849794
## 20100730 -2.30857883 0.16902594 -2.1395529 7.085925e-04 10.558207
## 20100831 0.22770813 -3.21858771 -2.9908796 6.146996e-03 9.606660
## 20100930 1.56357640 0.00938446 1.5729609 1.164398e-03 8.942725
## 20101029 4.96682807 2.13126222 7.0980903 1.386240e-02 9.579759
## 20101130 0.69697360 -1.92746175 -1.2304882 1.213165e-05 9.631950
## 20101231 0.41944075 1.76707161 2.1865124 1.905663e-04 9.111841
## 20110131 4.07249029 3.43658299 7.5090733 2.288144e-02 9.329095
## 20110228 1.57398135 10.42736372 12.0013451 1.629467e-02 9.152722
## 20110331 -3.90680955 2.42981982 -1.4769897 2.815608e-04 9.219599
## 20110429 -2.21139055 2.09828754 -0.1131030 8.730911e-04 8.526235
## 20110531 1.78352772 -0.71897356 1.0645542 2.200055e-03 7.684808
## 20110630 2.37557885 -1.57026100 0.8053178 8.728497e-07 9.042902
## 20110729 -1.78568333 9.32320511 7.5375218 1.033621e-05 10.826720
## 20110831 0.57745107 13.62326644 14.2007175 1.329892e-02 10.497113
## 20110930 -8.08491174 -12.58544459 -20.6703563 1.171755e-01 12.754920
## 20111031 0.73842683 11.92893829 12.6673651 4.522994e-02 10.313859
## 20111130 0.62039481 6.05542499 6.6758198 9.910000e-03 8.385709
## 20111230 -3.12637477 -10.77157383 -13.8979486 1.193010e-01 11.509917
## 20120131 3.83209651 -0.72662539 3.1054711 8.697444e-04 9.006328
## 20120229 2.78250343 4.20896233 6.9914658 1.259814e-02 8.232192
## 20120330 1.29603066 1.93316943 3.2292001 1.430192e-02 8.517169
## 20120430 0.70772341 7.06908371 7.7768071 6.766543e-02 9.671125
## 20120531 0.02516932 0.77521587 0.8003852 1.987880e-03 8.505345
## 20120629 -1.60434325 4.40654669 2.8022034 5.495914e-03 10.136726
## 20120731 -2.11993454 -0.92663903 -3.0465736 1.074480e-02 9.898911
## 20120831 0.52310340 -5.08074991 -4.5576465 4.650886e-03 7.295555
## 20120928 0.05659778 0.51117595 0.5677737 4.214609e-04 9.087631
## 20121031 -0.23779127 0.37185423 0.1340630 2.713459e-03 8.377153
## 20121130 2.00536228 -1.57489129 0.4304710 4.888700e-04 6.963249
## 20121231 -1.95429132 2.19968932 0.2453980 3.341025e-03 8.083581
## 20130131 -0.29090963 9.14237681 8.8514672 2.022032e-02 8.141436
## 20130228 3.74537978 2.28872860 6.0341084 2.562331e-02 7.258036
## 20130329 1.92061553 4.31622002 6.2368355 7.148353e-03 8.545509
## 20130430 4.83940591 -1.19840637 3.6409995 8.253624e-05 10.545379
## 20130531 -1.57245576 7.97683901 6.4043832 9.882217e-03 8.465437
## 20130628 6.69750147 -2.90917080 3.7883307 1.968991e-03 9.674799
## 20130731 -0.11998782 -1.07104708 -1.1910349 9.797558e-05 8.772422
## 20130830 1.34077359 9.87206502 11.2128386 3.540380e-02 9.101706
## 20130930 -3.80803191 6.17023329 2.3622014 2.317402e-05 10.142080
## 20131031 3.47095169 6.80909668 10.2800484 2.659929e-02 9.558073
## 20131129 4.82172181 4.48176530 9.3034871 9.850874e-03 9.418059
## 20131231 7.36582535 -9.48903691 -2.1232116 1.038420e-04 12.306525
## 20140131 8.71588554 -0.72788277 7.9880028 1.768703e-02 9.722693
## 20140228 -6.43266709 1.88881620 -4.5438509 4.598074e-02 9.251081
## 20140331 -9.38698918 -2.28673159 -11.6737208 6.641972e-02 9.699626
## 20140430 -4.21838698 0.75619705 -3.4621899 3.545542e-03 8.668348
## 20140530 5.98800987 -7.42259449 -1.4345846 1.341742e-03 9.899448
## 20140630 -0.98814354 -0.77379340 -1.7619369 2.488615e-03 9.389641
## 20140731 8.13736699 -4.25483957 3.8825274 4.045783e-04 8.749128
## 20140829 -4.64167473 4.30032614 -0.3413486 1.659678e-04 9.108906
## 20140930 -0.43646130 2.82873412 2.3922728 6.177938e-05 11.646524
## 20141031 3.24017972 5.88838519 9.1285649 4.314582e-02 9.929667
## 20141128 0.69624041 9.44601147 10.1422519 2.322371e-02 10.184646
## 20141231 3.32910254 8.04782296 11.3769255 2.426322e-02 10.459319
## 20150130 0.62160724 -6.84705564 -6.2254484 2.445408e-02 10.292621
## 20150227 6.53451777 12.04962323 18.5841410 3.950588e-02 9.150841
## 20150331 -3.83022242 -24.76201620 -28.5922386 2.128941e-01 11.148031
## 20150430 14.28192548 15.05480750 29.3367330 1.189118e-01 9.917572
## 20150529 6.50182847 6.96746221 13.4692907 5.988414e-02 8.753609
## 20150630 7.37529168 26.80369568 34.1789874 1.775662e-01 10.530589
## 20150731 -9.78038615 -1.04585056 -10.8262367 1.059950e-02 10.215449
## 20150831 -9.09721311 16.73157848 7.6343654 6.713816e-02 10.577300
## 20150930 -6.36406713 -0.36577902 -6.7298462 1.281589e-02 12.954967
No comments:
Post a Comment