I just finished ‘Trading Systems: A New Approach to System Development and Portfolio Optimisation’ by Tomasini and Jaekle. They mention that every system has a point at which optimal complexity is reached, meaning that further in-sample optimization will not lead to improvements in out-of-sample testing.
So my question is, at which point during development must we determine this optimal point?
I suppose it should be done prior to WFA. Lets say I have 5 parameters that can be optimized, and 10 years of historical data. I use the first 8 years for optimization, with the last 2 used for out of sample testing. After this I realize that the point of optimal complexity is reached after 3 parameters are optimized.
So with these 3 parameters, I proceed to do WFA, leaving the last 2 parameters untouched at all times. Is this the correct approach?
In the book you just mentioned, they also discuss calculating the degrees of freedom of your trading system. The length of your backtest/WFA is going to be, more or less, based on DoF.
A very realistic example, I’d say - because five is [I][U]very[/U][/I] likely to be too many degrees of freedom. Most authors I’ve read on the subject say “no more than three” is likely to be optimal.
It’s certainly a well and widely recommended approach, anyway.
thanks for replying. In the book, they did optimization runs for all the parameters over the entire price history (no out of sample testing). But later when doing out of sample testing, it was discovered that optimizing 3 parameters makes the system reach its optimal complexity. So when developing a system, does it still make sense to optimize so many parameters before commencing out of sample testing?
Thanks for replying. If you have 5 parameters and you know its very likely that optimal complexity is reached after optimizing only 3, does it still make sense to optimize the other 2 at any point during development?
Or do you optimize all the parameters over the entire price series without out of sample testing, in order to determine which parameters have the most impact, then focus on those for the WFA?
I would think so, unless I’ve misunderstood what you’re asking. You’re not going to know with certainty which “the main 3” are, and which are “the other 2”, without doing that or something similar?
I agree with you, but how do we know which order to optimize them in? It’s not recommend to optimize everything at once in order to avoid curve fitting. I think that as we start optimizing an unoptimized system, the first few parameters would have the most effect.
In the book I mentioned, they optimized a simple MA crossover system and the order was: MAs --> time filter --> fixed stop loss --> trailing stop --> profit target. Is this order determined through experience/acumen? Or do we try ordering them differently and choosing the best outcome?
People normally try ordering them differently and choose the best outcome, I think. I don’t know how valid it i, though.
I’ll be honest (with absolutely no offence intended at all) and mention that my own guess is that optimizing based on entering trades on the basis of MA-crossovers is going to be completely curve-fitted anyway, and of very little - if any - predictive value. :8: