I’ve been down my own ML/AI road! It’s fun and it’s frustrating. Ultimately, it’s doomed to fail. I don’t mean to be a downer here, but there are several fundamental principles that streaming market data violate when trying to apply ML techniques.
The first is that there are no Markovian paths in the market. The market doesn’t have rules based on any set, subset or superset of indicators, hours of the day, twitter sentiments, phases of the moon, … whatever. Just because we come up with some great set of inputs for a model, doesn’t mean the market used them for predicting the next leg of price movement. Even though we can create tens, hundreds, thousands of inputs, none of them are related to the NEXT candle from the market.
If you happen to have a model that looks profitable, it can’t be (not WON’T be, but CAN’T be) for very long. You might have over-optimized a local maxima, but it’ll disappear soon enough. Any given “market” is really a collection of agents, and each agent’s motivations change over time. Long-term HODL’s mixed with short-term high frequency traders. Large institutional accounts mixed with a zillion retail traders. And everyone has a different set of input factors for their own wetware, which is never a stable process due to human emotions. (How many times have you changed your system? Now multiply that by all traders playing in the same market as you. Now multiply THAT by the average trader lifetime who exits with a blown account, only to be replaced with a novice trader and fresh cash.) Classic ML, including NN, requires stable underlying processes. Reinforcement Learning can work with slowly changing ones. Neither condition exists in the markets.
AI and ML seem very appealing. But all ML relies on fundamental relationships connecting inputs to outputs. There are no rules in trading (no physics, no engineering boundaries, no code base, no statistical correlations) so any model that is temporarily successful is not successful for long.