Expectations vs reality!
I ran the programme on my local machine over the last day or so.
This is still a pretty simple network just using OCHL 1 EMA and the bots position as inputs.
The good news is that despite a slightly hacky implementation the bot seems to run well.
The bad news is that spreads seem larger than I had modelled for training and the pips returned are coming out to be quite different.
For the bots performance. It’s a small data set but it’s not matching the 80% win rate that it had in training. So far it has won 5 lost 6. Overall it’s down, due to one loss that was 4x larger than the others. Otherwise it would be just about profitable.
I’m going to leave it running as is for a bit longer to gather more data, but then will need to determine why the entry and exits are so different to the expected values and how I can get that into the training bot, and then determine a more accurate measure of profit to use as the fitness function.
All in all, I think these are very positive results. Quite a simple network is really not doing too bad in the wild and I’ve got some clear areas to improve for it’s training for next time around.
2 Likes
What timeframe are you using? GBPUSD has been in a long term downtrend and had a big up move. I’d guess that’s the reason. I find that losses are much bigger, which I suspect is down to the close signal not coming because it went the wrong way and holds out through to the next buy and then closes.
I tried to create a buy and sell bot to use in combination, but I can’t get both to work on any chart so far.
1h on GBP/USD there was a decent spike, which is I what caused the big loss. It did close out on the next candle, so it didn’t hold onto the loser.
Eye balling them, win amount is about even, win rate is a bit low though and clearly I’ve been training the bot on bad data, since the expected and actual values are so far off.
I ran the bot again over night, for trade executed at 23:01 last night the spread was ‘-0.0195’ That seems very expensive, I’m only trading with 100 units? I’ve hooked my bot up to Oanda.
I wonder if I should track both bid and ask prices to determine entry and exit positions. seems like training a bot without a better measure on profitability is not a good idea.
That’s a problem with this, the spread overnight can be crazy. That’s absolutely huge. I simulate a 5 pip spread because I thought that would cover large spreads at night, but looks like it’s not even close.
It’s the same problem with higher time frames, all entries would be at 00:00, so a big spread. Not sure how to deal with that. I was considering using 4H data and changing everything to be equivalent to daily and use it at 12:00 so the spread is small, but I’m not sure if RSI or momentum would work correctly
I’ve got accounts with Oanda and FXCM. That spread was Oanda. I’ll get a view on FXCM night time spreads and see if it’s any better. While that was the biggest it’s not a massive out lier. I couldn’t find a fixed spread broker with an API.
I’ve seen people make their own candles using Pandas, so you can create different time frames which could enable you to do 1D but run from 08:00 - 08:00 when the spreads are less bad. I don’t see why they wouldn’t work, but they would be different to everyone else who was using 00:00-00:00.
I may need to increase time frame to 4H and only allow entires during normal trading hours. That does raise the question of whether I should only train the bot with that data, rather than all data?
First up I’ve added real values to the performance print out so I can compare real vs expected across all metrics more easily.
Then I’ll see about getting a feed on spread, adding bid and ask prices or I’ll just model -20 pips as the start point for every trade and not allow the bot to open positions out of hours.
I also wonder if the strength of the out put from the network would have any correlation with win rate,
In terms of how the bot is doing in the wild. Pretty badly is the answer. By it’s own performance logging it should be profitable but only at 55% win rate, with a near evens average win to average loss. The reality is worse and certainly not close to being profitable.
To try and understand better I mapped the dif of the bid and ask price at the close of each candle on the 1H time frame.
After a few weeks away I’m back at it.
It turns out that I’d accidentally given my previous bot the ability to see the future as it was taking the opening price of the most recently closed candle as its entry - essentially enabling it to see 1 candle into the future.
This may go some way to explaining the massive spread I saw in real runs of the bot and also the near perfect performance of the bot under test conditions. I only identified this issue after running the bot in live (live trading but on a test account - so no real money) for a while to see how it performed.
Now those bugs are fixed it is essentially back to the drawing board regarding training.
In my test runs I’ve found that training on large data sets leads to long hold time for the bot - that is something that I want to avoid - due to cost but also I want to maximise cycle rate. You can see the difference in the screen shots below:
The top one trades multiple times in a week while the second will quite happily hold one trade for a whole week or longer.
Now, I’ve updated to bot so it can’t see the future any more and has a consideration for spread so the results should be closer to the real world.
For training, I’ve taken long data sets of 2020 to now and broken them down into weekly chunks. The bot will train 100 generations week by week. Hopefully by the end it’ll have learnt a bit about when to enter and exit trades! (and will do so multiple times in a week)
My current considerations are:
- Whether I should train the bot on multiple currencies, my idea is whether there are principals that would be be better taught by seeing lots of currencies.
- Inputs my current bot takes 15 inputs across RSI a few EMAs and a couple of others.
- How effective these same techniques maybe on other instruments eg. FTSE100
- Should I add a stoploss to the bot
For now, once the bot is running I’m going to hook into the FXCM API since they have a wider range of instruments than Oanda who I use right now.
1 Like
The results of the long week by week training are in!
When run on the most recent 500 candles:
All of this is looking pretty reasonable. It picks some good entry and exits. Not all 100% but that’s to be expected.
Then I run the same network on a longer time frame to see how it will do more generally:
I’ve added the candle chart for completeness despite it not being very useful.
The other two graphs show the bot performing really pretty badly until the final 2 weeks.
Putting it into numbers the bot made 647 trades with a 40% win rate, average win of 36 pips, average loss of 31 pips.
The bot as it stands now doesn’t seem to hold for very long which is what I was hoping for. My next step is to train the same bot on GBP/USD from 2020-2021 and then run it in 2021-2022 to see how it performs. That should give a reasonable view of it’s performance over the longer term.
In terms of other considerations:
- I’ve not trained trained this bot on multiple currencies yet, so that maybe worth trying.
- As yet I’ve not modified inputs
- I have connected to the FXCM API and reformatted the candles so that I can feed them into the same bot so I’ll try the SnP500, FTSE, and VIX soon.
- No stoploss yet.
1 Like
So first up the FX run. I trained the network on 2020-2021 using H1 candles, and after training ran the bot on 2021-2022 with good-ish results.
16 trades, Average win of 148 pips average loss of 35 and a win rate of 81%. The issue here is that the bot is back to taking very long trades.
Training the bot on year to date, I get 19 trades, 0.63% win rate with an average win of 252 and loss of 145.
Not too bad but again very long hold time especially for CFD’s.
The bot gets similar metrics 2018-2019 w/ 68% win rate and 16 trades. 2018-2019 gets 22 trades Final balance 559.0000000000244 PnL -220.99999999997556 win rate 0.45454545454545453 Av win 136.20000000000144 Av loss -143.90909090909
2017 - 2018 is worse. So it doesn’t feel like this bot has identified any real principals yet.
I also said I’d try some other instruments. I hooked up FXCM so I can use Sxp500 and a few others that Oanda doesn’t provide. Going through this process was pretty easy and I noticed how much better my programming is since I started this project - so that’s a bonus.
The bot is set up and running now.
With the most recent runs I clearly need to do something to reduce the hold time, possibly add some more inputs, and I’m wondering if I should try to modify the fitness function to ‘score’ based on how close it is to a peak or a trough, eg. how close to an optimal trade. With the current bot it often missed ‘optimal’ entries and exits.
First trial on SPX500. The fitness function that determines PnL needs some major modification since it’s still geared for FX. -that drives some of the odd numbers, but the general direction should still be accurate.
Bot was trained on the most recent 5000 candles and I ran the network on 2022.01.01 - 2022.03.01 data and these are the results:
Numbers wise: 53% win rate, 661 trades and a pretty even average win to average loss. This
Also 2022.06 to 2022.09:
In trying to see how well this network may work generallyI ran it on 2021.01 to 2021.03 data. Which is rather less good:
Strangely this doesn’t seem to have the issue with massive hold times the I’ve found with FX but the repeatability still doesn’t seem great.
