Making an Ai trader

rllewellyn · July 24, 2022, 7:47pm

Inspired by @wilczasty post: Laziest trading project with Neural Networks - #5 by wilczasty

I’ve decided to have a go at using NEAT for FX trading.

I started by going through this youtube series: https://www.youtube.com/watch?v=OGHA-elMrxI
It explains NEAT very well and I’ve used the code he provides as the basis for my own project.

So far I’m testing with 5 inputs and 4 outputs on small data sets 50-500 candles.
Inputs are the open, high, low, close of the most recent candle and the bots current position (long, short or none). Outputs are open long, open short, close long, close short.

My fitness function is simply pips won/lost + an initial balance at the end of the run, but if any bot loses it’s starting balance of 100 pips it is eliminated.

A few questions that I’m considering:

How to handle the bots logic. All I could make all of the decisions inside the network, but for me to do that the network has to learn that it can only close trades if one already exists and it can only open trades if none already exist (This is because the extra logic to close one or multiple trades, or manage positions seems like too much right now).
Trade entries. Should I buy at the best available price or put an entry order at the high of the previous candle for buy orders and low of previous candle for sell orders.
What inputs to use, but I want to get the bot running smoothly and putting out accurate results and graphs so I can track what’s happening before digging into that.
How to best train NEAT, There are tons of functions for saving and redeploying models that I’ve not come across before, so lots to learn there to optimise training.
How close will the back test match reality? In previous projects I’ve had very profitable back tests that fail miserably in the real world.

Some graphs of what’s going on:

Quick explainer:
The bot enters at the close price of each candle.
Profitable trades have a green background while losing trades have a red background.
Upward arrows show buys, while downward arrows show sells.

This is just the first version, so still plenty to do, but it’s a start.

Comments and thoughts welcome.

wilczasty · July 24, 2022, 8:42pm

Hi! It’s very good to read, that my adventure with neural network inspired you to your own research

I’m not sure if this will work. If NEAT will get only absolute values like prices, it will apply weights and biases and in the end the output may be garbage. It will not come up with relations between these values on its own. Network trained for EURUSD with around 1.00 prices will not work at all when price will get back to 1.20. You may want to consider changing input to relational (% change day to day, candle range to ATR, % of bullish candles for n last candles etc.)

My logic has a simple check, if position is opened. If not - it looks only on first output to make decision on opening new trade. If yes - it looks only on second output to decide on closing.

For simplification you can just go with candle close price

Go for thousands of candles in training dataset and look how equity curve looks around different market periods (you want a smooth one). I’ve seen better results with training on single huge dataset instead of changing multiple datasets. So instead of looking through 1-2 months of EURUSD D1 data I’m going D1 EURUSD data since 2006 to be sure network will learn through different kinds of environment.
Other strategy I’ve tried was to control number of trades by selecting random windows to look for trades within whole dataset.:

There are no open trades
Randomly select place within dataset
Go forward candle by candle until network will trigger trade open
Go forward candle by candle until network close the trade
Repeat
This allows you to precisely set how many trades should each network make. Yet, I believe that going through whole dataset is better training than jumping around.

Good Luck! Looking forward for next posts

ria_rose · July 25, 2022, 2:41pm

Helloooo @rllewellyn! I’ve only been reading about Python every now and then so I’m not in any way skilled enough to contribute to the discussion, but it was still a very interesting post (and video). I’ve also read Wil’s thread but I figured I wasn’t smart enough to try something similar, so I’m glad you decided to try it. I’ll be watching your journey and I wish you all the best on this project!

chesterjohn · July 25, 2022, 3:07pm

Interesting. I was speaking with a colleague about neural networks for forex, but didn’t have time to try it. Now I have lots of time, I might look into this more.

rllewellyn · August 1, 2022, 6:56am

Based on feedback from @wilczasty I modified the inputs to take % change. Still using high,low,open,close though.

The bot seems more sensible than the original even running on the same count of candles, but currently have a 1000 traders running for 1000 generations on 5000 candles to see how it does with a bit more chance to evolve.

As a quick note I was surprised at how easy this is to do. Pandas has an in build function to handle this so the modification only took a few minutes.

Here are graphs of some of the results from a short run of 50 or so generations. You can see several distinct strategies emerging although clearly some are less good than others:

Once my ‘massive’ run has finished, I intend to add EMA’s using the python TA library with %change per candle and some kind of indication of the distance between either the prices or of the EMA’s themselves.

Once I get something vaguely profitable I will hook it up to a test account on Oanda and see how closely the results match my back tests, so i can make sure my testing follows as close to reality as possible.

Blue2 · August 1, 2022, 8:53am

AI/neural networks sound v interesting. Total opposite to my finger-over-the-button way of trading and I just can’t understand this complicated stiff. So not sure why I’m replying!! So, do you think these DIY algorithm-things can beat those of the big players in the market?

wilczasty · August 1, 2022, 6:58pm

Not sure, why AI should beat any other AI. When you trade “manually” you don’t think about beating those Wall Street traders. You just want to be profitable for yourself (We can’t beat big players neither manually or using AI )

rllewellyn · August 1, 2022, 8:47pm

Massive run has finished.

Took 3 days on my Mac. 1000 traders for 1000 generations with 5000 1 hour candles.

The results are in! I’ve selected a few examples of strategies to talk about.

The first strategy is doing quite long holds. Currently there is no concept of hold cost, so the programme doesn’t factor that in. The large section in the middle is holding one position from July 2021 to July 2022. I’m not certain what my broker would charge me for that, but I’m guessing it would clean out any profit from the trade.

Next up. Much higher trading frequency but similar issues, with very long hold times. Also while this was profitable over all there are some huge losses.

Next one is one of the most profitable, interestingly it’s much more selective over it’s trades and generally isn’t holding on very long for each position.

In general it’s interesting there is a major bias for shorting, and it seems large down candles are often used as entries similar to results found by @wilczasty .

Finally, high frequency.

This is not a great performing strategy but is profitable. At this view you can’t see much detail about most of the trades, but if you zoom in you can see that it’s trading a lot.

From this run, I can see a major bias to shorting, which is not ideal since we want to make money on market movements in either direction.
I likely need to add some cost to holding positions so the bot doesn’t hang on so long to it’s positions.

Currently this programme can only see one candle and whether the trader is long, short or out.
I’ll add some EMA’s to give some context about the general market direction, and add a cost to holding onto positions so the bot is encouraged not to hold positions too long and see how that does.

These runs do miss the reliability of the trader and the ‘smoothness’ of the gains(or losses). As things stand this programme could lose tons of trades and win one massive one at the end and still make the cut, which is not ideal, but I think smoothness can come after some improved profitability and avoiding holding trades for a year.

Happy reading, feedback and thoughts very welcome!

rllewellyn · August 1, 2022, 8:53pm

Probably not, but they don’t need too.

As I understand it, this is a game of edge.

If I win a 1:1 trade 55%+ of the time, I make money long term. It’s just like betting, or roulette in that case. Fair odds for a 1:1 would be 50%, If I can move either the return/risk or win/loss ratio I can make money long term. - At least I think that’s how it works

Have you got strategies that are profitable which you’re using manually?

wilczasty · August 1, 2022, 9:37pm

That (and performance) was the reason why I train long and short networks separately. Short networks always perform better regardless of inputs. The best explanation I can think of, is that short movements are caused more by fear and are subjectively more violent than long moves. Networks show drive to volatility.

This is interesting. My current networks hold trades for one candle in >90% of time.

You may want to start track more statistics for your strategies. Despite being profitable, strategies can be not tradable. So some sort of average expectation and max drawdown can be helpful before running live testing.

I’ve had the same problem. To handle it, I’ve tinkered a lot with fitness function, until gave up and changed the way results are saved. Right now I’m looking only for profit during network training (fitness function), but if any network is profitable and has R-Squared > 85% it is being saved for manual review. Very high R-Squared eliminates both large drawdowns and single extreme gains, which “steal” the throne from better (smoother) networks It’s rare - around 1 in 3000 networks, but gives very good networks to play with. So generally I’m not saving best network from training, but inside evaluation function I’m trying to catch networks which meet certain criteria and save them aside. I was surprised to see, that “last network standing” is usually worse than other networks disregarded during the training.

It’s great to see your progress and looking forward for next posts

chesterjohn · August 2, 2022, 7:18am

I did some reading up on NEAT, and I’m not sure that it’s a great fit for trading. It’s essentially got the same problem as optimising an EA, it creates parameters based on the available data, but I find that very rarely extrapolates well

wilczasty · August 2, 2022, 8:18am

Machine learning in general creates multiple parameters based on inputs. But the number of parameters and the complexity (parameters derived from other parameters with biases and weights etc.) can be huge. Thus the name “network”. Other thing is the activation function with threshold.

So this works like this:

you give input(s) eg. price changes, indicators whatever
network takes these numbers and creates tens (or hundreds) of different numbers from these few inputs
all the numbers come down to single number (output) between -1 and 1 (with sigmoid activation function)
you say “if activation function is above X then perform action”

Important difference between this and optimizing EA is that EA does not have this huge complex “fuzziness”. Rllewellyn has a good example above - there is only OHLC % change as input, and yet millions of individual networks are created. What parameter is optimized here? We don’t know. Algorithm creates it’s own parameters we don’t see (understand)
EA is 100% dependent on developer - we give exact algorithm, define inputs and set exactly what and how is optimized. So it’s limited by our design.

Extrapolating is just a problem of proper training data, not the method itself. If you feed the network with large enough dataset, it will handle new environment to some extent.
But this is a problem for machine learning generally. If you teach network to recognize numbers by image recognition, it will be able to tell what number it sees written in a different font or by hand. If you show it totally different character - it will fail, because this cannot be extrapolated.
If you know English perfectly you cannot extrapolate this knowledge to eg. speak/write Japanese no matter how good you’re trained in English - you need new data. But if you know British English perfectly, most likely you will be able to communicate with Australians, Americans or even non native speakers from all around the world. Network can adapt like this. If you would like EA to do this kind of optimization - you would need to force feed it with all possible accents and small grammar mistakes Hope this makes sense.

rllewellyn · August 7, 2022, 7:00pm

This week, I added 4 EMA’s to the programmes inputs (given to the bot as a % change between last and current value) with a dif of the lowest to the highest EMA.

I also added a financing variable to capture the cost of holding positions and hopefully avoiding the bot holding positions for a year, as it was before.

After making these updates the bot has had two runs, 1000 traders for 100 generations on 5000 candles, but never traded at all. I hope that I made the daily financing too aggressive, rather than broke something, but I need to do some de-bugging to find that out. Sadly this means no graphs this week.

Next week, if I can get the bot running again I’ll look to capture some more trading data to get a better handle on how well the bot performs outside of the simple final PnL, and track a bit more of what is going on for each generation.

chesterjohn · August 15, 2022, 12:00pm

I have the same problem. Eventually it decides not trading is the best outcome, even if it got profitable in the mean time. I’m using completely different software to you, but I find that if I save the weights every 10000 steps and then test each of these, there’s a few that do well in the middle before it decides to stop.

What inputs do you use? I feel OHLC plus indicators may be confusing without an LSTM layer, which I don’t have right now. It results in 4 inputs that don’t really mean anything. Maybe removing price from the equation would help it determine the entry from multiple indicators and make better decisions…

rllewellyn · August 16, 2022, 6:56am

Not very much progress.

I’ve been having issues with the for loop that feeds data to the activation function. Initially I had candles as a list of dictionaries and that worked fine.

Now I’m calculating more values, I tried to move to pandas dataframes. However, the for loop fails to generate a working network regardless of how long it’s run for.

Strangely, the programme runs fine, but it never generates a bot that performs better than just holding the initial balance. The code is the same in every way and I checked the both the for loops and the input variables.

In the end I re-wrote the whole thing from scratch instead of just modifying the code from the Flappy Bird game which is what I started off with. While this is a bit annoying the bot is now working with Pandas and runs a lot faster. I still don’t know why the first bot failed but onwards and upwards.

My trader function can now generate some more detailed stats of it’s results and I have a connection to G sheets to dump all of the results.

I’m also trialling the checkpoint functionality to train the bot on different data sets.

This week I’m aiming to get G sheets hooked up to display results of runs.

Next week, I want to try and operationalise a network on a test trading account just to make sure my test environment looks a little like reality.

After that testing all sorts of different inputs to see what works best. I’m wondering how I can implement some kind of grid search type run, so that the bot will test large volumes of data by itself without assistance.

Interesting finding this week was that, removing bots that either lose all their money or not trade at all in a run does not seem to make much of a difference to training speed, so I’ve got rid of lots of extra bits of code that was there to get rid of ‘under performing’ bots.

rllewellyn · August 16, 2022, 7:00am

I am. using % change on OHLC and bots position eg. 0 for no position. -1 for short and 1 for long. My bot can only handle one stake at a time and stakes are currently fixed. Giving it the ability to build positions seemed to hard to start off with.

I’m aiming to get things running and then focus on the inputs. Ideally using something like a grid search to test lots of different variables without much hand holding.

@wilczasty has reported some good results with EMA’s?

ontario · August 16, 2022, 7:41am

With the same success you could just generate series of random 0 and 1 and bet up or down according to the number. Same result. And you could also see some winning or losing patterns in those trades but it wouldn’t tell you about anything. When you don’t see what pattern you are looking for this is no better than random guess.

wilczasty · August 16, 2022, 7:53am

One issue I’ve had with EMAs was lack of data for first records in dataset. For ema 200 first 200 lines of data are garbage and you need to handle this (delete).

I use below code:

    df.replace(r'^\s*$', np.nan, regex=True, inplace=True)
    df.dropna(inplace = True)

I also have min number of trades calculated from dataframe size. If network has anything below it gets punished with terrible result. Right now I’m using “ACTIVITY_FACTOR” = 70, which means, that I expect network to make a trade every 70 candles on average. (num of trades > len dataframe / 70). This helps to avoid networks, which are afraid to trade

chesterjohn · August 16, 2022, 8:45am

What are you replying to? You’re right about what you say if it was random, but it isn’t. A neural network is designed to learn how to recognise patterns and then use what it’s learnt to make predictions. This is how AI for speech bots, picture recognition etc works.

You tell it some data, it runs a whole load of things that you don’t know about and it gives an output. You then tell it whether that output was good or bad and it tries to adjust things to make itself better. That’s not random, but it’s also not easy. This falls under reinforcement learning, which is one of the hardest types of AI to do. If it was easy then we’d all be rich.

55% win rate is possible by random chance, but remember that trading isn’t just about opening a position and then you’ve won or lost. It’s about opening AND closing at the right time. That’s much harder to get with random chance.

Personally, I’m looking for bots with over 60% win rates and high returns because there’s no R:R when the bot decides everything, so we want a vey high win rate to minimise the chances of a big mistake.

rllewellyn · August 21, 2022, 9:51am

I had a bit of a delay getting my bot running again, but now it seems to working fine.

I’ve run training on 10,000 generations regularly switched between 5000 candles of input data across GBP/USD, EUR/USD, USD/CAD amongst other pairs all on the 1h time frame. Prediction inputs are still % change in OHLC, bot position (eg. long short or none) and % change in 10 EMA.

Running the winner by itself on GBP/USD gave these results: trades count 210 Final balance 4700.09999999998 PnL 4128.099999999983 win rate 0.8428571428571429 Av win 24.915254237288085 Av loss -8.809375000000259
See graphs below of the full run and a detailed view of last week:

These results look suspiciously good to me. So I graphed ‘bank’ and the pips return per trade:
Pips per trade looks fine.

Bank by trade. Looks wayy too perfect. You see my bot starts with 1000, and I add the pip profit/loss to that. I also use this as my fitness function.

I was expecting to need to add some additional rules to ensure that the bot didn’t continue holding for too long as it did in my original runs, or that I would need to add something to remove bots that never traded, but it turns out neither are required, the programme seems to have learnt that reasonable short trades are the way to go.
You may also notice from the graph it’s almost constantly ‘in’ the market almost never not having a position open, so trading frequency doesn’t seem to be an issue.

I ran the bot on all 1hr candles from GBP/USD since 2010 with these results:
trades count 40650 Final balance 744598.1000000001 PnL 824906.1000000004 win rate 0.8485608856088561 Av win 26.95219168551055 Av loss -17.024012997562988
Average win value and win rate have remained similar while average loss has increased massively. However, there is still a good distance between average win and loss values.

My next step is to hook this up to Oanda test account and see how it works for a week. I’m planning to use docker and will deploy using digital ocean and gitlab. Once that’s done I’ll set up more of a testing rig to try out more indicators and more time frames with a grid search type approach.