I've developed a system, now how can I accurately test it?

Hi everyone,

This is my first post so I’ll just introduce myself. I’m a postgrad student from Australia currently completing an MPhil in a completely unrelated field. I have always had an interest in technology and investing, and have been quite an avid value investor for a couple of years. About 2 years ago I started toying with the idea of building an intelligent computer. I started building an algorithm which structurally resembles what is commonly referred to as a feed forward neural network in the literature, although my program deviates from the conventional structures quite a lot.

Anyway, it took me about a year to build and another to fine-tune. It’s a pretty cool little machine, and is able to learn to play a perfect game of tic-tac-toe in less than 200 milliseconds (checkers takes about 3 hours on my PC). I had a lot of fun for a couple of months coming up with different tasks for it to solve. It was great to see it learn to solve problems; when I asked it to solve the fibonacci sequence, I was skeptical of the outcome because it just looked too quick. When I took a look at what it had done, I found that it ended up just fitting a logarithmic curve, then rounding it; I had no idea you could even do that to solve that sequence! Anyway so that was a lot of fun but kind of useless.

So recently I decided to step it up a bit and started feeding it currency data and asking it to predict whether the price of a given pair would go up or down in 4 weeks time (I went with such a high number because my data is only hourly, so it doesn’t have enough resolution to work with smaller time-frames). I downloaded a set of hourly forex datapoints for 65 different currency pairs spanning 10 years (1/1/2000 - 31/12/2009) and asked it to predict what the price of the AUD/USD pair would be 28 days from now at every datapoint in between. The only charting data I gave it (other than the prices for each of the 65 pairs) was the 200-day moving average for each pair, because I thought it might find it useful (I haven’t checked whether it’s using it).

At each time step the program is given the previous 24 hours of each currency pair, plus 5 200-day moving average points extending back at 3 month intervals (so 200-day moving averages from today, 3,6,9, and 12 months ago). So at at each of the 30,000 time steps (ranging about 9 years) it’s given just under 2,000 inputs.

Based on that (and I haven’t tinkered with the inputs yet), it can get about 90% accuracy on data that it is trained on. That’s nothing special because it could just be memorizing the movement of the chart. What [I]was[/I] interesting was that it was able to consistently exceed random guesses on data point outside of the training domain with an accuracy of 60-70%. That is to say, it could tell what a price movement would be at a time in the future [I]it had never seen before[/I] with over 60% accuracy.

Anyway, so basically I think I might potentially have a crystal ball (at least 60% of the time…) and I’d like to take it for a test drive. I’m in the process now of developing some conceptual automated trading algorithm based on the predictions, but I was hoping to plug it into an existing trading platform and start testing it in vivo. To do this I’d need a trading platform that gives me access to some sort of API that I can use to programmatically call my own program, and also historic, high resolution (the higher the better; by the second?) currency data for all major currencies extending back several years (preferably to 2000 or earlier) up to the present that it could use to keep training my program with the latest data. I’m familiar with javascript, that would be my preferred platform, but I’m willing to learn.

Could you recommend a trading platform that would give me access to something like that? I’m an Australian citizen / resident, so preferably something local or with an online-only option would be ideal. Also, it would be good if I could do some “pretend” trading on the platform first to fine-tune any trading algorithms I come up with.

Thanks in advance for your feedback.

EDIT: One more thing, I should add that if I train the algorithm on data prior to the GFC, and then ask it to predict performance after the crash, it does very poorly (accuracy of about 50% - ie. resembling random guesses). So basically if I had used the algorithm blindly through the GFC, I would basically have been flapping in the wind for a while until it was able to recognise what was happening and adjust (using hourly data, this means up to a year. On by-the-second data hopefully only a matter of weeks or days).

So part of the reason me wanting to do this on a trading platform “demo” account is so that I can test some basic code that will compare the volatility, volumes and behavior of the current market to the data that my neural network has been trained on, so that it can simply “call it off” if there is a discrepancy and the neural net is likely to fail. I could do a lot of this on my own using historic data, but it makes more sense to start building it on an actual platform that I can then switch over to a “live” account when it’s at a stage where it starts working…

60% on out of sample data sounds quite good. Can you put your neural network system in a DLL? Then you can use the Zorro platform for backtests and for demo trading.

Mql5/metatrader 5 is supposed to be pretty advanced and can too work with DLLs.

Take a look at MultiCharts.NET if you’re using VS2008 or VS2010, it’s fully compatible. It should have everything you need.

If you are familiar with C++ I would recommend MQL5

Just by going through example code provided with MetaTrader I think you can quickly find a way to adapt your algorithm or maybe directly use your dll.

Cool. Sounds great, I’ll check out those platforms and try to come up with a DLL.

I’ll let you know how it goes.

Isn’t Ninja Trader also C# compatible?, maybe not java script but C# and Java are very similar.

Ninja Trader has some issues with it’s data management and backtesting is pretty slow. At least that’s my experience.

Thats a shame, what is the best C# platform to use in your opinion?, I looked at MultiCharts but it is way to expensive for me.

MultiCharts.NET is only $1000 whereas the original MultiCharts is $1500. Personally, I prefer MultiCharts.NET way more than the original, and the price just happens to be better.

I mean, in the end, you get what you pay for. I would say MC.NET will be able to do anything an algorithmic trader needs/wants and more.