[TUTORIAL] Complete Backtesting and Analysis Setup (100% Free)

Trading Systems Expert Advisors and Automated Trading
1

In this thread, I will be teaching readers one of my methods for performing accurate backtests of automated trading systems as well as post-test analysis. This method is great for someone who does not have much start up capital as it is [B]completely free[/B], yet still very powerful and comprehensive.

My aim is to help readers set up a complete system development infrastructure. By the end of the tutorial, one will be able to very easily develop and test trading systems with a high efficiency.

The setup process is pretty straightforward:

  1. Download tick data
  2. Format the data so it can be used in our testing
  3. Streamline steps 1 & 2
  4. Further formatting so we have the data required to test various styles of algorithms
  5. The actual backtesting
  6. Analysis

It primarily focuses around MetaTrader 4 but also incorporates other software designed around MT4 that are easily accessible for any individual (as well as a small piece of Python code I wrote a couple days ago).

This thread will cover:
[ul]
[li]Setup and Tools Required
[/li][li]Getting Tick Data
[/li][li] Formatting Tick Data for MT4
[/li][li]Importing Data
[/li][li]Converting to Various Time Frames
[/li][li]Example Backtest
[/li][li]Backtest Result Analysis Technique
[/li][li]Portfolio Backtest and Analysis
[/li][/ul]

In the following posts I will start discussing each in detail. I recommend readers to read this thread in order as it is step-by-step.

Note: I am not endorsed by any of the companies or products I am discussing. In no way do I intend to sell anything. I’m in a giving mood today.

If you’d like to collect your own tick data from any broker via MetaTrader 4, visit this thread:

7 Likes
2

Before we do anything, let’s start with gathering all the tools and getting the preliminary setup covered.

The resources we will need is:
[ul]
[li]Python 3
[/li][li]Dedicated MT4 Client
[/li][li]StrategyQuant’s Tick Data Downloader
[/li][li]StrategyQuant’s EA Analyzer
[/li][li]D2MT4.py Script
[/li][li]Period_Converter_Opt MT4 Indicator
[/li][/ul]

I will talk about what each one does when we get there. For now, let’s start getting these and set up.

Edit: Babypips isn’t letting me make posts right now; seems to be buggy, so it may be a bit before I can keep posting.

3

Python 3
Python 3 is a programming language, it must be installed on your computer in order for the D2MT4.py script to work.

Download link:
Download Python

Note: For 32-Bit users download “Python 3.3.0 Windows x86 MSI Installer”. For 64-Bit users download “Python 3.3.0 Windows X86-64 MSI Installer”.

The installation process is quite straightforward; just follow the installation wizard, you should have a Python directory after in the directory: C:\Python33.

MT4 Client
For our setup we’re going to be downloading a separate MT4 client in order to keep ourselves organize and avoid confusing our trading with our testing. Another reason is we want to be disconnected from any servers when we are performing our tests (more on this later). In this thread, I will be using MetaQuotes’ MT4 with no broker licenses.

Download link:
http://files.metaquotes.net/metaquotes.software.corp/mt4/mt4setup.exe

After installation is complete, I like to prepare the client in order to avoid future issues:

  1. Open up your MetaTrader 4 folder.
  2. Find “terminal.exe”
  3. Right click on it and select “Properties”.
  4. Under the “Compatibility” tab, make sure “Run this program as an administrator” is checked. Then press “OK”.

StrategyQuant’s Tick Data Downloader
StrategyQuant’s free software which allows users to download tick data from Dukascopy automatically. We’ll be using this to download our raw tick data. For more information: StrategyQuant Tick Data Downloader

Download link:
http://www.strategyquant.com/licenses/d?code=tdd

Note: DO NOT install this in your Program Files or Program Files(x86). The software has issues saving data in there. I would recommend saving it in My Documents or Downloads directory.

StrategyQuant’s EA Analyzer
Another free software from StrategyQuant which allows further analysis tools for MT4 Strategy Reports. For more information: StrategyQuant EA Analyzer

Download link:
http://www.strategyquant.com/licenses/d?code=eaa

Note: DO NOT install this in your Program Files or Program Files(x86). The software has issues saving data in there. I would recommend saving it in My Documents or Downloads directory.

D2MT4.py & Period_Converter_Opt
These two tools will assist us in formatting data. D2MT4.py was a piece of code I created for this tutorial and Period_Converter_Opt was an indicator released by “wfy05” on the MQL4 Codebase.

I have attached these two in this thread.

  1. Place “D2MT4.py” in the “tickdata” folder of your StrategyQuant’s Tick Data Downloader path. Ex) “C:\Users\Admin\Downloads\SQ_TickDataDownloader ickdata”

  2. Place the “Period_Converter_Opt.mq4” and “Period_Converter_Opt.ex4” files in the indicators folder of your MetaTrader 4 you just downloaded. Ex) “C:\Program Files (x86)\MetaTrader 4\experts\indicators”

Tick Data Tools.zip (12.8 KB)

4

First off, we need data to perform our testing on. There are many free sources online, but most have large gaps and missing quotes. Some are only End-Of-Day which is useless for intraday strategies.

Fortunately, Dukascopy has kept a large archive of tick and bar data available for free online. Unfortunately, their bar/OHLC data is garbage. Some bars are off by over 300+ pips on some pairs. BUT their tick data is surprisingly very accurate and detailed. Why the huge difference between the two; I have no idea.

What we want is tick data. Dukascopy’s online interface is ridiculously slow. At the same time, one must download tick data one day at a time, that would be amazing slow.

Luckily for us, StrategyQuant offers the Tick Data Downloader software which allows us to download tick data automatically. All one needs to do is to check the pairs they want and the starting and ending dates. “Start Download” will begin the export process. Tick Data Downloader will automatically start downloading the time series one symbol at a time. Depending on how much data you are downloading it can range from 20 seconds to 10+ days straight. Just make sure to keep your computer on and uninterrupted when the downloading is happening. The tick data is stored in the “tickdata” folder inside the installation directory. It is also where we saved our D2MT4.py script.

Again, if you want more information: StrategyQuant Tick Data Downloader

“tickdata” folder

If you open the tickdata folder as you start downloading data, you can see that each symbol creates a new folder, within the folder, additional folders are created for each year, then each month (0 to 11), then the day. Finally you will see “xxh_ticks.bi5” files. XX ranges from 00 to 23; you guessed it, those represent each hour of the day. Tick Data Downloader first exports these “.bi5” files first. Then it “Regenerates” these files and creates a “.csv” file. Ex) “EURUSD.csv”

The format of the CSVs are:
YYYY.MM.DD HH:MM:SS.mmm, bid, ask, bidcount, askcount

Keep this in mind for the next post, where we start formatting these files so they are ready to be imported.

5

Let’s first talk about how to import data into MT4.

Many people and resources have talked about .fxt and .hst files and converting tick data into these to be imported into MT4. In my opinion, that’s just too much work and takes way too long.

Luckily there is an easier method - History Center. Let’s open up our MT4 client that we just downloaded. When started up it prompts you to create an account as well as login. Just cancel out of those windows. We are working offline for our purposes for a reason. When you are connected to a data feed, the History Center is filled up with the data from the server. So when you import your own data, there will be times where your data will actually get replaced. This is something we must avoid. Which is why we are working offline, on a dedicated MT4 client.

MT4 History Center

By pressing F2 or going to Tools -> History Center, we get the History Center. You can see the symbols offered. By double clicking on the symbols you can see the different timeframes. You’ll notice that MT4 does not support tick data, but only OHLC bars. This is one of the disadvantages of MT4 strategy testing (no tick data support, no variable spread, no commission calculations, or slippage).

What we will do is actually format our Dukascopy tick data into MT4 accepted 1 minute OHLC data. Yes, it will lose some precision, but the way I see it is if you are using MT4 to develop strategies, you shouldn’t be looking at HFT strategies to begin with. 1 Minute bars will be more than accurate for 95% of strategies and testing needs.

Okay, let’s start!
Once we have our data fully exported via Tick Data Downloader from Dukascopy, you will see the CSV files named after the symbol inside your “tickdata” folder in the installation directory of Tick Data Downloader. What we need to do now is read this tick data and create a 1M bar file.

As an example I have downloaded data for EURUSD. The file “regenerated” by TDD is named “EURUSD.csv”. This is where the D2MT4.py comes into play. Let’s run it. Command prompt will open up it will ask for the filename and we will simply type “EURUSD” (without the quotations) and then press Enter. You will see that a new CSV is created “EURUSD_1m.csv”. Wait until the Command Prompt closes to know that the new file is completed.

6

Now let’s import this file into MT4. Open up MT4, again cancel the Account Creation and Login prompt upon starting the client. Press F2 or Tools -> History Center. Double click EURUSD and then make sure you have selected “1 Minute (M1)” and click “Import” and the lower right of your screen.

The settings are just the default since the Python script has ensured that it is exactly the right format that we need it to be in. Press Okay, once the settings are set correctly and you see a table of your 1 minute bars.

Congrats! You just imported your first data file.

Next post I will cover how to turn this into the other time frames so you are able to test your strategies on any time frame.

7
  1. Set “Max bars in history” and “Max bars in chart”.

Tools -> Options -> Charts Tab
Let’s set both to 10 000 000 just to be on the safe side.

  1. Open EURUSD chart, make sure it’s 1 minute time frame and then apply the Period_Converter_Opt. Leave all settings alone, except PeriodMultiplier. If you want a 1H chart, set PeriodMultiplier to 60, if you want 15 minutes set it to 15, Daily is set to 1440, etc. Double check OutputCSVFile is set to 2.

  1. Open up your MT4 installation folder. Go to experts -> files and you’ll find your CSV files you just generated. Now all you have to do is import this data the same way you did with the 1m chart.

:slight_smile: This completes the data importation process. I know it’s long and tedious, but it’s all worth it, and this is probably one of the faster ways with MT4.

Next few posts will show you how to backtest, and how we take advantage of the Strategy Report generated by MT4’s Strategy Tester.

8

Let’s do an example backtest with the MACD Sample EA provided by MT4 upon installation. It doesn’t matter if this EA is profitable, it’s just to show how to perform a backtest.

Open up your Strategy Tester window. View -> Strategy Tester

Select the “MACD Sample” for Expert Advisor. “EURUSD, Euro vs US Dollar” for Symbol. “Every tick” for Model (most accurate). I am using the M15 time frame as an example for Period. Check “Use date” and choose your From and To date. You can choose to check visual mode if you want. And then make sure to uncheck Optimization. Clicking “Expert properties” will allow you to view and change the parameters of the EA, for this example I have left it as default. Once you’re ready, just click Start and it will begin the backtesting process. Depending on your computer and the strategy, the test will vary in time.

After it is complete you are able to view the test’s results via the Results, Graph, Report and Journal Tab. If you go to the “Report” tab you can right click on it and “Save as report”. Let’s do that now; we will use it later in future posts.

Under the Report tab, we can see that mismatched charts errors should 0 or least less very low. Modelling quality does not matter at all in our case whether it’s 25% or 99%.

“[I]The reason is because of inconsistent number of ticks. The terminal tries to model the ticks within a bar and the low quality comes from consistency. In our case, Quality != Accuracy.[/I]” This was taken directly from a Skype chat I had with a colleague of mine. Do not be worried if your Modelling quality is sub-par.

Note: Do not worry about the performance of this EA, this is for demonstration only.

9

The MT4 Strategy Report shows some common and basic analysis of the strategy as well as the trades. But in my opinion, this isn’t enough. Let’s open up EA Analyzer by StrategyQuant. Remember that Strategy Report you saved as a .html? Well let’s import that file in EA Analyzer. Once it is loaded you can see much more information compared to the normal report.

I will let you play around with the program and find out what it offers.

10

Many traders would like to incorporate multiple trading strategies at once. One of the drawbacks is MT4 cannot do an entire portfolio backtest. This is where EA Analyzer can help us again. The process is simple. We run a backtest of each trading strategy on the time frames and symbols desired and we save the strategy reports for each one. EA Analyzer gives us the ability to merge multiple reports as one and do an overall analysis of all the trades in combination.

In this example, I ran a backtest of MACD Sample on the EURUSD 1 minute timeframe and 15 minute timeframe.

We add all backtest reports into EA Analyzer and at the bottom we check the reports we want combined in our portfolio, and then we click “Create Portfolio” which will create a portfolio and the analysis.

This concludes the main tutorial. I hope everyone learned something new. I will post more information as I can come up with it. As well as clean up the previous posts for any spelling/grammar errors as I tend to make a lot of those. I’ll also fill the posts with more detail later on, but this should be enough to get you started.

I look forward to feedback, comments and sharing ideas. :slight_smile:

11

Reserved to sub and read later

12

This is very useful Clark, thanks for sharing! Esp the data download and convert part.
For testing I’m still willing to use some programming language other than MT4

13

Thanx for sharing…subscribed! :slight_smile:

Always wondered how to do this…

14

Many Thanks ClarkFX, I will use this to try.

Regards !

15

Thank you Clark! This is indeed a good tutorial that I’m going to follow for sure! I need to backtest and optimize two strats that i couldn’t do with my broker’s data!

16

Thank you for this very informative tutorial. Great job!
Do you have any advice on the analysis itself? What are the main areas we should look at when analyzing an EA?
I am a beginner, but very much willing to learn how to develop reliable EAs…(for myself, of course). I am working on 1 EA now (but it is in early stage) and I will use this technique described by you to tune it, eventually (I hope I will be able to end the development process sometime…:slight_smile: ).
Thanks again.

17

Glad some people are finding it useful. It is a long read. I didn’t expect much people to go through it all. :slight_smile:

18

I have this working and backtesting the MACD ea :slight_smile:

ClarkFX, may I ask you how to get 5 digits data, if possible?

Thanks!

19

Great! I’m glad it’s working out. :slight_smile:

Dukascopy data source is in points not pips (0.1 pip increments). So it will already be in 5, or 3 for Yen crosses, digits. Just off the top of my head, the CSV files attempt to save some memory by making quotes that are whole pips only 4 or 2 digits. So 1.39510 will be 1.3951, but 1.67897 will still be 1.67897.

If I am off I will take a look after the long weekend. :slight_smile:

20

Hello,

In terms of analysis, there are various ways you could approach it. Generally, it will come with experience. Every trading system is unique, so I like to think that each system should be developed differently.

Typically, drawdown, return ratios are something to keep in mind. As well, testing in-sample versus out-of-sample data. I would also take a look at walk forward testing. These generally test for reliability of the system.

Again, there are much more techniques, more than a couple bookshelves of text can cover. For instance, something I occasionally look at is the z-score of the algorithm just to optimize money management strategies, more specifically equity curve optimization, of a particular trading system.

Hope this helps. :slight_smile: