different backtesting results depending on dates tested

New comment
 
So here's the deal. Today is 2-19-10. If I start the backtest from 1-1-10 to today, it trades well, no major losses, looks good. But then if I trade from 12-1-09 (or any date earlier than mid december) to today, it gives this huge loss on 1-10-10. I didn't change anything else except the dates it's testing (still using M1). What's going on here? My strategy is even trading is a series: one thing happens, trade, another things happens, up/down the lots, etc. When I test from 1-1-10, it doesn't even get close to the end of that series, but when I start 12-1-10 it runs through the whole thing (7 stages) and ends with a loss. How is this possible? It's not like it's a tick off here or there, it's way off. If you use a longer timeframe, does MT4 make different averages or whatever to determine tick data? Is this just horrible data to use? I downloaded the historical data from MetaQuotes. Is it switching to that when I go back further? Pretty confused. I would really like to backtest this strategy, but I don't feel like I can trust the tests at all now. This is my first time testing with MT4, though i've backtested futures and stocks with several other programs and data sources. Never seen anything like this. Thanks all.
 

nebeno wrote >>
If I start the backtest from 1-1-10 to today, it trades well, no major losses, looks good. But then if I trade from 12-1-09 (or any date earlier than mid december) to today, it gives this huge loss on 1-10-10. I didn't change anything else except the dates it's testing (still using M1). What's going on here?

Did the account have the EXACT same conditions on 1-1-10 in both cases...? No. So why would u expect the same result?

My strategy is even trading is a series: one thing happens, trade, another things happens, up/down the lots, etc. When I test from 1-1-10, it doesn't even get close to the end of that series, but when I start 12-1-10 it runs through the whole thing (7 stages) and ends with a loss. How is this possible?

Why not? It's testing on different dates/different data.

If you use a longer timeframe, does MT4 make different averages or whatever to determine tick data?

Nope. Tick data is (almost) the same regardless of the time-frame chosen. See here -> https://www.mql5.com/en/forum/121630. If your EA relies on functions that are affected by the time-frame chosen (High[], Low[], iMACD(), etc.), then it might be affected, otherwise no.

Is this just horrible data to use? I downloaded the historical data from MetaQuotes.

Yes, that data is indicative data (an average of many sources) and should not be used with experts that rely on inner M1-bar movements. You are better off downloading/collecting data from YOUR broker.

 

Most likely, static/global variables are getting set incorrectly between 12-1 and 1-1 so it misbehaves on 1-10, while starting on 1-1 hasn't triggered the bug.

 
nebeno:
So here's the deal. Today is 2-19-10. If I start the backtest from 1-1-10 to today, it trades well, no major losses, looks good. But then if I trade from 12-1-09 (or any date earlier than mid december) to today, it gives this huge loss on 1-10-10. I didn't change anything else except the dates it's testing (still using M1). What's going on here? My strategy is even trading is a series: one thing happens, trade, another things happens, up/down the lots, etc. When I test from 1-1-10, it doesn't even get close to the end of that series, but when I start 12-1-10 it runs through the whole thing (7 stages) and ends with a loss. How is this possible? It's not like it's a tick off here or there, it's way off. If you use a longer timeframe, does MT4 make different averages or whatever to determine tick data? Is this just horrible data to use? I downloaded the historical data from MetaQuotes. Is it switching to that when I go back further? Pretty confused. I would really like to backtest this strategy, but I don't feel like I can trust the tests at all now. This is my first time testing with MT4, though i've backtested futures and stocks with several other programs and data sources. Never seen anything like this. Thanks all.


That problem of SPREAD.

It change frequently by few seconds then it will affect you result.

 
okay, I've been reading more around the forums here. i'm giving up on the idea of getting really meaningful results from backtesting. i can test for general bugs, general strategy, but not much for specific results. oh well. thanks for the replies all.
 

... i'm giving up on the idea of getting really meaningful (specific) results from backtesting...

Thats the way to do it - you are actually looking to fail the strategy by backtesting

You cannot validate it that way, for all the reasons you will find on the forum..

Good Luck

-BB-

 

Seconded. I use the strategy tester only for functional testing of my software.


CB

 
nebeno:
okay, I've been reading more around the forums here. i'm giving up on the idea of getting really meaningful results from backtesting. i can test for general bugs, general strategy, but not much for specific results. oh well. thanks for the replies all.

My opinion is that one is looking for specific results in that one wants profitable results from backtesting rather than huge losses. If you have losses, there is more that can be done at this backtesting stage.

 
cloudbreaker:

Seconded. I use the strategy tester only for functional testing of my software.

To relieve the doom and gloom slightly, it depends in part on the average duration of your trades. If your average holding period is 6 months, then the MT4 strategy tester is fine. If it's 6 minutes, then the strategy tester probably isn't reliable. I've seen MT4 back-tests which tallied with real-money trading over the same period to a margin of error of less than 1%.

 
Good posts that helped me understand backtesting a lot better.. Once again, thanks for the education.
 

With respects to everybody, I think u r overgeneralizing (IMHO).


I'll ask you all a simple question. Which one of the following is the reason u would not rely on the backtester (besides for debugging and functional testing):

a. MT4 Tester has many technical limitations which makes the results differ greatly from what would have happened in a Live account.

b. Backtesting (whether in MT4 Tester or not) does not give any useful information about the performance of experts in the FUTURE.


(if u have completely other reasons... then just add them).

12
New comment
Reason: