Why Do People Say That the Optimizer is Inaccurate

 

Hi,

I'm very serious about taking advantage of algotrading. However, the modelling quality of my bactest is 50% or less. Even with that number, I'm printing out the time of the tick and I'm getting it every 3 seconds or so. My strategy is based on hourly candlesticks and getting a price update every 3 seconds seems very accurate. This is with every tick mode. When I try it with control points, then I get an update every 2.5 minutes. So this seems very accurate too. So please tell me, why do people say that this is inaccurate and that you cannot create a profitable strategy with it?

Thanks 

 

There is only OHLCV. If you have M1 data (a big if) then everything that happens during that minute is simulated. No way to tell if the candle formed O-H-L-C or O-L-H-C.

Then there is no slippage (big difference) no delays in execution (big in lower TFs) stops are assumed to be hit exactly (news).

'Testing Features and Limits in MetaTrader 4' - MQL4 Articles

 
WHRoeder:

There is only OHLCV. If you have M1 data (a big if) then everything that happens during that minute is simulated. No way to tell if the candle formed O-H-L-C or O-L-H-C.

Then there is no slippage (big difference) no delays in execution (big in lower TFs) stops are assumed to be hit exactly (news).

'Testing Features and Limits in MetaTrader 4' - MQL4 Articles

All these things are true, but they are problems that one has to bear in mind when optimising in general, not specific problems with the MT4 Strategy Tester specifically.

Sure, the optimiser could be enhanced to simulate slippage (as some do) but then you are going to have a constant slippage or something which still isn't accurate. In fact it is no more accurate than not simulating it, just less optimistic. Maybe you could do something like have simulated slippage be a function of daily range, so when the market is moving fast the slippage gets bigger. That would be closer to what happens in real life, but still a simulation. Delays in execution are an interesting one, I suppose the optimiser could introduce delays of a random number of ticks or something, but as you said since ticks are generally being simulated, it's doubtful that would be meaningful.

Overall, then, these issues you have raised are all real issues with the process of backtesting in general, but they are not specific weaknesses of MT4's strategy tester. And yet MT4's Strategy Tester continues to have a lot of voices around the web saying it is inaccurate. Do you think it is just general problems being projected onto a specific product -- the price of popularity -- or do you think there are actual specific problems with MT4 in particular?

 
You answered your own question "real issues with the process of backtesting in general" I gave some of the specifics.
 
WHRoeder:

There is only OHLCV. If you have M1 data (a big if) then everything that happens during that minute is simulated. No way to tell if the candle formed O-H-L-C or O-L-H-C.

Then there is no slippage (big difference) no delays in execution (big in lower TFs) stops are assumed to be hit exactly (news).

'Testing Features and Limits in MetaTrader 4' - MQL4 Articles

Thanks! I'm so sorry that I'm now just getting back to you guys. Now that I have more insight, I will use it to my advantage.
 
mf456:

All these things are true, but they are problems that one has to bear in mind when optimising in general, not specific problems with the MT4 Strategy Tester specifically.

Sure, the optimiser could be enhanced to simulate slippage (as some do) but then you are going to have a constant slippage or something which still isn't accurate. In fact it is no more accurate than not simulating it, just less optimistic. Maybe you could do something like have simulated slippage be a function of daily range, so when the market is moving fast the slippage gets bigger. That would be closer to what happens in real life, but still a simulation. Delays in execution are an interesting one, I suppose the optimiser could introduce delays of a random number of ticks or something, but as you said since ticks are generally being simulated, it's doubtful that would be meaningful.

Overall, then, these issues you have raised are all real issues with the process of backtesting in general, but they are not specific weaknesses of MT4's strategy tester. And yet MT4's Strategy Tester continues to have a lot of voices around the web saying it is inaccurate. Do you think it is just general problems being projected onto a specific product -- the price of popularity -- or do you think there are actual specific problems with MT4 in particular?

Very useful information :)
 
BernWillChris:
Very useful information :)
The major issue (for some - mostly scalpers) i is one of granularity. ie the MT4  backtester cannot utilise tick data. 
 
rod178:
The major issue (for some - mostly scalpers) i is one of granularity. ie the MT4  backtester cannot utilise tick data. 

Yes, it can! You just have to generate the FXT data files yourself or use 3rd party tools like TickStory or Birt's Tick Data Suit (search for them on Google).

For multi-time-frame strategies, you will also have to generate the corresponding HST files so that they are in sync with the FXT files you produce with the tick data.

 
FMIC:

Yes, it can! You just have to generate the FXT data files yourself or use 3rd party tools like TickStory or Birt's Tick Data Suit (search for them on Google).

For multi-time-frame strategies, you will also have to generate the corresponding HST files so that they are in sync with the FXT files you produce with the tick data.

I generate all my M1 data from DK tick charts

Maybe you misunderstood my point, which was, as far as MT4 is concerned the tick data is lost forever ie it cannot determine what (ie ticks) are inside the M1 OHLC

 
rod178:

I generate all my M1 data from DK tick charts. Maybe you misunderstood my point, which was, as far as MT4 is concerned the tick data is lost forever ie it cannot determine what (ie ticks) are inside the M1 OHLC

Yes, it can see every single real tick data, provided you produce "Tick" FXT files and not "OHLC" FXT files!

Obviously, your code also has to use tick data provided by Ask, Bid or a MqlTick structure (provided by the SymbolInfoTick function) on every event call to OnTick() and not the Bar data.

I have been doing this for years, and so have many other users here. I work with real tick data and my EA's (and indicators) are tested against every single real tick.

 
FMIC:

Yes, it can see every single real tick data, provided you produce "Tick" FXT files and not "OHLC" FXT files!

Obviously, your code also has to use tick data provided by Ask, Bid or a MqlTick structure (provided by the SymbolInfoTick function) on every event call to OnTick() and not the Bar data.

I have been doing this for years, and so have many other users here. I work with real tick data and my EA's (and indicators) are tested against every single real tick.

OK, thanks

Do you scalp? 

Reason: