How can the backtester be fixed?

 

When I do backtests and get highly implausible results, including trades being executed at prices outside of the range at the time in question, I am drawn to the strong suspicion that there is at least one serious bug in the backtester.

I believe part of this error is that somehow imported data gets garbled for no obvious reason.

Example

A fresh installation of metatrader, a downloaded 10 years of 1 min data imported and converted to other time frames using the period_converter script included with the software.

Looking at July 21 at 1215 I see the following history data ranges


15 min 1.42150 1.42660

5 min 1.43804 1.43701


Obviously, the 5 minute range should be a subset of the 15 minute range. [I noticed this example when I found I could not make any sense of what was happening to the indicators on different charts at this time]


It's all too easy to wave hands and say "Oh, the data got corrupted". But this is not some cosmic ray hitting a memory location, this is some sort of bug in the software, in my opinion.


So, what should be done about it?

 
Elroch:


Example

A fresh installation of metatrader, a downloaded 10 years of 1 min data imported and converted to other time frames using the period_converter script included with the software.

Looking at July 21 at 1215 I see the following history data ranges

15 min 1.42150 1.42660
5 min 1.43804 1.43701

So, what should be done about it?

Perhaps you should start by giving all the relevant information for your example.

where did you get your data from ?

what pair ?

what year ?

 
More intresting, how did you convert the M1 data into the different timeframes?
 
zzuegg:
More intresting, how did you convert the M1 data into the different timeframes?
Are you wondering how period converter stuffed up converting M1 data into M5 and M15 as per the example ?
 

Could be a possible reason if the different timeframes do not show the same range. Neverless, the strategy tester should trow a lot of mismatched chart errors.

I don't believe this is a strategy tester bug, simply because i have consistent backtest results, and the forum here would be full of posts with that issue.

@Elroch: i can offer you a tick test with dukascopy data if you want.

 
Elroch:


Example

A fresh installation of metatrader, a downloaded 10 years of 1 min data imported and converted to other time frames using the period_converter script included with the software.

Looking at July 21 at 1215 I see the following history data ranges


15 min 1.42150 1.42660

5 min 1.43804 1.43701

Obviously, the 5 minute range should be a subset of the 15 minute range. [I noticed this example when I found I could not make any sense of what was happening to the indicators on different charts at this time]

I've just downloaded EURUSD, cleared all my EU data from the History centre for EURUSD and also any EURUSD history files from my C:\Program Files (x86)\MBT MetaTrader 4\history/MBTrading-Demo Server directory, then using the History Centre I imported the EURUSD M1 data, then used period_converter to make the following periods one by one, M5, M15, M30, M60 (H1), M240 (H4) . . . I then imported these, one by one, using the History Centre. I double clicked the relevant period under EURUSD and imported the relevant HST file . . . I did this for M5, M15, M30, M60 (H1), M240 (H4)

While I was doing ALL of this my Terminal was disconnected . . . and still is.

I don't see the problem you have with M5 12:15 21st July 2011 EURUSD

 

I would agree that this could be more of a bug in the data management part of the software than the backtester. But the result is that after performing perfectly normal actions, it is possible to get nonsense results from the backtester without the slightest indication that there is anything wrong.

In answer to the questions.

(1) I used forex data, presumably the same as RaptorUK, although I downloaded mine at the start of this month. As I pointed out in another thread this data has a bad gap at the start of July, but this is not something that should affect the content of bars weeks later in my opinion. The backtester obviously has to be able to cope with data with gaps in it, even if they are usually only at the weekend. Also, an important point is that I observed the problem with wildly inconsistent backtest results when I was only using data up to the end of June.

(2) To convert, my procedure has been to make sure the max bars options are 99999999 and 99999999, then apply the period converter to a fresh 1 minute chart with multiplier 5. Then I do the same with a fresh 5 minute chart, with multiple 3, and so on. At each stage I check that I have the expected number of records in history centre for each time frame. My belief is that this should work if the period converter does what it says on the tin, and it is more efficient than converting from the 1 minute chart in all cases. If the conversions work, later conversions should also work. Obviously in the case of 5 min versus 15 minute, only 2 conversions are relevant. On earlier occasions, I had done conversions all from the 1 minute chart.

@Raptor, but the terminal being offline, do you mean that you are actually disconnected from the server? When I try this, period_converter does not work at all: it seems to need to be connected.

A minor question - what is the second number separated by a slash in history centre (eg 0/2047). The first seems to be clearly the number of records, but the second I am not sure about.

@zzueg, I am interested in any data that will help me do reliable backtesting. So far I have only used the forex stuff, initially believing their claims about it being "high quality".

 

http://eareview.net/tick-data

i suggest using the method using jForexdemo, calculate some time in to download the data, and some space on your harddisk too.

@Raptor: Your data seems also to have some strange things. Volume seems constant. Since MT4 uses that data to generate the ticks make sure you don't accidentally exploit such things. Also any indicator relying on Volume[] will most likely be faulty.

 
Another puzzle I have is that often I have applied period converter to a chart, then when I examine the history centre there is no data for the converted period. But on another occasion, precisely the expected number of records are there. In the first case, usually closing the software and restarting results in the right amount of data being visible in the history centre. My understanding is that period_converter creates a HST file from the data for the shorter chart, but I am not sure exactly what is happening to explain this variation in behaviour.
 
Elroch:

@Raptor, but the terminal being offline, do you mean that you are actually disconnected from the server? When I try this, period_converter does not work at all: it seems to need to be connected.

A minor question - what is the second number separated by a slash in history centre (eg 0/2047). The first seems to be clearly the number of records, but the second I am not sure about.

At the bottom right of my terminal I have red bars and it says "No connection" . . . I do this by "logging in" with an invalid Login account number.

I suspect that your data is being overwritten by data that is being downloaded from your Broker . . .

Not sure what the / is, I don't get it now when I check the History Centre.

@zzuegg

I don't use the Volume at all . . . it's a good point though. I'm not doing any kind of performance testing at the moment, just debugging/testing for correct functionality of my code. I did plan to look at the stuff in eareview to get some better data, just not needed it yet.

 
@Raptor, when I disconnect in exactly the same way you described (using an invalid account like 999999), I get "Invalid account" bottom right. However, on the next startup, I can get "No connection" if I simply press cancel when the login dialog appears on startup.
Reason: