Back testing... or testing backwards... or has Alice slipped some potion into my tea, not sure of anything anymore !!!
Unfortunately this seems to be the case, backtesting is a rough idea of if your system is profitable. In the past I have run 4 different machines through the same data and got 4 different results, then took an average.
Forward testing is the only way IMO to get reliable feedback. Also you can backtest and manually confirm each signal, but still it is not as accurate as the real deal.
Can you detail the steps involved in "downloading history"?
Where from and how? Did you check what has been downloaded, whether the number of Bars are the same with the same start/end date? Did you did do that for all the timeframes referenced?
GET THIS...
Over the last week I've put some effort into a new EA - Targeting GBPUSD on M15. Ran it on my laptop in back test and got staggering results. I MEAN S..T..A..G..G..E..R..I..N..G..
I was so impressed that I decided to do some optimisation testing just for my own sanity's sake. I dug out the 2 old PC's I'd been given from work when they were throwing them out.
I installed the terminal on both, downloaded the history on both, copied my Indicators & EA on to both, copied my main settings file onto both and ran them through to make sure all was fine & dandy.
Hmmm.... Interetsing - now bear in mind EVERYTHING is the same, even the 2 PC's are IDENTICAL pieces of kit, freshly installed, freshly set-up with my code, both modelled with Every Tick, checked the accounts were OK (both new demo acocunts set up today) and all spreads were the same. The EA's and indicators are fine & re-compiled correctly.
The one PC showed a confirmation of my initial results. I saw a fantastic return on my EA, the second PC didn't. The second one showed a tiny TINY profit.
NOW - I understand that some "in-between" values are estimated by the tester, but seriously - what the heck is going on. The results are MASSIVELY different. I would expect some deviation, but O/C/H/L are supposed to be hit - right ? - so if the history is the same the O/C/H/L values are the same surely. My EA doesn't do anything until a new bar open's in respect of opening new orders, and stop losses are in place, which should get triggered on both systems the same way if the "Low" of the bar is visited correctly by the tester. So what's going on ?...
I decided to delete & download the history again onto PC 2. Guess what.... Yes... The second PC now shows results the same as the first. Hmmm......
Deleted & downloaded again - back to TINY profit...
So, are we the butt of some sick joke ?...
Does anyone have any reliable source for history ?...
If they do can we really trust the terminal to process it correctly ?...
Been there... done that... way more often than I care to remember.
I consdered the possibility that we need a "perfect representative data set", to test against, but that wouldn't work, would only result in optimising against a
a perfect data set and in the variability of real markets we would get a rude awakening.
Now I test on several broker platforms and several time sequences and look for "robust results" that is to say the ea should be good on different platforms and should not be effected by minor variations in parameters. For example if I use a 21ema for trend direction then the results hould be just about as good using the
20ema or the 22ema. If I have to use a particular set of parameters or time sequence it means that I have not found a reliable "system" just found one very good very unique solution for one set of data which will never occur again.
One thing I like to do is download the tester results to excel and look for broad ranges of successful parameters.
Needless to say; since I am still here, I haven't succeeded in finding a "robust system".
Keith
GET THIS...
Over the last week I've put some effort into a new EA - Targeting GBPUSD on M15. Ran it on my laptop in back test and got staggering results. I MEAN S..T..A..G..G..E..R..I..N..G..
I was so impressed that I decided to do some optimisation testing just for my own sanity's sake. I dug out the 2 old PC's I'd been given from work when they were throwing them out.
I installed the terminal on both, downloaded the history on both, copied my Indicators & EA on to both, copied my main settings file onto both and ran them through to make sure all was fine & dandy.
Hmmm.... Interetsing - now bear in mind EVERYTHING is the same, even the 2 PC's are IDENTICAL pieces of kit, freshly installed, freshly set-up with my code, both modelled with Every Tick, checked the accounts were OK (both new demo acocunts set up today) and all spreads were the same. The EA's and indicators are fine & re-compiled correctly.
The one PC showed a confirmation of my initial results. I saw a fantastic return on my EA, the second PC didn't. The second one showed a tiny TINY profit.
NOW - I understand that some "in-between" values are estimated by the tester, but seriously - what the heck is going on. The results are MASSIVELY different. I would expect some deviation, but O/C/H/L are supposed to be hit - right ? - so if the history is the same the O/C/H/L values are the same surely. My EA doesn't do anything until a new bar open's in respect of opening new orders, and stop losses are in place, which should get triggered on both systems the same way if the "Low" of the bar is visited correctly by the tester. So what's going on ?...
I decided to delete & download the history again onto PC 2. Guess what.... Yes... The second PC now shows results the same as the first. Hmmm......
Deleted & downloaded again - back to TINY profit...
So, are we the butt of some sick joke ?...
Does anyone have any reliable source for history ?...
If they do can we really trust the terminal to process it correctly ?...
Can you detail the steps involved in "downloading history"?
Where from and how? Did you check what has been downloaded, whether the number of Bars are the same with the same start/end date? Did you did do that for all the timeframes referenced?
I appreciate where you're coming from, but I'm here to write EA's and develop a trading strategy.
Sorry... but I'm not here to do software testing on a product that should be "telling me" when things didn't download properly or when my data is corrupt.
I can't believe that something as straightforward as a set of historic data files could be so problematic... After all its kind of fundamental to the process that we get accurate and reliable test data. I know all of the pro's and con's of trusting back testing and the results you get from it. But this is software development 101. Repeatable results (allowing for random tick patterns of course) should be a given from a tool like this. I should be able to refresh whatever random tick data I may have and request the terminal re-do it.
We should be able to get "in the ball park" each time with such testing and be fairly close to similar results each time. It's bizaare to say the least.
GET THIS...
Over the last week I've put some effort into a new EA - Targeting GBPUSD on M15. Ran it on my laptop in back test and got staggering results. I MEAN S..T..A..G..G..E..R..I..N..G..
I was so impressed that I decided to do some optimisation testing just for my own sanity's sake. I dug out the 2 old PC's I'd been given from work when they were throwing them out.
I installed the terminal on both, downloaded the history on both, copied my Indicators & EA on to both, copied my main settings file onto both and ran them through to make sure all was fine & dandy.
Hmmm.... Interetsing - now bear in mind EVERYTHING is the same, even the 2 PC's are IDENTICAL pieces of kit, freshly installed, freshly set-up with my code, both modelled with Every Tick, checked the accounts were OK (both new demo acocunts set up today) and all spreads were the same. The EA's and indicators are fine & re-compiled correctly.
The one PC showed a confirmation of my initial results. I saw a fantastic return on my EA, the second PC didn't. The second one showed a tiny TINY profit.
NOW - I understand that some "in-between" values are estimated by the tester, but seriously - what the heck is going on. The results are MASSIVELY different. I would expect some deviation, but O/C/H/L are supposed to be hit - right ? - so if the history is the same the O/C/H/L values are the same surely. My EA doesn't do anything until a new bar open's in respect of opening new orders, and stop losses are in place, which should get triggered on both systems the same way if the "Low" of the bar is visited correctly by the tester. So what's going on ?...
I decided to delete & download the history again onto PC 2. Guess what.... Yes... The second PC now shows results the same as the first. Hmmm......
Deleted & downloaded again - back to TINY profit...
So, are we the butt of some sick joke ?...
Does anyone have any reliable source for history ?...
If they do can we really trust the terminal to process it correctly ?...
I second what you wrote above. The problem is not with the data itself, etc. but with the defect in the tester. The tester still downloads history data even though there is enough data available in the history files. The tester downloads the data from your broker and this doesn't match data downloaded in history center. This is the real problem and no one seems to notice that. Your data basically gets corrupted every time you run the test. Check file sizes and timestamps. The way the tester handles historical data doesn't make any sense. I wrote another post yesterday on the same subject, please check this out. What I'm gonna try next is to delete broker account after downloading historical data, perhaps it will make things better. And of course you should have a dedicated MT4 instance just for testing.
Lukasz
Sounds like bad history data to me too. And I agree with some of the other commentors that the tool is a little weak at pulling in historical data that is needed for a backtest. Some of the problem may be the interaction with the borker's data sherver - not sure about that. Anyway, here's what I do to make sure I have good data for back test - copied from an earlier post that I commented on for someone else:
-------------------------------------
I do the following on the timeframe that my EA is testing on and each timeframe below - all the way down to 1 minute:
1) Open a chart with the currency you plan to test.
2) Select the highest time frame that you plan to use.
3) make sure you de-select the Auto Scroll button in your tool bar.
4) Repeatedly press the Home key on you keyboard until you a) get enough data to test with or b) stop making progress - that's all your broker has stored.
5) Select the next time frame down and repeate the above steps.....
6) keep doing this until you have collected lots of data - it takes a while
By the end of the exercise you will have a pretty good idea for how much data you have on all time frames - a measure for how good your modeling quality will be in the end. This will give you test date boundaries to work with.
Alternatively, you could go out to Alpari or some place like that, download history data, and re-direct your teminal to that data. I've never done that, but I see people discussing it regularly. If you did something like that then you would probably want a separate installation of MetaTrader for that purpose.
Hope this is helpful information.
Good luck,
Tovan
Sounds like bad history data to me too. And I agree with some of the other commentors that the tool is a little weak at pulling in historical data that is needed for a backtest. Some of the problem may be the interaction with the borker's data sherver - not sure about that. Anyway, here's what I do to make sure I have good data for back test - copied from an earlier post that I commented on for someone else:
-------------------------------------
I do the following on the timeframe that my EA is testing on and each timeframe below - all the way down to 1 minute:
1) Open a chart with the currency you plan to test.
2) Select the highest time frame that you plan to use.
3) make sure you de-select the Auto Scroll button in your tool bar.
4) Repeatedly press the Home key on you keyboard until you a) get enough data to test with or b) stop making progress - that's all your broker has stored.
5) Select the next time frame down and repeate the above steps.....
6) keep doing this until you have collected lots of data - it takes a while
By the end of the exercise you will have a pretty good idea for how much data you have on all time frames - a measure for how good your modeling quality will be in the end. This will give you test date boundaries to work with.
Alternatively, you could go out to Alpari or some place like that, download history data, and re-direct your teminal to that data. I've never done that, but I see people discussing it regularly. If you did something like that then you would probably want a separate installation of MetaTrader for that purpose.
Hope this is helpful information.
Good luck,
Tovan
Tovan,
The above method would use the broker's data which is not always reliable. If you import your own data or download from MetaQuotes, the problem remains. The tester downloads data from the broker, corrupting already downloaded data form the other source. I thought that you could remove that broker's account before running the test. It doesn't work, as you won't be able to select a currency pair for your backtest. It is simply broken...
The only method I found that works, is to temporarily disconnect computer from the network before running the test. It will be slower than usual but there won't be any corrupted data. It seems that MetaQuotes is not interested to fix this.
Lukasz
Dear Friends, G'day
I don't whether this is right thread to post.. Sorry, if it is wrong thread.
Can anyone help me here? I want one simple Indicator where that indicator is 2 horizontal lines trails every minute with reference to low & high of the same day. I use this one as a reversal lines. Is there anything like?
Keep Charging
Be Awesome,
Raj,
DreamJOBZ.com
Tovan,
The above method would use the broker's data which is not always reliable. If you import your own data or download from MetaQuotes, the problem remains. The tester downloads data from the broker, corrupting already downloaded data form the other source. I thought that you could remove that broker's account before running the test. It doesn't work, as you won't be able to select a currency pair for your backtest. It is simply broken...
The only method I found that works, is to temporarily disconnect computer from the network before running the test. It will be slower than usual but there won't be any corrupted data. It seems that MetaQuotes is not interested to fix this.
Lukasz
If you ever want to trade for real then you've got to use the broker's data at some point. I say bite the bullet now and start using it in your backtesting. Once it's there it doesn't change (that's my experience with IBFX anyway). It's not always perfect, but that's life. I prefer to test in an environment that's as close to real trading as possible so I know that I can expect something close when I get to the forward test phase. In my opinion, the only reason to go out and download someone else's data is if you want to test over a period where your broker doesn't have the data available.
These are my opionions - take them for what they're worth.
- Tovan
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
GET THIS...
Over the last week I've put some effort into a new EA - Targeting GBPUSD on M15. Ran it on my laptop in back test and got staggering results. I MEAN S..T..A..G..G..E..R..I..N..G..
I was so impressed that I decided to do some optimisation testing just for my own sanity's sake. I dug out the 2 old PC's I'd been given from work when they were throwing them out.
I installed the terminal on both, downloaded the history on both, copied my Indicators & EA on to both, copied my main settings file onto both and ran them through to make sure all was fine & dandy.
Hmmm.... Interetsing - now bear in mind EVERYTHING is the same, even the 2 PC's are IDENTICAL pieces of kit, freshly installed, freshly set-up with my code, both modelled with Every Tick, checked the accounts were OK (both new demo acocunts set up today) and all spreads were the same. The EA's and indicators are fine & re-compiled correctly.
The one PC showed a confirmation of my initial results. I saw a fantastic return on my EA, the second PC didn't. The second one showed a tiny TINY profit.
NOW - I understand that some "in-between" values are estimated by the tester, but seriously - what the heck is going on. The results are MASSIVELY different. I would expect some deviation, but O/C/H/L are supposed to be hit - right ? - so if the history is the same the O/C/H/L values are the same surely. My EA doesn't do anything until a new bar open's in respect of opening new orders, and stop losses are in place, which should get triggered on both systems the same way if the "Low" of the bar is visited correctly by the tester. So what's going on ?...
I decided to delete & download the history again onto PC 2. Guess what.... Yes... The second PC now shows results the same as the first. Hmmm......
Deleted & downloaded again - back to TINY profit...
So, are we the butt of some sick joke ?...
Does anyone have any reliable source for history ?...
If they do can we really trust the terminal to process it correctly ?...