Backtesting with tick data

mikey 2010.10.01 14:48

I think I have found a good way of doing this in metatrader - and I have got a fair way through this method but have got STUCK and seek some help with the final stage.

1) Got tick data from a company (commercial source) symbol: CL (oil) free sources eg. dukascopy etc. dont have this symbol.
2) Used the company software to render this tick data in a bar format (without any loss of resolution). The bar format:

date, time, open, high, low, close, volume

Each bar has only 1 tick in it! So, for a given bar the open, high, low, close are all the same because just 1 tick in the bar.

3) I get this tick data into the metatrader strategy tester by going to the metatrader history folders and deleting any relevant files present in there. I open metatrader offline and I use the History centre and the import function to get the data in there. Of course, although it is tick data - with its bar format - it is accepted ok by metatrader. When use strategy tester (still offline of course) can access this data.

4) BUT the problem: According to metatrader, in this data there are multiple bars with the same date and time stamp (because metatrader ignores seconds; smallest it considers are minutes). I think the solution to this is to CHANGE the data so that all bars have different date and time stamps (at a resolution that can be picked up by metatrader). So, for instance, if
one bar has a time stamp of 18:00:00 and the next has a time stamp of 18:00:08 - then this second one should be changed to have a time stamp of 18:01:00. So, that it can be picked up as the next bar by metatrader. This would change all the time stamps (and the date stamps in due course) of course but I dont think this really matters. My question is - HOW CAN I CHANGE ALL THE DATE AND TIME STAMPS IN THIS WAY?

Of course, the data has too many entries for Excel. Have access to microsoft Access.

Maybe some script that would add 1 second iteratively to each time entry would work. But would have to change the hour accordingly as we went up and then the date of course. This is what gets me - makes me stuck. Dont know how to do this.

a sample of my bar based tick data is cut and paste at below
(it is CL data - sweet crude oil future)

date, time, open, high, low, close, volume
20100831,18:00:00,71.70,71.70,71.70,71.70,2
20100831,18:00:00,71.70,71.70,71.70,71.70,1
20100831,18:00:00,71.70,71.70,71.70,71.70,1
20100831,18:00:00,71.70,71.70,71.70,71.70,1
20100831,18:00:00,71.70,71.70,71.70,71.70,1
20100831,18:00:01,71.73,71.73,71.73,71.73,2
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.73,71.73,71.73,71.73,1
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.74,71.74,71.74,71.74,1
20100831,18:00:01,71.75,71.75,71.75,71.75,7
20100831,18:00:01,71.75,71.75,71.75,71.75,1
20100831,18:00:04,71.75,71.75,71.75,71.75,1
20100831,18:00:05,71.74,71.74,71.74,71.74,1
20100831,18:00:05,71.75,71.75,71.75,71.75,1
20100831,18:00:05,71.75,71.75,71.75,71.75,3
20100831,18:00:08,71.75,71.75,71.75,71.75,7
20100831,18:00:09,71.75,71.75,71.75,71.75,1
20100831,18:00:10,71.75,71.75,71.75,71.75,1
20100831,18:00:10,71.75,71.75,71.75,71.75,1
20100831,18:00:10,71.71,71.71,71.71,71.71,1
20100831,18:00:11,71.70,71.70,71.70,71.70,1
20100831,18:00:11,71.68,71.68,71.68,71.68,1
20100831,18:00:19,71.71,71.71,71.71,71.71,1
20100831,18:00:19,71.71,71.71,71.71,71.71,1
20100831,18:00:19,71.69,71.69,71.69,71.69,1
20100831,18:00:19,71.69,71.69,71.69,71.69,1
20100831,18:00:19,71.68,71.68,71.68,71.68,2
20100831,18:00:19,71.67,71.67,71.67,71.67,1
20100831,18:00:34,71.72,71.72,71.72,71.72,1
20100831,18:00:34,71.72,71.72,71.72,71.72,1
20100831,18:00:35,71.72,71.72,71.72,71.72,1
20100831,18:00:37,71.72,71.72,71.72,71.72,1
20100831,18:00:56,71.71,71.71,71.71,71.71,1
20100831,18:00:56,71.71,71.71,71.71,71.71,1
20100831,18:00:56,71.70,71.70,71.70,71.70,1
20100831,18:01:00,71.71,71.71,71.71,71.71,1
20100831,18:01:02,71.72,71.72,71.72,71.72,1
20100831,18:01:11,71.71,71.71,71.71,71.71,1
20100831,18:01:12,71.71,71.71,71.71,71.71,1
20100831,18:01:13,71.71,71.71,71.71,71.71,1
20100831,18:01:16,71.70,71.70,71.70,71.70,1
20100831,18:01:26,71.70,71.70,71.70,71.70,3
20100831,18:01:26,71.70,71.70,71.70,71.70,5
20100831,18:01:28,71.70,71.70,71.70,71.70,1
20100831,18:01:33,71.71,71.71,71.71,71.71,1
20100831,18:01:36,71.70,71.70,71.70,71.70,4
20100831,18:01:45,71.70,71.70,71.70,71.70,1
20100831,18:01:46,71.71,71.71,71.71,71.71,2
20100831,18:01:46,71.71,71.71,71.71,71.71,4
20100831,18:01:49,71.71,71.71,71.71,71.71,1
20100831,18:02:19,71.70,71.70,71.70,71.70,1
20100831,18:02:25,71.70,71.70,71.70,71.70,1
20100831,18:02:25,71.70,71.70,71.70,71.70,1
20100831,18:02:25,71.70,71.70,71.70,71.70,1
20100831,18:02:25,71.70,71.70,71.70,71.70,1
20100831,18:02:26,71.70,71.70,71.70,71.70,1
20100831,18:02:26,71.70,71.70,71.70,71.70,1
20100831,18:02:39,71.74,71.74,71.74,71.74,1
20100831,18:03:10,71.74,71.74,71.74,71.74,1
20100831,18:03:29,71.75,71.75,71.75,71.75,1
20100831,18:03:34,71.75,71.75,71.75,71.75,1
20100831,18:03:34,71.77,71.77,71.77,71.77,1
20100831,18:03:41,71.78,71.78,71.78,71.78,1
20100831,18:03:41,71.78,71.78,71.78,71.78,5
20100831,18:03:41,71.78,71.78,71.78,71.78,5
20100831,18:03:41,71.79,71.79,71.79,71.79,1
20100831,18:03:44,71.78,71.78,71.78,71.78,1
20100831,18:03:50,71.78,71.78,71.78,71.78,1
20100831,18:03:52,71.79,71.79,71.79,71.79,1
20100831,18:04:14,71.79,71.79,71.79,71.79,1
20100831,18:04:14,71.79,71.79,71.79,71.79,1
20100831,18:04:14,71.80,71.80,71.80,71.80,1
20100831,18:04:14,71.80,71.80,71.80,71.80,2

Note that I can actually render the data without seconds and only in minutes which might help with the solution.

20100831,18:00,71.70,71.70,71.70,71.70,2
20100831,18:00,71.70,71.70,71.70,71.70,1
20100831,18:00,71.70,71.70,71.70,71.70,1
20100831,18:00,71.70,71.70,71.70,71.70,1
20100831,18:00,71.70,71.70,71.70,71.70,1
20100831,18:00,71.73,71.73,71.73,71.73,2
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.73,71.73,71.73,71.73,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.75,71.75,71.75,71.75,7
20100831,18:00,71.75,71.75,71.75,71.75,1
20100831,18:00,71.75,71.75,71.75,71.75,1
20100831,18:00,71.74,71.74,71.74,71.74,1
20100831,18:00,71.75,71.75,71.75,71.75,1
20100831,18:00,71.75,71.75,71.75,71.75,3
20100831,18:00,71.75,71.75,71.75,71.75,7
20100831,18:00,71.75,71.75,71.75,71.75,1
20100831,18:00,71.75,71.75,71.75,71.75,1
20100831,18:00,71.75,71.75,71.75,71.75,1
20100831,18:00,71.71,71.71,71.71,71.71,1
20100831,18:00,71.70,71.70,71.70,71.70,1
20100831,18:00,71.68,71.68,71.68,71.68,1
20100831,18:00,71.71,71.71,71.71,71.71,1
20100831,18:00,71.71,71.71,71.71,71.71,1
20100831,18:00,71.69,71.69,71.69,71.69,1
20100831,18:00,71.69,71.69,71.69,71.69,1
20100831,18:00,71.68,71.68,71.68,71.68,2
20100831,18:00,71.67,71.67,71.67,71.67,1
20100831,18:00,71.72,71.72,71.72,71.72,1
20100831,18:00,71.72,71.72,71.72,71.72,1
20100831,18:00,71.72,71.72,71.72,71.72,1
20100831,18:00,71.72,71.72,71.72,71.72,1
20100831,18:00,71.71,71.71,71.71,71.71,1
20100831,18:00,71.71,71.71,71.71,71.71,1
20100831,18:00,71.70,71.70,71.70,71.70,1
20100831,18:01,71.71,71.71,71.71,71.71,1
20100831,18:01,71.72,71.72,71.72,71.72,1
20100831,18:01,71.71,71.71,71.71,71.71,1
20100831,18:01,71.71,71.71,71.71,71.71,1
20100831,18:01,71.71,71.71,71.71,71.71,1
20100831,18:01,71.70,71.70,71.70,71.70,1
20100831,18:01,71.70,71.70,71.70,71.70,3
20100831,18:01,71.70,71.70,71.70,71.70,5
20100831,18:01,71.70,71.70,71.70,71.70,1
20100831,18:01,71.71,71.71,71.71,71.71,1
20100831,18:01,71.70,71.70,71.70,71.70,4
20100831,18:01,71.70,71.70,71.70,71.70,1
20100831,18:01,71.71,71.71,71.71,71.71,2
20100831,18:01,71.71,71.71,71.71,71.71,4
20100831,18:01,71.71,71.71,71.71,71.71,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.70,71.70,71.70,71.70,1
20100831,18:02,71.74,71.74,71.74,71.74,1
20100831,18:03,71.74,71.74,71.74,71.74,1
20100831,18:03,71.75,71.75,71.75,71.75,1
20100831,18:03,71.75,71.75,71.75,71.75,1
20100831,18:03,71.77,71.77,71.77,71.77,1
20100831,18:03,71.78,71.78,71.78,71.78,1
20100831,18:03,71.78,71.78,71.78,71.78,5
20100831,18:03,71.78,71.78,71.78,71.78,5
20100831,18:03,71.79,71.79,71.79,71.79,1
20100831,18:03,71.78,71.78,71.78,71.78,1
20100831,18:03,71.78,71.78,71.78,71.78,1
20100831,18:03,71.79,71.79,71.79,71.79,1
20100831,18:04,71.79,71.79,71.79,71.79,1
20100831,18:04,71.79,71.79,71.79,71.79,1
20100831,18:04,71.80,71.80,71.80,71.80,1
20100831,18:04,71.80,71.80,71.80,71.80,2
20100831,18:04,71.80,71.80,71.80,71.80,1
20100831,18:04,71.80,71.80,71.80,71.80,3
20100831,18:04,71.80,71.80,71.80,71.80,17
20100831,18:04,71.80,71.80,71.80,71.80,1
20100831,18:04,71.80,71.80,71.80,71.80,4
20100831,18:04,71.79,71.79,71.79,71.79,1

gordon 2010.10.01 15:24 #1

Tester uses FXT files and these have seconds as well. Unfortunately these FXT files are overwritten on each successive test and so users cannot put their own data their (in old versions of MT4 it was possible -> https://www.mql5.com/en/forum/103180/page6#321081). There is a way to overcome this obstacle... See here for detailed info -> http://eareview.net/tick-data.

MT4 Backtesting - How Backtesting/Optimization New MetaTrader 4 Client

mikey 2010.10.01 15:29 #2

that website seems to be the standard answer to anything tick.

but what I am trying to do is something different (I have gone about it a different route becaase my tick data is NOT from the same source as that website assumes - it has a diffferent start point and so is not suitable substrate for the offered conversion files/systems).

Indeed with my method there is NO need to use an old metatrader version. If i am correct. I am almost there as well - i just need help with the last final stage,

To have the ability Strategic foresight systems Generation of uniformly distributed

mikey 2010.10.01 16:10 #3

What you think?

1) Am on a valid path?

As I say - i just cant use the path a la that website (Birt EA) as its not valid for my start point.

2) If so - have u any idea to get over this final sticking point I am on?

Cant add new product Hedge saved you, who SuperWoodieCCI

gordon 2010.10.01 17:56 #4

Either u didn't understand Birt's method or I don't understand u. The following makes no sense:

mikey:

[...]

That won't solve anything since the Tester will always first interpolate Ticks to backtest on, regardless of what u do in the history center. The main idea behind Birt's method is to replace the interpolated ticks file with a file containing real ticks for the Tester to test on. Even if u succeed in your current attempt, the Tester would still interpolate ticks to test on, so it makes no sense.

I recommend you read this -> https://www.mql5.com/en/articles/1511 and go over Birt's tutorial again. Your aim should be to create a proper FXT file with real ticks to replace the FXT file generated by the Tester. Using Birt's method you can disable "recalculation" and so make the Tester use your FXT file (with your real ticks).

Backtesting with 99 percent Data-Feed: This is Ridiculous 9.2 Billion profit in

mikey 2010.10.01 18:56 #5

It all hinges on this:

1) Does metatrader interpolate ticks WITHIN bars only? (I think it does)

2) And it does this as guided and constrained by the given values of the bar: open, high, low, close?

If [1] and [2] are true I think my method is valid because in my method each bar only has 1 tick. So, the open, high, low and close are all equal in value. So, when metatrader uses these to interpolate ticks - all the interpolated ticks are given exactly the same value/price. So, metatrader's interpolation of new ticks is not a problem - it doesnt change anything.

I think I do undertstand the issues and I think I might be on the path to what I believe is a better way of doing things. BUT I stand to be corrected. Maybe i am being naive.

My EA is a When would this statement Any Strategy Tester experts

gordon 2010.10.01 19:25 #6

mikey:

1) Does metatrader interpolate ticks WITHIN bars only? (I think it does)

2) And it does this as guided and constrained by the given values of the bar: open, high, low, close?

I see... Thanks for clarifying that. Well, if the bar has O=C=H=L and volume=1 then indeed 1 tick only should be formed. But that would mean that u are 'stretching' the timescale by the average bar volume. And u are stretching it 'uniformly' although different bars have different volumes. This should have severe affects on most experts. It would also mean that dates/times on testing reports would be stretched and that swap would be taken countless times more than it should. This sounds extremely unreliable and I think it just won't 'work' for most experts.

Anyway, to answer your question of 'how to do it'. Simply create an M1 HST file with each bar representing a single tick. This can be done via a script. HST file structure is documented in the manual (Terminal Help) and u can reuse some code from csv2fxt script.

p.s. If you are interested in how the interpolation is done in the Tester, there is a good article about how it's done in the MT5 Tester... Should be very similar (if not the same) to what's going on in the MT4 Tester -> https://www.mql5.com/en/articles/75.

Strategy Tester: fxt file Unexpected MT4 Strategy Tester function sleep alternative

mikey 2010.10.01 19:27 #7

As a further clarification to my last:

Each tick is given its own M1 bar (a bar with a unique date and time stamp, with time at the resolution of whole minutes).

So, this means the time gets shifted - the bar of a tick may have a very different time to when that tick actually happened.

But this is a NOT a problem for me. I dont need to know the time accurately. I know what data I put into the backtest (from this date to that date) and I shall know at the end if a profit or loss was made at the end of this period with this method. (the exact date/timings on the output graph dont concern me)

Again - I stand to be corrected. Is this all sound or madness? Please take the time to understand review what ive said rather than just refer me to Birts. because if i can properly get this across and relay it and it is sound - i think it might be a better way of doing things.

Bug in bar shift Indicators with alerts/signal RefreshRates()

gordon 2010.10.01 19:32 #8

mikey:

[...] But this is a NOT a problem for me. I dont need to know the time accurately. I know what data I put into the backtest (from this date to that date) and I shall know at the end if a profit or loss was made at the end of this period with this method. (the exact date/timings on the output graph dont concern me) [...]

OK. But are u absolutely sure that this won't affect the expert? For example - any expert that relies on timeframe consideration or indi's that are based on timeframes won't work as its supposed to. You will also have to reset swap to zero, cause for sure it won't be calculated properly. There might be other side affects I'm not thinking of right now... Sounds very dangerous to me.

Increasing Signal Lot Size The market is a Apply the template programmatically?

gordon 2010.10.01 19:34 #9

BTW, u are going to have to scrap the volume data since it's irrelevant to the way it's calculated in MT4 (volume = number of ticks received in a bar).

mikey 2010.10.01 19:35 #10

Yep. your reply just came up as I sent my last.

RE: volume...- if my EA doesnt "read" volume am I safe on this? Dont have to worry about that? I didnt think about volume - but then as I say my EA doesnt "read" it for anything. So, just to clarify - i can ignore volume? Or should I maybe set volume to 1 for all bars?

Elite indicators :) Getting previous histogram information discount coupon for the

1 2 3 4

New comment