Recent HST format changes

New comment
 
I have been writing a HST file reader, that I have confirmed to work with an older HST file ( originating from an older version of MT4 ). But since LiveUpdate updated my MT4 build to 225, I don't seem to be able to read the header properly, and not any of the RateInfo.

I have quoted the docs on this topic below, which doesn't seem to be describing this newer HST file. So, may I know what's the latest HST format ? Also, I have attached an image, showing that the file contains an additional field containing CompChart entry, which is also not described below.



Historical File Format (HST Files)

The database header is the first

struct HistoryHeader
{
  int               version;            // database version
  char              copyright[64];      // copyright info
  char              symbol[12];         // symbol name
  int               period;             // symbol timeframe
  int               digits;             // the amount of digits after decimal point in the symbol
  time_t            timesign;           // timesign of the database creation
  time_t            last_sync;          // the last synchronization time
  int               unused[13];         // to be used in future
};

then goes the bars array (single-byte justification)

#pragma pack(push,1)
//---- standard representation of the quote in the database
struct RateInfo
{
  time_t            ctm;                // current time in seconds
  double            open;
  double            low;
  double            high;
  double            close;
  double            vol;
};
#pragma pack(pop)
 

?

 

Let me upload the file...


Rename .zip to .hst. Of course you can open it directly with a hex editor. But it certainly is not a zip file.

Files:
eurusd_1999.zip  92 kb
 

Well...

.

Your zip file is invalid, can't extract whatever is in it.

.

Try your reader on an HST file that MT4 creates.

My code was created a year ago, hasn't changed.

.

2009.09.06 14:52:35 Phy__HistoryHeaderDecoder USDCAD,M15: Volume = 444
2009.09.06 14:52:35 Phy__HistoryHeaderDecoder USDCAD,M15: Close = 1.371
2009.09.06 14:52:35 Phy__HistoryHeaderDecoder USDCAD,M15: Low = 1.3736
2009.09.06 14:52:35 Phy__HistoryHeaderDecoder USDCAD,M15: High = 1.3707
2009.09.06 14:52:35 Phy__HistoryHeaderDecoder USDCAD,M15: Open = 1.3724
2009.09.06 14:52:35 Phy__HistoryHeaderDecoder USDCAD,M15: Time = 1237779000
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Volume = 500
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Close = 1.3724
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Low = 1.3725
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: High = 1.3681
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Open = 1.3685
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Time = 1237777200
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Volume = 494
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Close = 1.3685
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Low = 1.3699
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: High = 1.367
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Open = 1.3676
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Time = 1237775400
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_unused = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: last_sync = 0
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: timesign = 2009.05.22 14:30:32
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: i_digits = 5
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: period = 30
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Equity = EURUSD
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: c_copyright = (C)opyright 2003, MetaQuotes Software Corp.
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: Version = 400
2009.09.06 14:52:34 Phy__HistoryHeaderDecoder USDCAD,M15: loaded successfully

 

As a matter of interest, what's your rationale for writing an hst file reader?

Would you not use the history center csv import/export capability to move OHLC data in and out of hst files?

You writing an elegant data cleanser or something?


CB

 

If you write your own history files, you might need to be able to see what you wrote.

I imported ticks (real time) from MBTrading and created various offline chart types from the data, tick charts, range bars, sub-minute time charts, etc.

An "offline" chart is a homebrew history file.

If you update the history, and auto-refresh the chart that reads the data, then offline becomes realtime.

.

http://ellipse.mine.nu/public/movies/mbt-five-second-2.wmv

.

upper chart looks like 5-seconds-bars, lower, some number of ticks, maybe 13 tick.

Console is collecting ticks on 29 pairs and creating 7 histories with each.

Other window is MBTrading "level II"

 

@ Phy,


As I said, the attachment is not a zip file. Please don't open it up with WinZip and such.


It's an HST file, not created by me, but downloaded by MT4 History Center. I just changed the extension in order to upload it here, so you may check it with your parser/hex editor.


The motivation of parsing a HST file ? LOTS. Flexibility in performing statistical analysis on the data in whatever way I want, while retaining the small database size.

 
I am still interested to hear any comments on this topic. TQ.
 

Hi,SleekX

Your file is NOT an HST-file.

It's a DAT-file.

When you start downloading historical data from History Center,

MT4 saves DAT-files in MetaTrader\history\downloads\EURUSD folder.

When downloading are finished,MT4 makes HST-files from these DAT-files, and saves in MetaTrader\history\Server-Name folder.

 

Oh, duh on me... not a zip file...

Sorry.

Can't try it without a reboot, for some reason my MT4 compiler gets in a state it will not work without a full reboot.

 
fai is right. I was trying to open a .DAT file. Thanks and sorry for the trouble.
12
New comment
Reason: