Re: Read binary file with line delimeter
Hello Paul,
I run with debugger and appear this file refering the error to line 142 (in red):
Code:
/***
*xtoa.c - convert integers/longs to ASCII string
*
* Copyright (c) Microsoft Corporation. All rights reserved.
*
*Purpose:
* The module has code to convert integers/longs to ASCII strings. See
*
*******************************************************************************/
#include <cruntime.h>
#include <stdlib.h>
#include <limits.h>
#include <tchar.h>
#include <internal.h>
#include <internal_securecrt.h>
#ifdef _UNICODE
#define xtox_s xtow_s
#define _itox_s _itow_s
#define _ltox_s _ltow_s
#define _ultox_s _ultow_s
#define x64tox_s x64tow_s
#define _i64tox_s _i64tow_s
#define _ui64tox_s _ui64tow_s
#define xtox xtow
#define _itox _itow
#define _ltox _ltow
#define _ultox _ultow
#define x64tox x64tow
#define _i64tox _i64tow
#define _ui64tox _ui64tow
#else /* _UNICODE */
#define xtox_s xtoa_s
#define _itox_s _itoa_s
#define _ltox_s _ltoa_s
#define _ultox_s _ultoa_s
#define x64tox_s x64toa_s
#define _i64tox_s _i64toa_s
#define _ui64tox_s _ui64toa_s
#define xtox xtoa
#define _itox _itoa
#define _ltox _ltoa
#define _ultox _ultoa
#define x64tox x64toa
#define _i64tox _i64toa
#define _ui64tox _ui64toa
#endif /* _UNICODE */
..
.
.
.
#else /* _SECURE_ITOA */
length++;
} while (val > 0 && length < sizeInTChars);
/* Check for buffer overrun */
if (length >= sizeInTChars)
{
buf[0] = '\0';
_VALIDATE_RETURN_ERRCODE(length < sizeInTChars, ERANGE);
}
#endif /* _SECURE_ITOA */
/* We now have the digit of the number in the buffer, but in reverse
order. Thus we reverse them now. */
*p-- = _T('\0'); /* terminate string; p points to last digit */
In the calls stack window appears this
Code:
=> msvcr110d.dll!xtoa_s(unsigned long val, char * buf, unsigned int sizeInTChars, unsigned int radix, int is_neg) Line 142 C
msvcr110d.dll!_itoa_s(int val, char * buf, unsigned int sizeInTChars, int radix) Line 176 C
Get_Blocks.exe!main(int argc, char * * argv) Line 224 C++
Get_Blocks.exe!__tmainCRTStartup() Line 536 C
Get_Blocks.exe!mainCRTStartup() Line 377 C
kernel32.dll!7695336a() Unknown
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
ntdll.dll!76f19f72() Unknown
ntdll.dll!76f19f45() Unknown
It seems could be becuase to _itoa_s(), I'm using like this:
Code:
_itoa_s(CONVDEC(i), num, 10, 10);
sub += num;
1 Attachment(s)
Re: Read binary file with line delimeter
I use MSVS. It ran OK for me. But as other gurus on this forum keep saying, just because it works for me on my system doesn't mean the program is correct. It just means it works for me on my system. I haven't gone through it with the debugger as I haven't the time. There may well still be code problems which I'm not seeing on my system but which are showing on yours. This is why with c/c++ programs testing and debugging are all important. There may well be a case occuring in the data which I'm not handling properly but which 'appears' to work for me but isn't properly but which fails on your system.
UPDATE. I've now tried it on a Windows 7 64bit system using VS2012 - a different computer to which I've been using up to now (32 bit XP). I've recompiled the source under this system and again it runs with the data files without any error and produces from the bin1kb file the output
Code:
425983|532064023803664|81494511592||2,13,8149526905,0|||12,13,8149526905,1,1||12,13,81495269559,0||1,2,1,2,1,0,1,255,255,255,2,1,2,1
425984|532064017692652|81494511593||2,13,8149526905,0|||12,13,8149526905,1,1||12,13,81495269596,0||1,2,1,2,1,0,1,255,255,255,2,1,2,1
425985|532064022126005|81494511594|||||||||
425986|532064025030933|81494511596|||||||||
425987|532064012223291|81494511597|||||||||
425988|532064022602120|81494511599|||||||||
Time taken: 0
If it is still producing errors for you, you need to find the issue with the debugger as I can't help further as its compiled and run cleanly for me on two different versions of windows using two different MS compilers.
The attached is my MSVS 2012 solution that compiles OK and runs without problem. I just get a warning for _itoa. The source is test3.cpp and the exe is philidor.exe.
Good luck!
Re: Read binary file with line delimeter
The problem is showing itself with _itoa_s.
In main change num[10] to num[20] and change _itoa(xxx, num, 10, 10) to _itoa_s(xxx, num, 20, 10)
The problem is with a malformed sub-block which turns out not to be a sub-block at all.
Thats why with c/c++ you need to debug and test carefully and thoroughly.
Re: Read binary file with line delimeter
Quote:
Originally Posted by
2kaud
The problem is showing itself with _itoa_s.
In main change num[10] to num[20] and change _itoa(xxx, num, 10, 10) to _itoa_s(xxx, num, 20, 10)
The problem is with a malformed sub-block which turns out not to be a sub-block at all.
Thats why with c/c++ you need to debug and test carefully and thoroughly.
Changing to what you suggested works this time, even when some messages appeared in debugger but the file was processed.
And testing with 2GB file with the exe program you attached works just fine and was processed in 7.61 min. So, it seems the compiler here has something different. I'll reinstall this and if fixes the issues.
Thanks so so so much 2kaud for all the help, time and patience during these days and the willingness help others more that has more credit since those over there are unkown persons for you:wave:.
Many thanks Paul for your interventions, suggestions and comments.
Even I have to learn a lot, I have learned many things from your solutions and comments:thumb:.
Sorry for any bothering and inconvience caused.
Many thanks,
I hope I could help you sometime in something.
Best regards
Re: Read binary file with line delimeter
Quote:
And testing with 2GB file with the exe program you attached works just fine and was processed in 7.61 min. So, it seems the compiler here has something different. I'll reinstall this and if fixes the issues.
The exe program in the attached zip file uses _itoa and not _itoa_s. However, that program still has the size of the num buffer as 10 and so has the problem - you're just not seeing it! - because of the malformed sub-block. The size of this buffer in the program needs to be increased from 10 to say 20 to deal with it.
I'm pleased to be of help.
Re: Read binary file with line delimeter
For the sake of completeness and for those who have been following this thread 'with interest??', this is the final program which appears to work following the comment from philidor in post #154.
Code:
#include <iostream>
#include <fstream>
#include <string>
#include <ctime>
#include <cstdlib>
using namespace std;
typedef unsigned char BYTE;
typedef unsigned short int WORD;
typedef unsigned long int DWORD;
#ifndef LOBYTE
#define LOBYTE(w) ((BYTE)((WORD)(w) & 0xff))
#endif
#ifndef HIBYTE
#define HIBYTE(w) ((BYTE)((WORD)(w) >> 8))
#endif
#define CONVDEC(num) (convh[cx[c + (num)] - '0'] * 16 + convh[cx[c + (num) + 1] - '0'])
const char hconv[16] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
const int convh[23] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 10, 11, 12, 13, 14, 15};
const int aindx[23] = {0, 1, 3, 4, 8, 9, 6, 7, 9, 9, 9, 9, 9, 9, 9, 9, 9, 2, 5, 9, 9, 9, 9};
//#define FTYPE1
#define FTYPE2
#ifdef FTYPE1
const WORD SEPAR = 0xFF77;
const char SBLOCK[] = "FF79";
const char SFIND[] = "059";
const char SCHAR = '9';
#endif
#ifdef FTYPE2
const WORD SEPAR = 0xFF32;
const char SBLOCK[] = "FF34";
const char SFIND[] = "038";
const char SCHAR = '8';
#endif
class FileFields
{
private:
ifstream ifs;
bool opened;
public:
FileFields() : opened(false) {}
~FileFields() {
if (opened)
ifs.close();
}
bool open(const char* name);
bool getBlock(string& field, DWORD& number, string& firstpart, WORD delim = SEPAR);
bool getField(string& field, WORD delim = SEPAR);
};
bool FileFields::open(const char* name) {
ifs.open(name, ios::binary);
return (opened = ifs.is_open());
}
bool FileFields::getBlock(string& field, DWORD& number, string& firstpart, WORD delim)
{
BYTE num[3],
first[16],
by,
ub,
lb;
number = 0;
firstpart = "|";
if (!opened || !ifs.good())
return false;
ifs.read((char*)num, 3);
number = (num[0] << 16) + (num[1] << 8) + num[2];
if (!ifs.good())
return false;
ifs.read((char*)first, 16);
for (int p = 1; p <= 2; p++) {
const int last = p * 8;
for (int i = (p - 1) * 8; i < last; i++)
if ((ub = ((by = first[i]) >> 4)) < 0xf) {
firstpart += hconv[ub];
if ((lb = (by & 0x0f)) < 0xf)
firstpart += hconv[lb];
else
break;
} else
break;
if (p == 1)
firstpart += '|';
}
return getField(field, delim);
}
bool FileFields::getField(string& field, WORD delim)
{
char by;
bool cont = true;
field = "";
if (!opened || !ifs.good())
return false;
for (ifs.get(by); cont && ifs.gcount(); ifs.get(by)) {
if ((BYTE)by == HIBYTE(delim))
if ((BYTE)ifs.peek() == LOBYTE(delim))
cont = false;
if (cont) {
field += hconv[(BYTE)by >> 4];
field += hconv[(BYTE)by & 0xf];
}
}
return true;
}
int main()
{
FileFields ff;
//if (!ff.open("d:\\philidor\\bin2g")) {
//if (!ff.open("d:\\philidor\\binsmall")) {
//if (!ff.open("d:\\philidor\\bin1mb.txt")) {
if (!ff.open("d:\\philidor\\bin1kb.txt")) {
cout << "Cannot open file!" << endl;
return 1;
}
string header;
ff.getField(header);
string block;
block.reserve(7000);
string preliminar;
preliminar.reserve(7000);
string cx;
cx.reserve(7000);
string sub;
sub.reserve(7000);
DWORD number;
const int NUMSIZE = 20;
char num[NUMSIZE];
time_t timest = time(NULL);
const int smlblk = 68;
for (DWORD blk = 1; ff.getBlock(block, number, preliminar); blk++) {
size_t ff79;
bool got4 = false;
string sblk[10] = {"", "", "", "", "", "" ,"", "", "", ""};
if ((ff79 = block.find(SBLOCK)) != string::npos && (ff79 + smlblk + 4 < block.size())) {
size_t five;
while ((five = block.find(SFIND, ff79)) != string::npos) {
if (five + smlblk >= block.size()) {
five = string::npos;
break;
}
if (block[five + 6] == '0' && convh[block[five + 7] - '0'] <= 10)
break;
else
ff79 += 3;
}
if (five != string::npos) {
cx = block.substr(five + 2);
for (size_t c = 0; c < cx.size() && !got4; c+= 2) {
char styp;
sub = "";
if (cx[c] == SCHAR && (((styp = cx[c + 1]) >= '0' && styp <= '7') || styp == 'A' || styp == 'B')) {
const int slen = CONVDEC(2) * 2;
if (got4 = (styp == '4')) {
for (int i = 4; i < slen + 4; i += 2) {
_itoa(CONVDEC(i), num, 10);
//_itoa_s(CONVDEC(i), num, NUMSIZE, 10);
sub += num;
if (i != slen + 2)
sub += ',';
}
sblk[aindx[styp - '0']] = sub;
} else
if (cx[c + 1] != '5') {
_itoa(CONVDEC(6), num, 10);
//_itoa_s(CONVDEC(6), num, NUMSIZE, 10);
sub += num;
sub += ',';
int dec = 0;
for (int s = 8; s < 16; s += 2)
dec = (dec << 8) + CONVDEC(s);
_itoa(dec, num, 10);
//_itoa_s(dec, num, NUMSIZE, 10);
sub += num;
sub += ',';
for (size_t s = c + 16; s < c + 32; s++)
if (cx[s] != 'F')
sub += cx[s];
else
break;
sub += ',';
_itoa(CONVDEC(32), num, 10);
//_itoa_s(CONVDEC(32), num, NUMSIZE, 10);
sub += num;
if (slen == 32) {
sub += ',';
_itoa(CONVDEC(34), num, 10);
//_itoa_s(CONVDEC(34), num, NUMSIZE, 10);
sub += num;
}
sblk[aindx[styp - '0']] = sub;
}
c += slen + 2;
}
}
}
}
if (got4)
for (int a = 0; a <= 8; a++)
preliminar += "|" + sblk[a];
else
preliminar += "|||||||||";
cout << number << preliminar << endl;
}
cout << "Time taken: " << time(NULL) - timest << endl;
return 0;
}