Read binary file with line delimeter - Page 11
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 11 of 11 FirstFirst ... 891011
Results 151 to 156 of 156

Thread: Read binary file with line delimeter

  1. #151
    Join Date
    Oct 2013
    Posts
    63

    Re: Read binary file with line delimeter

    Hello Paul,

    I run with debugger and appear this file refering the error to line 142 (in red):
    Code:
    /***
    *xtoa.c - convert integers/longs to ASCII string
    *
    *       Copyright (c) Microsoft Corporation. All rights reserved.
    *
    *Purpose:
    *       The module has code to convert integers/longs to ASCII strings.  See
    *
    *******************************************************************************/
    
    #include <cruntime.h>
    #include <stdlib.h>
    #include <limits.h>
    #include <tchar.h>
    #include <internal.h>
    #include <internal_securecrt.h>
    
    #ifdef _UNICODE
    #define xtox_s     xtow_s
    #define _itox_s    _itow_s
    #define _ltox_s    _ltow_s
    #define _ultox_s   _ultow_s
    #define x64tox_s   x64tow_s
    #define _i64tox_s  _i64tow_s
    #define _ui64tox_s _ui64tow_s
    #define xtox       xtow
    #define _itox      _itow
    #define _ltox      _ltow
    #define _ultox     _ultow
    #define x64tox     x64tow
    #define _i64tox    _i64tow
    #define _ui64tox   _ui64tow
    #else  /* _UNICODE */
    #define xtox_s     xtoa_s
    #define _itox_s    _itoa_s
    #define _ltox_s    _ltoa_s
    #define _ultox_s   _ultoa_s
    #define x64tox_s   x64toa_s
    #define _i64tox_s  _i64toa_s
    #define _ui64tox_s _ui64toa_s
    #define xtox       xtoa
    #define _itox      _itoa
    #define _ltox      _ltoa
    #define _ultox     _ultoa
    #define x64tox     x64toa
    #define _i64tox    _i64toa
    #define _ui64tox   _ui64toa
    #endif  /* _UNICODE */
    ..
    .
    .
    .
    
    #else  /* _SECURE_ITOA */
                length++;
            } while (val > 0 && length < sizeInTChars);
    
            /* Check for buffer overrun */
            if (length >= sizeInTChars)
            {
                buf[0] = '\0';
                _VALIDATE_RETURN_ERRCODE(length < sizeInTChars, ERANGE);
            }
    #endif  /* _SECURE_ITOA */
            /* We now have the digit of the number in the buffer, but in reverse
               order.  Thus we reverse them now. */
    
            *p-- = _T('\0');            /* terminate string; p points to last digit */
    In the calls stack window appears this
    Code:
    =>	msvcr110d.dll!xtoa_s(unsigned long val, char * buf, unsigned int sizeInTChars, unsigned int radix, int is_neg) Line 142	C
     	msvcr110d.dll!_itoa_s(int val, char * buf, unsigned int sizeInTChars, int radix) Line 176	C
     	Get_Blocks.exe!main(int argc, char * * argv) Line 224	C++
     	Get_Blocks.exe!__tmainCRTStartup() Line 536	C
     	Get_Blocks.exe!mainCRTStartup() Line 377	C
     	kernel32.dll!7695336a()	Unknown
    	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
     	ntdll.dll!76f19f72()	Unknown
     	ntdll.dll!76f19f45()	Unknown
    It seems could be becuase to _itoa_s(), I'm using like this:
    Code:
     _itoa_s(CONVDEC(i), num, 10, 10);
    	    sub += num;
    Last edited by Philidor; November 2nd, 2013 at 04:15 PM.

  2. #152
    Join Date
    Dec 2012
    Location
    England
    Posts
    1,993

    Re: Read binary file with line delimeter

    I use MSVS. It ran OK for me. But as other gurus on this forum keep saying, just because it works for me on my system doesn't mean the program is correct. It just means it works for me on my system. I haven't gone through it with the debugger as I haven't the time. There may well still be code problems which I'm not seeing on my system but which are showing on yours. This is why with c/c++ programs testing and debugging are all important. There may well be a case occuring in the data which I'm not handling properly but which 'appears' to work for me but isn't properly but which fails on your system.

    UPDATE. I've now tried it on a Windows 7 64bit system using VS2012 - a different computer to which I've been using up to now (32 bit XP). I've recompiled the source under this system and again it runs with the data files without any error and produces from the bin1kb file the output

    Code:
    425983|532064023803664|81494511592||2,13,8149526905,0|||12,13,8149526905,1,1||12,13,81495269559,0||1,2,1,2,1,0,1,255,255,255,2,1,2,1
    425984|532064017692652|81494511593||2,13,8149526905,0|||12,13,8149526905,1,1||12,13,81495269596,0||1,2,1,2,1,0,1,255,255,255,2,1,2,1
    425985|532064022126005|81494511594|||||||||
    425986|532064025030933|81494511596|||||||||
    425987|532064012223291|81494511597|||||||||
    425988|532064022602120|81494511599|||||||||
    Time taken: 0
    If it is still producing errors for you, you need to find the issue with the debugger as I can't help further as its compiled and run cleanly for me on two different versions of windows using two different MS compilers.

    The attached is my MSVS 2012 solution that compiles OK and runs without problem. I just get a warning for _itoa. The source is test3.cpp and the exe is philidor.exe.

    Good luck!
    Attached Files Attached Files
    All advice is offered in good faith only. You are ultimately responsible for effects of your programs and the integrity of the machines they run on.

  3. #153
    Join Date
    Dec 2012
    Location
    England
    Posts
    1,993

    Re: Read binary file with line delimeter

    The problem is showing itself with _itoa_s.

    In main change num[10] to num[20] and change _itoa(xxx, num, 10, 10) to _itoa_s(xxx, num, 20, 10)

    The problem is with a malformed sub-block which turns out not to be a sub-block at all.

    Thats why with c/c++ you need to debug and test carefully and thoroughly.
    Last edited by 2kaud; November 2nd, 2013 at 05:14 PM.
    All advice is offered in good faith only. You are ultimately responsible for effects of your programs and the integrity of the machines they run on.

  4. #154
    Join Date
    Oct 2013
    Posts
    63

    Re: Read binary file with line delimeter

    Quote Originally Posted by 2kaud View Post
    The problem is showing itself with _itoa_s.

    In main change num[10] to num[20] and change _itoa(xxx, num, 10, 10) to _itoa_s(xxx, num, 20, 10)

    The problem is with a malformed sub-block which turns out not to be a sub-block at all.

    Thats why with c/c++ you need to debug and test carefully and thoroughly.
    Changing to what you suggested works this time, even when some messages appeared in debugger but the file was processed.
    And testing with 2GB file with the exe program you attached works just fine and was processed in 7.61 min. So, it seems the compiler here has something different. I'll reinstall this and if fixes the issues.

    Thanks so so so much 2kaud for all the help, time and patience during these days and the willingness help others more that has more credit since those over there are unkown persons for you.

    Many thanks Paul for your interventions, suggestions and comments.

    Even I have to learn a lot, I have learned many things from your solutions and comments.

    Sorry for any bothering and inconvience caused.

    Many thanks,

    I hope I could help you sometime in something.


    Best regards

  5. #155
    Join Date
    Dec 2012
    Location
    England
    Posts
    1,993

    Re: Read binary file with line delimeter

    And testing with 2GB file with the exe program you attached works just fine and was processed in 7.61 min. So, it seems the compiler here has something different. I'll reinstall this and if fixes the issues.
    The exe program in the attached zip file uses _itoa and not _itoa_s. However, that program still has the size of the num buffer as 10 and so has the problem - you're just not seeing it! - because of the malformed sub-block. The size of this buffer in the program needs to be increased from 10 to say 20 to deal with it.

    I'm pleased to be of help.
    All advice is offered in good faith only. You are ultimately responsible for effects of your programs and the integrity of the machines they run on.

  6. #156
    Join Date
    Dec 2012
    Location
    England
    Posts
    1,993

    Re: Read binary file with line delimeter

    For the sake of completeness and for those who have been following this thread 'with interest??', this is the final program which appears to work following the comment from philidor in post #154.

    Code:
    #include <iostream>
    #include <fstream>
    #include <string>
    #include <ctime>
    #include <cstdlib>
    using namespace std;
    
    typedef unsigned char BYTE;
    typedef unsigned short int WORD;
    typedef unsigned long int DWORD;
    
    #ifndef LOBYTE
    	#define LOBYTE(w)	((BYTE)((WORD)(w) & 0xff))
    #endif
    
    #ifndef HIBYTE
    	#define HIBYTE(w)	((BYTE)((WORD)(w) >> 8))
    #endif
    
    #define CONVDEC(num)	(convh[cx[c + (num)] - '0'] * 16 + convh[cx[c + (num) + 1] - '0'])
    
    const char hconv[16] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
    const int convh[23] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 10, 11, 12, 13, 14, 15};
    const int aindx[23] = {0, 1, 3, 4, 8, 9, 6, 7, 9, 9, 9, 9, 9, 9, 9, 9, 9, 2, 5, 9, 9, 9, 9};
    
    //#define FTYPE1
    #define FTYPE2
    
    #ifdef FTYPE1
    	const WORD SEPAR = 0xFF77;
    	const char SBLOCK[] = "FF79";
    	const char SFIND[] = "059";
    	const char SCHAR = '9';
    #endif
    
    #ifdef FTYPE2
    	const WORD SEPAR = 0xFF32;
    	const char SBLOCK[] = "FF34";
    	const char SFIND[] = "038";
    	const char SCHAR = '8';
    #endif
    
    class FileFields
    {
    private:
    	ifstream	ifs;
    	bool		opened;
    
    public:
    	FileFields() : opened(false) {}
    
    	~FileFields() {
    		if (opened)
    			ifs.close();
    	}
    
    	bool open(const char* name);
    
    	bool getBlock(string& field, DWORD& number, string& firstpart, WORD delim = SEPAR);
    	bool getField(string& field, WORD delim = SEPAR);
    
    };
    
    bool FileFields::open(const char* name) {
    	ifs.open(name, ios::binary);
    	return (opened = ifs.is_open());
    }
    
    bool FileFields::getBlock(string& field, DWORD& number, string& firstpart, WORD delim)
    {
    BYTE	num[3],
    	first[16],
    	by,
    	ub,
    	lb;
    
    	number = 0;
    	firstpart = "|";
    
    	if (!opened || !ifs.good())
    		return false;
    
    	ifs.read((char*)num, 3);
    	number = (num[0] << 16) + (num[1] << 8) + num[2];
    
    	if (!ifs.good())
    		return false;
    
    	ifs.read((char*)first, 16);
    
    	for (int p = 1; p <= 2; p++) {
    		const int last = p * 8;
    		for (int i = (p - 1) * 8; i < last; i++)
    			if ((ub = ((by = first[i]) >> 4)) < 0xf) {
    				firstpart += hconv[ub];
    				if ((lb = (by & 0x0f)) < 0xf)
    					firstpart += hconv[lb];
    				else
    					break;
    			} else
    				break;
    
    		if (p == 1)
    			firstpart += '|';
    	}
    
    	return getField(field, delim);
    }
    
    bool FileFields::getField(string& field, WORD delim)
    {
    char	by;
    
    bool	cont = true;
    
    	field = "";
    
    	if (!opened || !ifs.good())
    		return false;
    
    	for (ifs.get(by); cont && ifs.gcount(); ifs.get(by)) {
    		if ((BYTE)by == HIBYTE(delim))
    			if ((BYTE)ifs.peek() == LOBYTE(delim))
    				cont = false;
    
    		if (cont) {
    			field += hconv[(BYTE)by >> 4];
    			field += hconv[(BYTE)by & 0xf];
    		}
    	}
    
    	return true;
    }
    
    int main()
    {
    FileFields	ff;
    
    	//if (!ff.open("d:\\philidor\\bin2g")) {
    	//if (!ff.open("d:\\philidor\\binsmall")) {
    	//if (!ff.open("d:\\philidor\\bin1mb.txt")) {
    	if (!ff.open("d:\\philidor\\bin1kb.txt")) {
    		cout << "Cannot open file!" << endl;
    		return 1;
    	}
    
    string	header;
    	ff.getField(header);
    
    string	block;
    	block.reserve(7000);
    
    string preliminar;
    	preliminar.reserve(7000);
    
    string cx;
    	cx.reserve(7000);
    
    string sub;
    	sub.reserve(7000);
    
    DWORD	number;
    
    const int NUMSIZE = 20;
    char num[NUMSIZE];
    
    time_t timest = time(NULL);
    
    const int smlblk = 68;
    
    	for (DWORD blk = 1; ff.getBlock(block, number, preliminar); blk++) {
    		size_t ff79;
    		bool got4 = false;
    		string sblk[10] = {"", "", "", "", "", "" ,"", "", "", ""};
    
    		if ((ff79 = block.find(SBLOCK)) != string::npos && (ff79 + smlblk + 4 < block.size())) {
    			size_t five;
    
    			while ((five = block.find(SFIND, ff79)) != string::npos) {
    				if (five + smlblk >= block.size()) {
    					five = string::npos;
    					break;
    				}
    
    				if (block[five + 6] == '0' && convh[block[five + 7] - '0'] <= 10)
    					break;
    				else
    					ff79 += 3;
    			}
    
    			if (five != string::npos) {
    				cx = block.substr(five + 2);
    				for (size_t c = 0; c < cx.size() && !got4; c+= 2) {
    					char styp;
    					sub = "";
    					if (cx[c] == SCHAR && (((styp = cx[c + 1]) >= '0' && styp <= '7') || styp == 'A' || styp == 'B')) {
    						const int slen = CONVDEC(2) * 2;
    						if (got4 = (styp == '4')) {
    							for (int i = 4; i < slen + 4; i += 2) {
    								_itoa(CONVDEC(i), num, 10);
    								//_itoa_s(CONVDEC(i), num, NUMSIZE, 10);
    								sub += num;
    								if (i != slen + 2) 
    									sub += ',';
    							}
    							sblk[aindx[styp - '0']] = sub;
    						} else 
    							if (cx[c + 1] != '5') {
    								_itoa(CONVDEC(6), num, 10);
    								//_itoa_s(CONVDEC(6), num, NUMSIZE, 10);
    								sub += num;
    								sub += ',';
    								int dec = 0;
    								for (int s = 8; s < 16; s += 2)
    									dec = (dec << 8) + CONVDEC(s);
    
    								_itoa(dec, num, 10);
    								//_itoa_s(dec, num, NUMSIZE, 10);
    								sub += num;
    								sub += ',';
    								for (size_t s = c + 16; s < c + 32; s++)
    									if (cx[s] != 'F')
    										sub += cx[s];
    									else
    										break;
    
    								sub += ',';
    								_itoa(CONVDEC(32), num, 10);
    								//_itoa_s(CONVDEC(32), num, NUMSIZE, 10);
    								sub += num;
    								if (slen == 32) {
    									sub += ',';
    									_itoa(CONVDEC(34), num, 10);
    									//_itoa_s(CONVDEC(34), num, NUMSIZE, 10);
    									sub += num;
    								}
    								sblk[aindx[styp - '0']] = sub;
    							}
    
    						c += slen + 2;
    					}
    				}
    			}
    		}
    		if (got4)
    			for (int a = 0; a <= 8; a++)
    				preliminar += "|" + sblk[a];
    		else 
    			preliminar += "|||||||||";
    
    		cout << number << preliminar << endl;
    	}
    
    	cout << "Time taken: " << time(NULL) - timest << endl;
    	return 0;
    }
    All advice is offered in good faith only. You are ultimately responsible for effects of your programs and the integrity of the machines they run on.

Page 11 of 11 FirstFirst ... 891011

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Azure Activities Information Page

Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center