Crash Reading & Writing More Files

Hi. I have a program that reads from a couple large files and extracts select smaller files. The code below was working until I went from 22 extractions to 56. And this number may increase in the future.

At first, I thought one (or more) of the new entries had an error. However, the program works when I removed either the original half or the new half of entries.

Since it crashes with all entries, this leads me to believe that the problem is the volume of extractions and that the code needs to be optimized in some way.

I tried added pauses between extractions, but it still crashes immediately upon running. So I'm not sure if that indicative of something else.

Here is a snippet of the code...

Code:

---

#include <iostream>
#include <fstream>
#include <windows.h>

using namespace std;

int main(){

//Still crashes if pause is before all this code.
//std::cin.get();

ifstream infile001 ("Source1.xxx", ifstream::binary);
ofstream outfile001 ("001.yyy", ofstream::binary);
infile001.seekg(0x2a00000);
char buffer001[15000];
infile001.read (buffer001,15000);
outfile001.write (buffer001,15000);
outfile001.close();
infile001.close();

ifstream infile002 ("Source1.xxx", ifstream::binary);
ofstream outfile002 ("002.yyy", ofstream::binary);
infile002.seekg(0x3f00000);
char buffer002[27000];
infile002.read (buffer002,27000);
outfile002.write (buffer002,27000);
outfile002.close();
infile002.close();

//And so on...

return 0;

}

---

I've also tried making a extract call out of this code, but haven't been successful. Unless the code to fix this is more complex than the above code, I'm okay with not making an extract call as long as everything works. Any help is appreciated. Thanks!

1. How does it crash?
2. What is the total size of data you have read in the moment of crash?

3. Would something change if you updated your code like

Code:

---

ifstream infile001 ("Source1.xxx", ifstream::binary);
ofstream outfile001 ("001.yyy", ofstream::binary);
infile001.seekg(0x2a00000);
{
      char buffer001[15000];
      infile001.read (buffer001,15000);
      outfile001.write (buffer001,15000);
}
outfile001.close();
infile001.close();

---

and so on...

You're running out of stack space. Either make a function to process the files or wrap each file section in braces {}

Thanks for the replies. Unfortunately, after trying both suggestions on using braces, running the compiled file still crashes.

Code:

---

ifstream infile001 ("Source1.xxx", ifstream::binary);
ofstream outfile001 ("001.yyy", ofstream::binary);
infile001.seekg(0x2a00000);
{
      char buffer001[15000];
      infile001.read (buffer001,15000);
      outfile001.write (buffer001,15000);
}
outfile001.close();
infile001.close();

---

Code:

---

{
ifstream infile001 ("Source1.xxx", ifstream::binary);
ofstream outfile001 ("001.yyy", ofstream::binary);
infile001.seekg(0x2a00000);
char buffer001[15000];
infile001.read (buffer001,15000);
outfile001.write (buffer001,15000);
outfile001.close();
infile001.close();
}

---

Like before, the compiled file crashes immediately. I don't think it executes any of the extraction since the pause I added previously didn't seem to get executed since it crashed before it paused. Any other thoughts on what could cause a crash beforehand?

1. Define "immediately".
2. You have not explained yet, how it "crashes".
3. Did you try to debug your code? You have to!

I'm not sure how to define "immediately" without repeating my earlier posts relating to the early pause feature not seeming to be executed.

I'm also not sure how to answer "how" it crashes. It seems to be a run-time error where I click on the compiled program and it crashes before reaching the pause feature added to help troubleshoot.

The debugger seems to confirm earlier suspicions that it's a stack overflow issue. Are there any other strategies to handle stack overflow issues since the braces didn't seem to solve it?

If you debugged your code then you have to be able to answer my other question:
2. What is the total size of data you have read in the moment of crash?

Quote:

---

Originally Posted by Mr. Smith

Are there any other strategies to handle stack overflow issues since the braces didn't seem to solve it?

---

yes, of course!
It is what GCDEF already wrote you:
"make a function to process the files"

Thanks for pointing me in the right direction. I did try making a call function before posting, as I mentioned. However, I wasn't able to get the syntax right.

Trying again, I found the specific stumbling block was having a variable length buffer array.

Code:

---

char buffer[length];

---

So I removed the variable and used the buffer integer needed by the largest file. Everything appears to be working. Thanks again for the pointers.

(If there is a more elegant solution, I'd be interested. The alternatives I was able to find didn't seem as simple as that.)

Quote:

---

Originally Posted by Mr. Smith

...
(If there is a more elegant solution, I'd be interested. The alternatives I was able to find didn't seem as simple as that.)

---

Yes. Using MFC CString or std::string instead of the plain char array.

I've tried using std::string, but the code won't compile. Maybe I'm using the incorrect syntax or wrong variable type for length (int). I've been searching for examples online for a bit, but haven't had much success. What should this line of code look like?

Quote:

---

Originally Posted by Mr. Smith

I've tried using std::string, but the code won't compile.

---

Show your code.
What errors and where?

Quote:

---

Originally Posted by Mr. Smith

Are there any other strategies to handle stack overflow issues...?

---

Quote:

---

Originally Posted by Mr. Smith

(If there is a more elegant solution, I'd be interested. The alternatives I was able to find didn't seem as simple as that.)

---

The best strategy to avoid stack overflow is to NOT allocate large arrays on the stack. Are you familiar with the new[] operator?
Also, your original code sample has comment

Code:

---

//And so on...

---

Does it mean that you have multiple buffers defined? Like that?

Code:

---

char buffer001[15000];
char buffer002[27000];
char buffer003[XXXXX];

---

Since you write the content of those buffers to the output file, couldn't you have reused just ONE buffer?

Quote:

---

Originally Posted by Mr. Smith

Here is a snippet of the code...

Code:

---

#include <iostream>
#include <fstream>
#include <windows.h>

using namespace std;

int main(){

//Still crashes if pause is before all this code.
//std::cin.get();

ifstream infile001 ("Source1.xxx", ifstream::binary);
ofstream outfile001 ("001.yyy", ofstream::binary);
infile001.seekg(0x2a00000);
char buffer001[15000];
infile001.read (buffer001,15000);
outfile001.write (buffer001,15000);
outfile001.close();
infile001.close();

ifstream infile002 ("Source1.xxx", ifstream::binary);
ofstream outfile002 ("002.yyy", ofstream::binary);
infile002.seekg(0x3f00000);
char buffer002[27000];
infile002.read (buffer002,27000);
outfile002.write (buffer002,27000);
outfile002.close();
infile002.close();

//And so on...

return 0;

}

---

---

Don't you see a repeating pattern in your code? It's time to think about introducing a function, something like that:

Code:

---

void pipe_data(ifstream& infile, ofstream& outfile, size_t readSize)
{
    char* buf = new char[readSize];
    infile.read(buf, readSize);
    outfile.write(buf, readSize);
    delete[] buf;
}

---

Besides, I'd think about re-using in-stream and out-stream objects instead of creating a bunch of one-shot dummies. Or maybe about a loop like this:

Code:

---

typedef struct task {
    char* src;
    char* dst;
    size_t size;
} Task;

Task tasks[] = {
    { "Source.001", "Destination.001" 15000 },
    ...
    { "Source.055", "Destination.055", 27000 },
    { NULL, NULL, 0}
};

for (Task* theTask = tasks; theTask->src != NULL; theTask++)
{   
    ifstream infile (theTask->src, ifstream::binary);
    ofstream outfile (theTask->dst, ofstream::binary);
    pipe_data(infile, outfile, theTask->size);
    infile.close();
    outfile.close();
}

---

I think it is a compiler problem.If your compiler is ok then you can try this code

// cin with strings
#include <iostream>
#include <string>
using namespace std;

int main ()
{
string mystr;
cout << "What's your name? ";
getline (cin, mystr);
cout << "Hello " << mystr << ".\n";
cout << "What is your favorite team? ";
getline (cin, mystr);
cout << "I like " << mystr << " too!\n";
return 0;
}

Quote:

---

Originally Posted by camycent.solutions

I think it is a compiler problem.If your compiler is ok then you can try this code

// cin with strings
#include <iostream>
#include <string>
using namespace std;

int main ()
{
string mystr;
cout << "What's your name? ";
getline (cin, mystr);
cout << "Hello " << mystr << ".\n";
cout << "What is your favorite team? ";
getline (cin, mystr);
cout << "I like " << mystr << " too!\n";
return 0;
}

---

and the relevancy of this post to the thread is....... :thumbd:

Quote:

---

Originally Posted by camycent.solutions

I think it is a compiler problem.If your compiler is ok then you can try this code

---

It's almost never a compiler problem. In this case, it's clearly a stack space problem.

Thanks for all the feedback. Sorry if I wasn't clearer in post #9 of this thread. I was able to get a function working that seemed to solve the stack overflow issue.

The issue with that function, however, was not being able to use a variable for the size of the buffer. I needed to use the integer of the largest buffer. Below are the bare-bones examples...

Code:

---

//Doesn't compile.

#include <iostream>
#include <fstream>
#include <string>
#include <windows.h>
using namespace std;

void extract(int length){
char buffer[length];
}

int main(){
extract (150000);
return 0;
}

---

Code:

---

//Compiles.

#include <iostream>
#include <fstream>
#include <string>
#include <windows.h>
using namespace std;

void extract(int length){
char buffer[150000];
}

int main(){
extract (150000);
return 0;
}

---

I also tried the other suggested buffer method below and it worked. It made the full program run about 10 seconds faster.

Code:

---

//Compiles.

#include <iostream>
#include <fstream>
#include <string>
#include <windows.h>
using namespace std;

void extract(size_t length){
char* buffer = new char[length];
delete[] buffer;
}

int main(){
extract (150000);
return 0;
}

---

Before my original post, I tried this method. However, I must've had a syntax error elsewhere that caused the program not to compile. This method, from what I read elsewhere, is promoted for reentrancy. I'm not sure if that's an issue in this case.

I also tried using std:string to create a buffer, but couldn't get the syntax right.

The instances where the code wouldn't compile made me wonder if I was using incorrect syntax or, since I'm using Visual C++ 2008, newer syntax. But if char buffer[length] and std::strings (and any other buffers) should work with 2008, I'm still interested in trying other buffer methods that may improve performance or are better for "best practices."

Thanks again for any insights on this.

Why do you care about the size of the buffer. Just make sure it's big enough for the largest file, or read the file in chunks.

Quote:

---

Originally Posted by Mr. Smith

Code:

---

//Doesn't compile.

void extract(int length){
char buffer[length];
}

---

---

you cannot create a global or local scope buffer with a size that is variable like this.
there are non-portable ways to allocate a variable amount of memory on the stack (such as _alloca() and _malloca on VC), but this may not be available on your compiler.

The solution is as you already found out in the next attempt. to allocate the memory dynamically. from the free store using new/delete is the Obvious choice, but there are others. (usually non portable).


an even better way, and one that "looks" as though it magically does what you were trying is to use the C++ container class std::vector.

Code:

---

void extract(int length){
std::vector<char> buffer(length);

// use buffer.data() to get a pointer to the first element in that buffer.
}

---

the nice part is, it does the new/delete for you, and it automatically cleans up after itself even if you get thrown out of the function with an exception (this is known as RAII and is an important concept in proper C++ development).