How can I speed up ostringstream writing to an ofstream

[JonnoA]

I am using an ostringstream to hold text which is written to a file using an ofstream object, but it is unacceptably slow - it takes more than 10 secs to write 200kb on a fast pc. Please can you tell me how to speed this up significantly.

In the document's OnSave function(I'm using MFC) I create an ostringstream and pass it to a function to be filled with data, then I write the text string to the ofstream:

BOOL CMyDocument::OnFileSave(lpszPathName)
{
ofstream fout;
fout.open(lpszPathName);
ostringstream os;
pData->TextOut(os);
fout << os.str();
}

where pData is a pointer to a data object (CData*) held by the document. CData is a base class for CDataReal and CDataComplex, which hold real and complex data respectively. In the derived class I do the following to write each row of a 2d array on separate lines with a comma delimiter:

void CDataReal::TextOut(ostringstream& os)
{
   for (int i=0; i < someIndex; ++i) {
      for (int j=0; j < someIndex; ++j) {
         os << "," << m_TemplateData.begin()[j][i];
      }
      os << endl;
   }
}

I don't necessarily have to pass an ostringstream object - it just seemed like a neat solution. Could someone please explain why the above is SO slow.

Thank you

[Philip Nicoletti]

The inernal buffer does not increase in size by
a large amount when it needs to, causing a lot of
re-copying of data. Dinkumware has a fix for this if you
want (grows the internal buffer similar to vector).

see "fix to <sstream>"

http://www.dinkumware.com/vc_fixes.html

As you mentioned, I'm not sure why you need a stringstream
object here. Passing the ofstream object should work.

[JonnoA]

Thanks for the SSTREAM fix. Unfortunately this made a negligible difference.

I am passing an ostringstream object to the TextOut function so that I can use the function to write text to the clipboard, for example.

As an example, to write each element of a 512x256 array (130k elements) to a file takes OVER 5 MINUTES - not acceptable, but that's 2.5ms to write each element. I'm not sure whether that's fast or not. However, I read in the file in about 3 seconds (cf 320 secs to write it).

Any other ideas? Thanks

[JonnoA]

It's definitely the ostringstream object slowing things down. Passing an ofstream reference instead cuts the time down from 320 secs to 0.7sec.

I don't really want to pass the ofstream because I can't then use my TextOut function for the clipboard. Any help would be much appreciated.

[alex75014]

Not sure if it can be optimized a lot, but here are some ideas :

BOOL CMyDocument::OnFileSave(lpszPathName)
{
ofstream fout;
fout.open(lpszPathName);

/* customizing buffer */
char[512] buffer;
fout.rdbuf()->pubsetbuf(buffer, 512);

pData->TextOut(&fout);
fout.flush();
}

void CDataReal::TextOut(ostream* os)
{
   for (int i=0; i < someIndex; ++i) {
      for (int j=0; j < someIndex; ++j) {
         *os << "," << m_TemplateData.begin()[j][i];
      }
      *os << "\n"; // not endl
   }
}

Firstly I got rid of this stringstream. In my opinion it just add some unecessary computing.

Secondly, I would replace endl by "\n": It should avoid unecessary synchronisation.

Thirdly, you could tune the filebuf buffer for best performance. I put 512 buffering in above example.

I hope this help, if you get better performance this way please keep us informed

[alex75014]

Well I missed your message, but my above post still will help you : use an ostream. Then you will be able to call it with any ostream derived class.

[JonnoA]

doh - of course! Thanks
I'm still curious to know why writing to an ofstream via a ostringstream is over 100 times slower than writing to the ofstream directly.

[alex75014]

I know that microsoft CString are seriously bugged : += cause a reallocation and a copy of the string.
So may be your stringstream class is built on top of CString no?

[MrViggy]

No. ostringstream is STL, CString is MFC. There is no MFC in the Microsoft delivered STL.

Viggy

[Philip Nicoletti]

After changing the sstream file, did you do a rebuild all ?
I simple build will not recompile the code.

In the sample code below, after making the change to sstream,
the code runs 43 times faster on my computer.

#include <sstream>
#include <fstream>
#include <ctime>
#include <iostream>

using namespace std;

int main()
{
    ofstream out("out.txt");

    stringstream ss;

    clock_t start,finish;

    start = clock();

    for (int i=0; i<100000; ++i)
        ss << "," << i;

    ss << endl;

    finish = clock();

    cout << finish-start << endl;

    out << ss.str();

    return 0;
}

[JonnoA]

Yes, I did rebuild all, but the plot thickens:
I put your (P. Nicoletti's) test code into 2 new projects, a console app and an MFC application (in OnSaveDocument). Exactly the same code takes the following number of ticks to execute:

The original SSTREAM file (unpatched)
console release=26958
console debug=38094
mfc app release=27670
mfc app debug=37534

Patched SSTREAM file
console release=201
console debug=1001
mfc app release=27690
mfc app debug=36562

I put a #pragma message ("including sstream") to be sure it was being included.

So the console app indeed shows dramatic speed increase, but there is no change for the MFC application.

Does anybody know what is going on?

Thanks

[JonnoA]

The problem seems to be with single and multi threading.

If I set the compiler option (i'm using vc6 with the lastest sp5) for the console app to /MDd instead of MLd (ie use multithread rather than single thread dll) the the execution time goes right up again.

Suggestions?