concept for saving data

[crazy boy]

I would like to create program which will analyse bitmap so would need good concept to save data. I am interested about the theory and I realize that i must to think this carefully because bad concept could create insufficient memory or inefficient program. Basically I want my program work with HSV or HSL model so I would need to convert the bitmap to HSL, but I am not sure if I should convert it first and then analyse all pixels or should I start to analyse the bitmap and make the conversion to HSL during it. But my main question is what method to choose to save the data in memory. Even that I would start with very small, it should work also with bigger image like image having 1200 or even 4200 px on height. So the program should first analyse all columns of pixels in the image so for example 1200x800 px image has 1200 columns. So I would like to know if is it possible to create such object which would have such structure like this
Obj->basicColumnData->black->columns[name]->group
and in the place of columns should be placed data for every column. I would look for groups of pixels in the column, so in the result the column x could bear e.g. 500 groups of information and every group should contain the range of pixels e.g. group 1 should contain y value from 0 to 20, group 2 should contain value from 25-27 and so on. So I would create 1200 columns bear many of groups. This would be contained in "black" or "white" member to contain the data. This is just simplified idea, but the whole object should contain next data not just basicColumnData... So there should be another members bearing information calculated from the selected data.

So my question is what kind of method of saving data use for this? Should I use heap and dynamic allocated memory or should I create custom class, which will define every member, but these members will have to be dynamic memory? With the dynamic memory is there problem that there could be not enough memory to create such big object?

[Arjay]

You may want to look at the STL and work through some collection, allocator and stream examples.

[crazy boy]

Arjay, do you think that using a Container will be fast enough? I mean not to use something what uses slow "look up" technique or compression technique. I looked on containers last year especially vectors, looks I could have a lot of fun with it but not sure if the performance is the best choice for image analyse purposes. I have no idea how could I use streams. Can you give example how could I use streams for it?

[Arjay]

There are several containers so you need to pick the one with the lookup features you requires. You mentioned saving data, I misread that to mean saving to a file. Streams might still be useful if you don't need all the data before you start processing (but that probably isn't the case).

[crazy boy]

But how would you solve if you would analyse big image and you would run out of memory? Is there a way, how you could save content of your object in memory, eg. of type STL to temporal folder? Is there some special function, which can save a variable like string, array or object and save all addresses which are bound to this variable? I mean save the object to file after I processed/analysed part of the image? And then all parts are done and saved to temporal files, then to read the object again to memory and use it again? I am pretty curious about it because I believe graphical programs often uses such feature.

[OReubens]

typically speaking, if your image data is larger than actual available RAM memory,
your application may suffer (a lot) because of continuous paging to and from the pagine file.

if you image data is larger than available addressable memory in your process (2Gb at best on Win32, but in practice a lot less) then you have to deal with a system for managing that which will slow things down even more.

STL in and by itself won't solve any of this. That doesn't mean the STL containers are useless even in that scenario.
you're still responsible for choosing the right kind of container, and yes, maybe, the container you really need isn't part of STL and you may have to build your own.

THe thing to keep in mind is that the container classes are there to make your life as a programmer easier, they tend to be thouroughly tested (literally thousands of programs use them) and very well optimized (again, lots of peer review allows for lots of improvements, and compilers themselves often have optimizations specifically for dealing with common actions you do in the STL containers, iterators and algorithms.