Printable View
Hi,
How do I save an array:
static array<String^,3>^ picsByFileName = gcnew array<String^,3> (256,256,256);
on the hard drive. I know how to open and save files such as Jpeg files but I can't figure out how to save this three dimensional array of handles to string. The array contains names of picture files organized according to their average color characteristics.
:ehh:
Zinn
This is managed c++ code so the question should be posted here.
http://forums.codeguru.com/forumdisp...ed-C-and-C-CLI
If the file is not meant to be exchanged with other programs, IOW just gets written out by your program and read back in by the very same program, the most straightforward approach probably is simply using a binary writer and reader.
This is how writing out the file would work then:
And this is reading back the data:Code:FileStream ^strmOut = File::Create("Data.dat");
BinaryWriter ^bwr = gcnew BinaryWriter(strmOut);
for (int i = 0; i < 256; ++i)
for (int j = 0; j < 256; ++j)
for (int k = 0; k < 256; ++k)
bwr->Write(picsByFileName[i, j, k]);
bwr->Close(); // Also implicitly closes the underlying stream
Both code snippets asssume that the array has been properly declared and created, and the writing snippet also assumes that it has been populated with valid entries, of course. All the classes related to file I/O I'm using here are from the System::IO namespace.Code:FileStream ^strmIn = File::OpenRead("Data.dat");
BinaryReader ^brd = gcnew BinaryReader(strmIn);
for (int i = 0; i < 256; ++i)
for (int j = 0; j < 256; ++j)
for (int k = 0; k < 256; ++k)
picsByFileName[i, j, k] = brd->ReadString();
brd->Close(); // Also implicitly closes the underlying stream
Be aware, however, that the array holds more than 16 million strings. For testing I used short strings that were made up of the numerical product of the three indices converted to a string, and already that amounted to a data file of roughly 120 MB. Commonly used file names (in particular if they're fully qualified) can be expected to be much longer. (And I don't even dare to imagine how much storage space your 16 million image files would occupy.)
Hi Eri523,
Many thanks for your response. Your soulution looks promising and I will test it tomorrow (it is too late right now). The array is only used for short strings such as: 1.bmp, 2.bmp etc. and of course it will not be used for the image files themselves as they would require about ninety Gigabytes !! The images have been reduced to mini-pictures 40 x 40 pixels square, so each one is about 6.9KB. I don't need to exchange data with other programs. Once again, many thanks.
Zinn
Some more thoughts...
Assuming your files are named 0.bmp to 16777215.bmp (file number calculated as (i << 16) + (j << 8) + k), the average file name length is 11.3 characters. This amounts to an average of 12.3 bytes per file name stored in the disk file, since the binary writer stores the strings as one length byte plus the characters encoded as UTF-8. In memory, it would even occupy an average of 22.6 bytes (not accounting for any additional management information stored by/for the String object), due to the fact that System::String internally handles characters as UTF-16LE wide chars. The average length coud be recuced by four (35%) by simply omitting the .bmp extension from the stored file names. It would always be the same anyway, and you can add it on the fly when accessing the files. This would save you a total of 64 MB in the disk file, twice as much in memory.
However, when calculating the file numbers used as names that way, you can easily determine the corresponding array slot indices from the file name itself, like that:
Making use of that, you can enumerate the files using a directory scan and store the names into the appropriate, calculated slot, eliminating the need for a separate disk file to store the file names. Or, perhaps, as there is a direct mapping between slot indices and file names anyway, you may not even need to store the file names in memory, just a boolean flag for each slot that indicates whether the corresponding file actually exists. ... or not even that, if it can be assumed that there always is an existing file corresponding to each slot.Code:int nFileNumber = int::Parse(Path::GetFileNameWithoutExtension(strFileName));
int i = nFileNumber >> 16;
int j = (nFileNumber >> 8) & 0xFF;
int k = nFileNumber & 0xFF;
Another consideration: I wouldn't bet on that NTFS (or any other file system for that matter) would really be happy about having 16 million files in a single directory. One way to avoid this would be to store the image files in a hierarchical directory structure rather than the flat way. In such a scenario, you can construct file names like this (whith extension):
Of course you'd need to take care that the required sub directories get created appropriately.Code:String ^strFileName = String::Format("{0}\\{1}\\{2}.bmp", i, j, k); // As an alternative you may use Path::Combine()
Enumerating the files wouldn't be complicated too much by this approach: Directory::GetFiles() and Directory::EnumerateFiles() can be simply instructed to do a recursive scan by passing a simple option value.
You'll need to turn the absolute path returned by the directory scan into names relative to your image storage directory (however, you'd need to do that anyway, even with a flat hierarchy). Provided you did that, you then can reconstruct the array indices like that:
Finally, in a quick test (saved by Paint .NET) I got a size of 4854 byte for a 40x40 .bmp file with 24 bits per pixel, 54 bytes (1.1%) of which comprise the bitmap header structures. That doesn't sound much, but for 16 M files it would amount to a total of 864 MB. So you may consider writing the raw bitmap data to your files and re-attaching the header, that would remain the same for all files anyway, after reading each file to construct your bitmap object in memory. That would save you almost 1 GB, but it would complicate writing and reading of the image files, and they wouldn't be compatible with common image handling software anymore. Also, on my 1 TB partition, the cluster size is 4 KB, and after reducing the file size from 4854 to 4800 byte, the file would still occupy 8 KB of disk space anyway, so not much would be gained in this respect.Code:array<String ^> ^astrNameComponents = strRelativeFileName->Split(L'\\');
int i = int::Parse(astrNameComponents[0]);
int j = int::Parse(astrNameComponents[1]);
int k = int::Parse(Path::GetFileNameWithoutExtension(astrNameComponents[2]));
Hi Eri523,
Thanks for these extra comments, I appreciate the time you have taken to make them. I think I understand the gist of what you have said and I may review my approach to the problem. Actually the number of 40 x 40 pixel photos will probably be a lot less than 16M since I don't envisage having access to that many photos. I was using the three dimensional array approach because that seemed the most obvious way to get the name of a photo from three color integers, but your approach of manipulating the file name is worth investigating. Your code for storing and retrieving the array works well, thanks once again.
You're welcome. :)Quote:
Originally Posted by Zinn
In this case, depending on your actual coverage of the array and its distribution, it may be advatageous to treat your array as a sparse matrix. (... extending what's described in the article to three dimensions in some way. Also, performing matrix math on the data is certainly irrelevant in your usage scenario, so most of the advanced math stuff in the article may simply be ignored.)Quote:
[...] Actually the number of 40 x 40 pixel photos will probably be a lot less than 16M since I don't envisage having access to that many photos. I was using the three dimensional array approach because that seemed the most obvious way to get the name of a photo from three color integers [...].
So you're doing a lookup of small images by a single color value... Let me guess: photo mosaic? :) In such an application you'll probably want to be able to find the "nearest matching" entry from an uncovered array slot. IMO a 3D-array is rather sub-optimal for that kind of lookup, at least as long as it's not backed by an auxiliary data structure. It may be worth looking for a data structure that's better suited for this kind of lookup, before you write complicated and inefficient code to do the lokup in the array, and later replacing that by something that's not only more efficient, but also would have been simpler in the first place.
In the case here, i.e. when the coordinates represent RGB colors, it may be practical to create filenames that allow human users to identify the color represented. (Relevant only if you store your images in a flat hierarchy; would already be rather obvious in the case of a three-level hierarchy.) That can be achieved with just small changes to the file name creation scheme I suggested. Already expressing the file number in hexadecimal rather than decimal would be a significant step into that direction.Quote:
[...] but your approach of manipulating the file name is worth investigating.
Hi Eri523,
Yes, you're correct in your description of my project: I've almost completed a program that tiles a photo with mini-pictures. The program selects a mini-photo to replace a piece of the photo based on analysing the color content. It works quite well and will give better results when I harness more of my library of photos for the tile selection. Currently the program has about nine thousand mini-pics to choose from. My current stumbling block is finding a way of copying all my library of photos in many folders, and sub and sub sub folders, to another folder without having to use folderBrowserDialog or openFileDialog; in other words without the user having to be involved. I want just the files, not the folders. I'm thinking of using a recursive method but I don't know how to get a file without the user having to select it.
Since I'm still very much a novice programming for Windows I didn't really analyse the project to begin with, so it turned out harder than I imagined. The program is inefficient and lacks elegance but it works and has been fun to work on.
The sparse matrix method I'll by-pass :sick: it's a long time since I did anything with matrices ! There's no need for humans to relate a file to a color signature since the program determines color signatures and picks an appropriate mini-pic from the library of mini-pics.
Zinn
Hi Eri523,
Sorry ! I just re-read your early post:
"Enumerating the files wouldn't be complicated too much by this approach: Directory::GetFiles() and Directory::EnumerateFiles() can be simply instructed to do a recursive scan by passing a simple option value."
I'm sure this will solve the problem.
Zinn
Hi Eri523,
Sorry ! I just re-read your early post:
"Enumerating the files wouldn't be complicated too much by this approach: Directory::GetFiles() and Directory::EnumerateFiles() can be simply instructed to do a recursive scan by passing a simple option value."
I'm sure this will solve the problem.
Zinn
Don't get deterred by all that math stuff. As I said, for the purpose of your image lookup you can simply ignore most of that. In this context, just think of "matrix" as a fancy term for "multi-dimensional array".Quote:
Originally Posted by Zinn
With a coverage as low as 9000 images (0.054%), the DOK representation looks like almost ideal for your data. And it wouln't instantly solve the proximity lookup problem as well, but as, naturally, the keys would be the color values, it would go about half that distance.