-
June 28th, 2009, 07:08 PM
#1
Large Structs
I am creating a project that will allow a user to keep track of all the products that we create. I have to create a struct to keep all the data for each product we make. Here's the struct.
Code:
struct PRODUCT_DATA {
int IL;
int freq;
int serial;
int date;
bool recal;
double CorrFacVal;
double CorrFreq;
double std_diff;
double error;
double IL_d_3;
};
The problem: For each product we make, it get's it own struct worth of data. I can't declare the amount of structs that I need, it's too large to be put on the stack.
The usage: (We have around 500k worth of products that we have made and each of them needs their data stored). MyProduct[500000];
When the program is loaded up it will read from a CSV file that will input each product into it's own struct and from there I can modify any existing product, search through them to recover data about it, add more products, or delete products.
Question: Is there a better way to do this? I've looked at SQLite although that seems a bit overkill for what I am trying to do, and a bit complicated for my skill level. I just need one very large array of structs. Around a 500,000k array of these structs.
-
June 28th, 2009, 07:27 PM
#2
Re: Large Structs
The problem: For each product we make, it get's it own struct worth of data. I can't declare the amount of structs that I need, it's too large to be put on the stack.
That's what the heap is for. While 500k of these structs is far to much for the heap, it's only actually about 28MB of data, which is fairly trivial for the average modern PC.
However, it really sounds like you need some kind of database setup, or something better than a simple array of items. If you have to brute-force search your way through 500000 entries for every item lookup, modification or whatever, your program will probably crawl.
-
June 28th, 2009, 07:50 PM
#3
Re: Large Structs
My first reaction is that SQL is the correct response to this design specification.
It may seem overkill, but frankly that's because it's unfamiliar. If you were familiar with any of the many forms of SQL development, this would seem a natural choice for many reasons.
People have the same reaction to the STL, but it's also a much better idea than putting large volumes of data on the stack.
With the volume of entries you're indicating, I'd opt for a larger SQL engine than SQLLite. You're going to want a number of workstations to enter that much data and to manage it, and multiple connections to the free SQL engine from Microsoft would be a much, much nicer way to pull this off.
If you picked up a sample code for a single table, you'd be close to the final product as you need.
If my post was interesting or helpful, perhaps you would consider clicking the 'rate this post' to let me know (middle icon of the group in the upper right of the post).
-
June 28th, 2009, 08:04 PM
#4
Re: Large Structs
Thanks for the replies. I do agree that a database would be a good solution. I've literally spent hours looking through articles on SQLite but could not find any (helpful) examples for a C project. I was looking at SQLite mostly because all I need is one table. 10 columns, 500k rows.
Most of what I COULD find for SQLite was either for a different language or command line interface. I found reference material but not examples to see how the functions are actually being used -- which would help greatly.
-
June 28th, 2009, 08:16 PM
#5
Re: Large Structs
Whatever you come up with you'll want it to be persistent I assume, and that means a database. Pperhaps C++ isn't the right tool. Maybe Access or similar, with it's built in VB scripting would be more appropriate and easier for a novice to get up and running. It's pretty much made for problems like yours.
-
June 28th, 2009, 09:24 PM
#6
Re: Large Structs
I've literally spent hours looking through articles on SQLite but could not find any (helpful) examples for a C project. I was looking at SQLite mostly because all I need is one table. 10 columns, 500k rows.
Then spend about 30 minutes more and download one of the well documented and sampled forms of SQL. You might consider the free MS SQL if you're running in Windows, especially if you're using any version of MS tools that have the MFC objects for database work. They have plenty of samples, probably at least one that's already 50% or more of what you need, already working.
Firebird is another free SQL worth considering.
SQLLite has it's place, but not because of a single table requirement. It's primarily for embedded use, as I recall - that is, the choice of that over something else is more associated with a zero client install of a SQL engine. Your purpose is primarily ease of deployment. Any of those which demonstrate SQL in well documented code would require no more than about one hour to get to a functional position, at least knowing where to go next. Spending more than a few hours in research sounds like a dead end for a lack of documentation resources (it's been a while since I researched SQLLite).
You MIGHT do ok with an XML library. Instead of reading in CSV data, you'd read in (and write out) XML data. This would result in a container of XML data. Boost and the STL have combinations that work for multiple platforms.
For 500K entries I don't recommend this, but if this is how you plan to go, then you need to consider creating a map as a reference, so you can find data according to important criteria. It can work, but one workstation creating and/or maintaining a database of 500k entries sounds impractical to me, which is why SQLLite might not be a good choice either (it's single user in the embedded mode).
If my post was interesting or helpful, perhaps you would consider clicking the 'rate this post' to let me know (middle icon of the group in the upper right of the post).
-
June 29th, 2009, 11:17 AM
#7
Re: Large Structs
Originally Posted by dmoore210
I've literally spent hours looking through articles on SQLite but could not find any (helpful) examples for a C project. (...)
Most of what I COULD find for SQLite was either for a different language or command line interface. I found reference material but not examples to see how the functions are actually being used -- which would help greatly.
The main SQLite documentation is an excellent reference, but a rather poor tutorial. You can find C code samples in the SQLite wiki, but frankly I think you would be better off searching the Web for tutorials. You could also borrow Michael Owens' The Definitive Guide to SQLite from your local library, though it is not so definitive as the recommended C interface has changed slightly since the time the book was written.
Originally Posted by JVene
With the volume of entries you're indicating, I'd opt for a larger SQL engine than SQLLite. You're going to want a number of workstations to enter that much data and to manage it, and multiple connections to the free SQL engine from Microsoft would be a much, much nicer way to pull this off.
If we're talking about 500K entries, I believe SQLite would still handle it fine. The problem would be if we are talking about the same database being accessed simultaneously over a network. See Appropriate Uses For SQLite.
-
June 29th, 2009, 03:43 PM
#8
Re: Large Structs
I am going to stick with SQLite -- mostly *because* it is embedded, the way it's coded in will make it invisible to the end-user, and it will not be used over a network, it's just to handle a few large tables.
Ive started looking at some tutorials -- most are out of date and use the functions incorrectly now. And the reference material IS good, but the tutorials are horrible! I am able to create a table and add items to the table.
I am just simply writing the appropriate SQL command into a string and passing it into sqlite3_exec( ); So getting data out this way I don't think is the way to go, I have to use sqlite3_prepare... although finding the appropriate way to use this is a task
-
June 29th, 2009, 05:47 PM
#9
Re: Large Structs
500k isn't really all that much. I occasionally have sets of 50 million, looking at the struct that you have, that's really not that much RAM / disk space. You could optimize your searching by putting your data into a tree of some sort. Take the most commonly requested field, and sort them by that.
Don't use an array for this, use a vector.
-
June 29th, 2009, 06:26 PM
#10
Re: Large Structs
Originally Posted by ninja9578
500k isn't really all that much. I occasionally have sets of 50 million, looking at the struct that you have, that's really not that much RAM / disk space. You could optimize your searching by putting your data into a tree of some sort. Take the most commonly requested field, and sort them by that.
Don't use an array for this, use a vector.
But he's still going to need the data to persist between runs of the app I assume. Reading it all in and saving it all out would be time consuming. This app really calls for a database.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|