CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 10 of 10

Thread: Large Structs

  1. #1
    Join Date
    May 2009
    Posts
    4

    Large Structs

    I am creating a project that will allow a user to keep track of all the products that we create. I have to create a struct to keep all the data for each product we make. Here's the struct.

    Code:
    struct PRODUCT_DATA {
        int    IL;         
        int    freq;      
        int    serial;    
        int    date;
        bool   recal;        
        double CorrFacVal;     
        double CorrFreq;     
        double std_diff;          
        double error;          
        double IL_d_3;    
    };
    The problem: For each product we make, it get's it own struct worth of data. I can't declare the amount of structs that I need, it's too large to be put on the stack.

    The usage: (We have around 500k worth of products that we have made and each of them needs their data stored). MyProduct[500000];

    When the program is loaded up it will read from a CSV file that will input each product into it's own struct and from there I can modify any existing product, search through them to recover data about it, add more products, or delete products.

    Question: Is there a better way to do this? I've looked at SQLite although that seems a bit overkill for what I am trying to do, and a bit complicated for my skill level. I just need one very large array of structs. Around a 500,000k array of these structs.

  2. #2
    Join Date
    Aug 2007
    Posts
    858

    Re: Large Structs

    The problem: For each product we make, it get's it own struct worth of data. I can't declare the amount of structs that I need, it's too large to be put on the stack.
    That's what the heap is for. While 500k of these structs is far to much for the heap, it's only actually about 28MB of data, which is fairly trivial for the average modern PC.

    However, it really sounds like you need some kind of database setup, or something better than a simple array of items. If you have to brute-force search your way through 500000 entries for every item lookup, modification or whatever, your program will probably crawl.

  3. #3
    Join Date
    Nov 2006
    Posts
    1,611

    Re: Large Structs

    My first reaction is that SQL is the correct response to this design specification.

    It may seem overkill, but frankly that's because it's unfamiliar. If you were familiar with any of the many forms of SQL development, this would seem a natural choice for many reasons.

    People have the same reaction to the STL, but it's also a much better idea than putting large volumes of data on the stack.

    With the volume of entries you're indicating, I'd opt for a larger SQL engine than SQLLite. You're going to want a number of workstations to enter that much data and to manage it, and multiple connections to the free SQL engine from Microsoft would be a much, much nicer way to pull this off.

    If you picked up a sample code for a single table, you'd be close to the final product as you need.
    If my post was interesting or helpful, perhaps you would consider clicking the 'rate this post' to let me know (middle icon of the group in the upper right of the post).

  4. #4
    Join Date
    May 2009
    Posts
    4

    Re: Large Structs

    Thanks for the replies. I do agree that a database would be a good solution. I've literally spent hours looking through articles on SQLite but could not find any (helpful) examples for a C project. I was looking at SQLite mostly because all I need is one table. 10 columns, 500k rows.

    Most of what I COULD find for SQLite was either for a different language or command line interface. I found reference material but not examples to see how the functions are actually being used -- which would help greatly.

  5. #5
    GCDEF is offline Elite Member Power Poster
    Join Date
    Nov 2003
    Location
    Florida
    Posts
    12,635

    Re: Large Structs

    Whatever you come up with you'll want it to be persistent I assume, and that means a database. Pperhaps C++ isn't the right tool. Maybe Access or similar, with it's built in VB scripting would be more appropriate and easier for a novice to get up and running. It's pretty much made for problems like yours.

  6. #6
    Join Date
    Nov 2006
    Posts
    1,611

    Re: Large Structs

    I've literally spent hours looking through articles on SQLite but could not find any (helpful) examples for a C project. I was looking at SQLite mostly because all I need is one table. 10 columns, 500k rows.

    Then spend about 30 minutes more and download one of the well documented and sampled forms of SQL. You might consider the free MS SQL if you're running in Windows, especially if you're using any version of MS tools that have the MFC objects for database work. They have plenty of samples, probably at least one that's already 50% or more of what you need, already working.

    Firebird is another free SQL worth considering.

    SQLLite has it's place, but not because of a single table requirement. It's primarily for embedded use, as I recall - that is, the choice of that over something else is more associated with a zero client install of a SQL engine. Your purpose is primarily ease of deployment. Any of those which demonstrate SQL in well documented code would require no more than about one hour to get to a functional position, at least knowing where to go next. Spending more than a few hours in research sounds like a dead end for a lack of documentation resources (it's been a while since I researched SQLLite).

    You MIGHT do ok with an XML library. Instead of reading in CSV data, you'd read in (and write out) XML data. This would result in a container of XML data. Boost and the STL have combinations that work for multiple platforms.

    For 500K entries I don't recommend this, but if this is how you plan to go, then you need to consider creating a map as a reference, so you can find data according to important criteria. It can work, but one workstation creating and/or maintaining a database of 500k entries sounds impractical to me, which is why SQLLite might not be a good choice either (it's single user in the embedded mode).
    If my post was interesting or helpful, perhaps you would consider clicking the 'rate this post' to let me know (middle icon of the group in the upper right of the post).

  7. #7
    Join Date
    Jan 2006
    Location
    Singapore
    Posts
    6,765

    Re: Large Structs

    Quote Originally Posted by dmoore210
    I've literally spent hours looking through articles on SQLite but could not find any (helpful) examples for a C project. (...)

    Most of what I COULD find for SQLite was either for a different language or command line interface. I found reference material but not examples to see how the functions are actually being used -- which would help greatly.
    The main SQLite documentation is an excellent reference, but a rather poor tutorial. You can find C code samples in the SQLite wiki, but frankly I think you would be better off searching the Web for tutorials. You could also borrow Michael Owens' The Definitive Guide to SQLite from your local library, though it is not so definitive as the recommended C interface has changed slightly since the time the book was written.

    Quote Originally Posted by JVene
    With the volume of entries you're indicating, I'd opt for a larger SQL engine than SQLLite. You're going to want a number of workstations to enter that much data and to manage it, and multiple connections to the free SQL engine from Microsoft would be a much, much nicer way to pull this off.
    If we're talking about 500K entries, I believe SQLite would still handle it fine. The problem would be if we are talking about the same database being accessed simultaneously over a network. See Appropriate Uses For SQLite.
    C + C++ Compiler: MinGW port of GCC
    Build + Version Control System: SCons + Bazaar

    Look up a C/C++ Reference and learn How To Ask Questions The Smart Way
    Kindly rate my posts if you found them useful

  8. #8
    Join Date
    May 2009
    Posts
    4

    Re: Large Structs

    I am going to stick with SQLite -- mostly *because* it is embedded, the way it's coded in will make it invisible to the end-user, and it will not be used over a network, it's just to handle a few large tables.

    Ive started looking at some tutorials -- most are out of date and use the functions incorrectly now. And the reference material IS good, but the tutorials are horrible! I am able to create a table and add items to the table.

    I am just simply writing the appropriate SQL command into a string and passing it into sqlite3_exec( ); So getting data out this way I don't think is the way to go, I have to use sqlite3_prepare... although finding the appropriate way to use this is a task

  9. #9
    Join Date
    Jan 2009
    Posts
    1,689

    Re: Large Structs

    500k isn't really all that much. I occasionally have sets of 50 million, looking at the struct that you have, that's really not that much RAM / disk space. You could optimize your searching by putting your data into a tree of some sort. Take the most commonly requested field, and sort them by that.

    Don't use an array for this, use a vector.

  10. #10
    GCDEF is offline Elite Member Power Poster
    Join Date
    Nov 2003
    Location
    Florida
    Posts
    12,635

    Re: Large Structs

    Quote Originally Posted by ninja9578 View Post
    500k isn't really all that much. I occasionally have sets of 50 million, looking at the struct that you have, that's really not that much RAM / disk space. You could optimize your searching by putting your data into a tree of some sort. Take the most commonly requested field, and sort them by that.

    Don't use an array for this, use a vector.
    But he's still going to need the data to persist between runs of the app I assume. Reading it all in and saving it all out would be time consuming. This app really calls for a database.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured