dcsimg
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 4 of 4

Thread: Reduce Startup memory usage..HOW???

Hybrid View

  1. #1
    Join Date
    Apr 2004
    Posts
    123

    Reduce Startup memory usage..HOW???

    Hi,

    I'm facing an issue with my application startup memory usage(>200 MB) and is looking for some ways to optimize it.

    I'm getting a huge XML from the server at startup with more than 1 million nodes..and Iím parsing it to form a data table(using XMLDataDocument class,loading to DOM and using a pre-defined schema) and later building up a dictionary for manipulation of data by the UI controls.

    Steps till building dictionary

    1. Pre-defined schema is used to prepare data table
    2. XML is loaded into DOM for preparing the data table
    3. Data table is parsed to prepare a dictionary
    4. The whole process of retrieving data till preparing dictionary is running on a background thread which is initiated from the forms constructor.


    The XML structure (AuthorName and Book have one to many mapping,Link node is present if it exists and ID can be empty) and the code for populating the dictionary is given below..

    XML:
    Code:
    <IntrestedBookSet>
    <IntrestedBook AuthorName="XXX" Book="YYY" ID="1"/>
    <IntrestedBook AuthorName="XXX" Book="AAA" ID="2"/>
    <IntrestedBook AuthorName="XXX" Book="BBB" ID=" "/>
    <IntrestedBook AuthorName="CCC" Book="YYY" ID="5" Link="tttt"/>
    <IntrestedBook AuthorName="CCC" Book="YZY" ID="7"/>
    <IntrestedBook AuthorName="XXX" Book="XDY" ID="10"/>
    <IntrestedBook AuthorName="CCC" Book="BBB" ID="3" Link="ssss"/>
    </IntrestedBookSet>
    Prepare Dicitonary:
    Code:
    Dictionary<string,List<string>> intrestedbooks = new Dictionary<string,List<string>>();
    //Get the datatable by using XMLDatadocument methods
    Datatable theData = service.GetIntrestedBookSet();
    //Prepare the dictionary
    foreach(DataRow row in theData.Rows)
    {
    List<string> books = new List<string>();
    
    if(intrestedBooks.TryGetValue(row["AuthorName"].ToString(),out books))
    {
    if(!string.IsnullorEmpty(row["Book"].ToString())
      books.Add(row["Book"].ToString());
    }
    else
    {
     books = new List<string>();
    intrestedbooks .Add(row["AuthorName"].ToString(),books);
    if(!string.IsnullorEmpty(row["Book"].ToString())
      books.Add(row["Book"].ToString());
    }
    }
    Now,after loading the whole set of data into my dictionary, various controls collect information from it at runtime..so cant avoid that.

    Let me know if there is any better algorithm to do it.

    Thanks in advance,
    Mmx

  2. #2
    Join Date
    Nov 2003
    Posts
    2,185

    Re: Reduce Startup memory usage..HOW???

    If you want to load a full dictionary in memory, then there aren't many ways to optimize this.

    A thing you can try is not load the dictionary into memory but keep it on disk. When the user enters, and does not enter a new key within 3 seconds, you can use an XPath expression to see if the object is inside the xml file on disk. I think using XPath to a disk file is as fast as looping a whole array of 1 million records (of course memory is faster than a hard-disk, but I don't think the user will notice anything as long as you do it in the background).

  3. #3
    Join Date
    May 2007
    Posts
    1,546

    Re: Reduce Startup memory usage..HOW???

    Quote Originally Posted by Tischnoetentoet
    If you want to load a full dictionary in memory, then there aren't many ways to optimize this.

    A thing you can try is not load the dictionary into memory but keep it on disk. When the user enters, and does not enter a new key within 3 seconds, you can use an XPath expression to see if the object is inside the xml file on disk. I think using XPath to a disk file is as fast as looping a whole array of 1 million records (of course memory is faster than a hard-disk, but I don't think the user will notice anything as long as you do it in the background).
    Or store the data locally in an sqlite DB and query that. It'd be immensely faster than linearly searching a flatfile on disk. Searching a flatfile would be insanely slow. Even xpath in memory would be insanely slow.

    One way to reduce memory usage would be to use an XmlReader to iterate through the data as opposed to loading it all in memory in an XmlDocument. Then just store what you need (authors + books) in your dictionary.

    Of course, the biggest *** is that you're querying a server for > 1,000,000 rows when you probably don't need > 1,000,000 rows. Why not just query it for what ya need?
    www.monotorrent.com For all your .NET bittorrent needs

    NOTE: My code snippets are just snippets. They demonstrate an idea which can be adapted by you to solve your problem. They are not 100% complete and fully functional solutions equipped with error handling.

  4. #4
    Join Date
    Apr 2004
    Posts
    123

    Re: Reduce Startup memory usage..HOW???

    I guess i can give a try on XMLReader rather than using XmlDocument and prepare the dictionary while looping through the XML.

    SqlLite and direct DB querying is not possible due to other reasons..

    Will give it a try and let you know folks...

    Meanwhile,i still invite opinions...

    Thanks,
    Mmx

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width




On-Demand Webinars (sponsored)