Reduce Startup memory usage..HOW???

[mmx_nexus]

Hi,

I'm facing an issue with my application startup memory usage(>200 MB) and is looking for some ways to optimize it.

I'm getting a huge XML from the server at startup with more than 1 million nodes..and I’m parsing it to form a data table(using XMLDataDocument class,loading to DOM and using a pre-defined schema) and later building up a dictionary for manipulation of data by the UI controls.

Steps till building dictionary

1. Pre-defined schema is used to prepare data table
2. XML is loaded into DOM for preparing the data table
3. Data table is parsed to prepare a dictionary
4. The whole process of retrieving data till preparing dictionary is running on a background thread which is initiated from the forms constructor.


The XML structure (AuthorName and Book have one to many mapping,Link node is present if it exists and ID can be empty) and the code for populating the dictionary is given below..

XML:

<IntrestedBookSet>
<IntrestedBook AuthorName="XXX" Book="YYY" ID="1"/>
<IntrestedBook AuthorName="XXX" Book="AAA" ID="2"/>
<IntrestedBook AuthorName="XXX" Book="BBB" ID=" "/>
<IntrestedBook AuthorName="CCC" Book="YYY" ID="5" Link="tttt"/>
<IntrestedBook AuthorName="CCC" Book="YZY" ID="7"/>
<IntrestedBook AuthorName="XXX" Book="XDY" ID="10"/>
<IntrestedBook AuthorName="CCC" Book="BBB" ID="3" Link="ssss"/>
</IntrestedBookSet>

Prepare Dicitonary:

Dictionary<string,List<string>> intrestedbooks = new Dictionary<string,List<string>>();
//Get the datatable by using XMLDatadocument methods
Datatable theData = service.GetIntrestedBookSet();
//Prepare the dictionary
foreach(DataRow row in theData.Rows)
{
List<string> books = new List<string>();

if(intrestedBooks.TryGetValue(row["AuthorName"].ToString(),out books))
{
if(!string.IsnullorEmpty(row["Book"].ToString())
  books.Add(row["Book"].ToString());
}
else
{
 books = new List<string>();
intrestedbooks .Add(row["AuthorName"].ToString(),books);
if(!string.IsnullorEmpty(row["Book"].ToString())
  books.Add(row["Book"].ToString());
}
}

Now,after loading the whole set of data into my dictionary, various controls collect information from it at runtime..so cant avoid that.

Let me know if there is any better algorithm to do it.

Thanks in advance,
Mmx

[Tischnoetentoet]

If you want to load a full dictionary in memory, then there aren't many ways to optimize this.

A thing you can try is not load the dictionary into memory but keep it on disk. When the user enters, and does not enter a new key within 3 seconds, you can use an XPath expression to see if the object is inside the xml file on disk. I think using XPath to a disk file is as fast as looping a whole array of 1 million records (of course memory is faster than a hard-disk, but I don't think the user will notice anything as long as you do it in the background).

[Mutant_Fruit]

If you want to load a full dictionary in memory, then there aren't many ways to optimize this.

A thing you can try is not load the dictionary into memory but keep it on disk. When the user enters, and does not enter a new key within 3 seconds, you can use an XPath expression to see if the object is inside the xml file on disk. I think using XPath to a disk file is as fast as looping a whole array of 1 million records (of course memory is faster than a hard-disk, but I don't think the user will notice anything as long as you do it in the background).

Or store the data locally in an sqlite DB and query that. It'd be immensely faster than linearly searching a flatfile on disk. Searching a flatfile would be insanely slow. Even xpath in memory would be insanely slow.

One way to reduce memory usage would be to use an XmlReader to iterate through the data as opposed to loading it all in memory in an XmlDocument. Then just store what you need (authors + books) in your dictionary.

Of course, the biggest *** is that you're querying a server for > 1,000,000 rows when you probably don't need > 1,000,000 rows. Why not just query it for what ya need?

[mmx_nexus]

I guess i can give a try on XMLReader rather than using XmlDocument and prepare the dictionary while looping through the XML.

SqlLite and direct DB querying is not possible due to other reasons..

Will give it a try and let you know folks...

Meanwhile,i still invite opinions...

Thanks,
Mmx