Printable View
I have an app that has to load large amounts of data in chunks (maybe like, half a MB in a string) but for some reason I'll get out of memory errors even though there is TONS of ram left + page file. The memory used is then cleared once the string has been processed.
Is there a way to make the program less sensible to large amounts of ram being used? If the ram is there to use, then it should use it instead of crapping out.
OutOfMemory exception usually occurs when you are trying to save something whereas the variable that you are trying to save it in does not have enough memory. It does not mean that if you declare a string object, if there is 1 GB memory free on your system, it will not use 1 GB, it will only use the specified size.
I would suggest using stringbuilder instead of a string if your data is too large.
Remember objects over 80,000 bytes (that is 40,000 chars in Unicode) get allocated on the LArgeObject Heap, which is sucbject to fragmentation.Quote:
Originally Posted by Red Squirrel
Use your debugger tools to look at how the memory is being used at the time of the OOM.
How would I go about ensuring it does not get fragmented?
I've also read I should try to reuse objects instead of redeclaring, will that help? Right now the string that holds all the data gets redeclared for each object (over 800k) some are small while some are very large.
Dont allocate and release blocks of differig sizes that are over 80,000 bytes in size. :rolleyes:Quote:
Originally Posted by Red Squirrel
[qupte]
I've also read I should try to reuse objects instead of redeclaring, will that help?
[/quote]
Re-Read above...
[/quote]Right now the string that holds all the data gets redeclared for each object (over 800k) some are small while some are very large.[/QUOTE]
Strings are immutable, and ALWAYs involve reallocation.
If you combine these, irt means one very simple thing...A robust application should not have strings over 40,000 (actually slightly smaller). PERIOD.
Do you actually need all that in memory at the same time? If so, do you actually need it to be a single chunk of memory? As TheCPUWizard said, you're more than likely hitting issues with the large object heap. If you avoided having to have all that data in-memory in one single chunk of memory, you'd completely avoid the issue.Quote:
Originally Posted by Red Squirrel
Strings are immutable, and ALWAYs involve reallocation.Quote:
Originally Posted by TheCPUWizard
If you combine these, irt means one very simple thing...A robust application should not have strings over 40,000 (actually slightly smaller). PERIOD.
[/quote]
I don't have a choice since I need to load that data into memory, and it comes from a SQL database so it needs to be taken in one chunk, processed, then move to next row. Thing is, it's freed up after.
What's the point of having 3GB of ram FREE if I can't even use it? If I had the choice this app would be written in C++ which has less limitations but I don't have a choice here. There's got to be a way to get around these errors without "not using that much ram in one variable" since I'm not even using close to what there is available.
Yes these strings have to be in single chunks. The way the app was engineered at first was not very SQL friendly, but I could not keep it using files as it was getting inefficient. So now each Item, Mobile, etc is stored as a string of data into a SQL row. This row is then read and put in a string, this string is then processed and the individual variables of the object are populated, then the string is discarded. This process only happens when the world has to load.Quote:
Originally Posted by Mutant_Fruit
The program is for a game server so we're talking TONS of data here, of various lengths. I will be optimizing things as I go. The biggest culpit now is house data so I may find another way to store these but I can't see how this method can't work considering I have enough ram anyway.
Eventually I may convert this app to C++ to overcome .net / C# limitations, but that's a long way ahead. I prefer dealing with pointers and stuff as at least I have full control over memory management.
I sincerely doubt that you will be able to get a stable solution without a complete re-design. This one symptom alone indicates that the developer(s) had no clue as to how to write a proper .NET application.
Translation....
If I was approached to take this on as a paid contract, the ONLY way I would approach the job is as a complete re-design and re-write. REPEATED experience has shown that unless an application is designed from the ground up following best practices, that attempting to retro-fit it, is unlikely to succeed and will almost always result in major cost over runs, and a low probability of success...
I agree with that, and that is actually plan B, to redo the whole thing from scratch, but it will take a very long time, and I wish to end this project by 2010. So I'm hoping there's a way around this issue such as a function of some sort to make it defragment the memory (probably very often) durring the load process.Quote:
Originally Posted by TheCPUWizard
An idea I also had was to have a C++ app read the DB and dump the contents to file, then have the program read from the file this way I can control my buffer size, but this will add more complexity as well.
I was testing and I can hold a few GB of data in a single string. I had a loop that just kept concatenating a char.
I'll bet you that if you take the "lets fix it incrementally" approach that you will NOT have a robust working product by the end of 2010.....
There is NO way to defragment the LOH. EVERY application developer should be using a performance monitoring tool while they are writing and testing code and be measuring the number of LOH allocations, and their size distribution, ESPECIALLY if they have a program that they intend to be continously executing (ie a "sever" program).
Hmm so guess I'll have to have a secondary app that reads the data from the DB then outputs to file then. This way I can read it in smaller buffers. Thing is, it's not like I'm keeping those strings filled, it's just a buffer that is temp storage for a SQL row and then formatted properly in the program. a proper DB structure is not an option for this app unless I recode it from scratch which I'm trying to avoid.
Yes if I do it from scratch it would be 10 times more efficient, but I just don't want to spend that much time on it.
I've asked myself many times if I should though as I keep running into road blocks and it's usually a limitation of C# or just something that is horribly coded.
I have yet to find a SINGLE instance of the former (limitation of the language), but have professionally encountered thousands of the latter (horrible code).Quote:
Originally Posted by Red Squirrel
Quote:
The program is for a game server so we're talking TONS of data here, of various lengths..
C# should not be a limitation when writing a game server. There are some massive games out there using C# which have no scalability issues. SecondLife would be the biggest one that I can think of right now. Bear in mind that if decide to rewrite in C++, you'll be starting from scratch. Maybe it'd be better to just start from scratch and rewrite it properly in C#.Quote:
Eventually I may convert this app to C++ to overcome .net / C# limitations, but that's a long way ahead. I prefer dealing with pointers and stuff as at least I have full control over memory management.
If performance is a concern and the current architecture does not allow for performance to scale, then the current architecture design needs to revisited and altered. If that requires a rewrite, then so be it. It's better to do it now rather than 3 months before the release. This is a question you need to answer asap.Quote:
I've asked myself many times if I should though as I keep running into road blocks and it's usually a limitation of C# or just something that is horribly coded.
Does the whole row need to be read in one go?Quote:
Originally Posted by Red Squirrel
Is there no way to read part of a row?
Not that I know of. This is the code that reads a row:
Code:Console.WriteLine("Loading Mobiles...");
if(!Core.SQLCON.Query("SELECT * FROM mobiles;"))
{
Console.WriteLine("Error loading mobiles! This is fatal and stuff. Terminating");
Console.ReadKey();
return;
}
m_Mobiles = new Dictionary<Serial, Mobile>();
while(Core.SQLCON.FetchRow())
{
int serial = Core.SQLCON.Reader.GetInt32(0);
string serialstr = serial.ToString();
int typeid = Core.SQLCON.Reader.GetInt32(1);
string typestr = Core.SQLCON.Reader.GetString(2);
string serializedata = Core.SQLCON.Reader.GetString(3);
//Do stuff, store serializedata in an array to be used later, then discarded
}
serializedata is the string that gets very large. I checked my DB and the biggest one I got is 80k. I can't see it getting any bigger then 100k per, though, but it is possible. This process is only required at the very start of app startup. Once the string array is processed then the string is broken up into variables (smaller strings, ints, bools, etc).
It's not the best storage engine, idealy I should be structuring everything into a proper SQL structure, but that would require a HUGE rewrite that I'm not ready to do. I rather go this method and just optimize as best as I can.
I was able to store over 1GB of data in a C++ string, and to top it off I was using concatenation with += and it was a non brainer, so I'm sure there must be a way to get C# to allow this too.
And again, ram is not the issue as I have over a gig free during this operation, and even if it was, then it should be using the swap file.
What is Core.SQLCON.Reader ?
Does it have a GetChars method like IDataRecord.GetChars ?
If so, you can avoid reading that string in one go.
I am 99% certain that IDataRecord.GetChars(..) uses an internal buffer. If this buffer is in managed code, then it does nothing to address the LOH issue. If it is in native code, then it may help.Quote:
Originally Posted by Zaccheus@Work
It would require some fairly deep code analysis to determine, but even that may be a waste of time since the method is NOT documented either way (and thus a change could occur at anytime without violating the contract). Counting on undocumented implementation behaviour is always dangerous.
There are two easy steps to figuring out the cause:
Step 1) Profile your application
Step 2) Diagnose the actual issue
Step1)
The CLRProfiler should be good enough for the task (http://msdn.microsoft.com/en-us/magazine/cc163491.aspx)
Step 2) If you don't understand the output of the CLR Profiler, paste screenshots of the graphs here (or online somewhere) and I'll try and diagnose the issue for you. Until you profile, you're just making random guesses as to what you think is happening.
The SQLCON object looks like this:
Code:using System;
using System.IO;
using MySql.Data.MySqlClient;
namespace Server
{
public class SqlConnector
{
private MySqlConnection m_con;
private MySqlDataReader m_Reader;
public MySqlDataReader Reader { get {return m_Reader;} }
private string m_lasterror;
public string GetLastError()
{
return m_lasterror;
}
private void ThrowError(string query,string error)
{
m_lasterror=error;
Debugger.Write("mysql","Query: {0} ---------- Error: {1}",query,error);
Console.WriteLine("");
Console.WriteLine("");
Console.WriteLine("--- SQL ERROR ---");
Console.WriteLine(query);
Console.WriteLine("---------");
Console.WriteLine(error);
Console.WriteLine("--- END SQL ERROR ---");
Console.WriteLine("");
Console.WriteLine("");
}
public bool Connect()
{
try
{
//connect to mysql server:
string connString = "";
StreamReader tr = new StreamReader("data/sqlconf.txt");
connString = tr.ReadLine();
tr.Close();
//create your mySQL connection
m_con = new MySqlConnection(connString);
m_con.Open();
}
catch (Exception ex)
{
ThrowError("Connect to DB",ex.ToString());
return false;
}
Debugger.Write("mysql","Connect to DB");
return true;
}
public void Disconnect()
{
if(m_con==null)return;
Debugger.Write("mysql","Disconnect from DB");
m_con.Close();
}
public bool Query(string query)
{
try
{
if(m_Reader!=null) m_Reader.Close();
MySqlCommand command = m_con.CreateCommand();
command.CommandText = query;
m_Reader = command.ExecuteReader();
}
catch (Exception ex)
{
ThrowError(query,ex.ToString());
return false;
}
Debugger.Write("mysql","Query:{0}",query);
return true;
}
public bool FetchRow()
{
if(m_Reader==null || !m_Reader.Read())return false;
return true;
}
}
}
I have not been able to reproduce the issue lately but I'll try to force it some time.
Though if it keeps working properly it may buy me some time to further optimize my load sequence as there's a few things I can do to avoid having to load ALL the strings into memory at once, which may possibly be part of the problem.
Have your process crash with an OOM isn't necessary to see if it's a garbage collection issue. Just run the app through a normal startup sequence (or even better, a really stressful one - load everything three times in a loop) and then look at the results in the memory profiler. If you see a lot of data retained in memory, then you've found an issue. If you don't, then it's not the loading sequence causing the issue.