How to prevent outofmemory exeption?

[Red Squirrel]

Not that I know of. This is the code that reads a row:

			Console.WriteLine("Loading Mobiles...");
			
			if(!Core.SQLCON.Query("SELECT * FROM mobiles;"))
			{
				Console.WriteLine("Error loading mobiles!  This is fatal and stuff.  Terminating");
				Console.ReadKey();
				return;
			}
			
			m_Mobiles = new Dictionary<Serial, Mobile>();
			
			
			
			while(Core.SQLCON.FetchRow())
			{
				int serial = Core.SQLCON.Reader.GetInt32(0);
				string serialstr = serial.ToString();
				int typeid = Core.SQLCON.Reader.GetInt32(1);
				string typestr = Core.SQLCON.Reader.GetString(2);
				string serializedata = Core.SQLCON.Reader.GetString(3);

//Do stuff, store serializedata in an array to be used later, then discarded
}

serializedata is the string that gets very large. I checked my DB and the biggest one I got is 80k. I can't see it getting any bigger then 100k per, though, but it is possible. This process is only required at the very start of app startup. Once the string array is processed then the string is broken up into variables (smaller strings, ints, bools, etc).

It's not the best storage engine, idealy I should be structuring everything into a proper SQL structure, but that would require a HUGE rewrite that I'm not ready to do. I rather go this method and just optimize as best as I can.

I was able to store over 1GB of data in a C++ string, and to top it off I was using concatenation with += and it was a non brainer, so I'm sure there must be a way to get C# to allow this too.

And again, ram is not the issue as I have over a gig free during this operation, and even if it was, then it should be using the swap file.

[Zaccheus@Work]

What is Core.SQLCON.Reader ?

Does it have a GetChars method like IDataRecord.GetChars ?

If so, you can avoid reading that string in one go.

[TheCPUWizard]

What is Core.SQLCON.Reader ?

Does it have a GetChars method like IDataRecord.GetChars ?

If so, you can avoid reading that string in one go.

I am 99% certain that IDataRecord.GetChars(..) uses an internal buffer. If this buffer is in managed code, then it does nothing to address the LOH issue. If it is in native code, then it may help.

It would require some fairly deep code analysis to determine, but even that may be a waste of time since the method is NOT documented either way (and thus a change could occur at anytime without violating the contract). Counting on undocumented implementation behaviour is always dangerous.

[Mutant_Fruit]

There are two easy steps to figuring out the cause:

Step 1) Profile your application
Step 2) Diagnose the actual issue

Step1)
The CLRProfiler should be good enough for the task (http://msdn.microsoft.com/en-us/magazine/cc163491.aspx)

Step 2) If you don't understand the output of the CLR Profiler, paste screenshots of the graphs here (or online somewhere) and I'll try and diagnose the issue for you. Until you profile, you're just making random guesses as to what you think is happening.

[Red Squirrel]

The SQLCON object looks like this:

using System;
using System.IO;
using MySql.Data.MySqlClient;
		
namespace Server
{

	public class SqlConnector
	{
		private MySqlConnection m_con;
		private MySqlDataReader m_Reader;	
		public MySqlDataReader Reader { get {return m_Reader;} }
		
		private string m_lasterror;
		
		public string GetLastError()
		{
			return m_lasterror;
		}
		
		private void ThrowError(string query,string error)
		{
                m_lasterror=error;
				Debugger.Write("mysql","Query: {0} ---------- Error: {1}",query,error);
				
				Console.WriteLine("");
				Console.WriteLine("");
				Console.WriteLine("--- SQL ERROR ---");
				Console.WriteLine(query);
				Console.WriteLine("---------");
				Console.WriteLine(error);
				Console.WriteLine("--- END SQL ERROR ---");
				Console.WriteLine("");
				Console.WriteLine("");
		}
		
	
		public bool Connect()
		{
			try
			{
				//connect to mysql server:
							
				string connString = "";
				
				StreamReader tr = new StreamReader("data/sqlconf.txt");			
				connString = tr.ReadLine();			
				tr.Close();
				

				//create your mySQL connection

				m_con = new MySqlConnection(connString);	
				
				m_con.Open();
			}
			catch (Exception ex)
            {
				ThrowError("Connect to DB",ex.ToString());
				return false;
            }
			
			 Debugger.Write("mysql","Connect to DB");
		
			return true;
		}
		
		
		
		public void Disconnect()
		{
			if(m_con==null)return;
			
			Debugger.Write("mysql","Disconnect from DB");
			
			m_con.Close();
		}
		
		
		public bool Query(string query)
		{
			try
			{
				if(m_Reader!=null) m_Reader.Close();
				
				MySqlCommand command = m_con.CreateCommand();
				command.CommandText = query;
							
				m_Reader = command.ExecuteReader();
			}
			catch (Exception ex)
            {
				ThrowError(query,ex.ToString());
				return false;
            }
			
			Debugger.Write("mysql","Query:{0}",query);
		
			return true;
		}

		public bool FetchRow()
		{
			if(m_Reader==null || !m_Reader.Read())return false;
			
		
			return true;
		}
		
		
	}
}

I have not been able to reproduce the issue lately but I'll try to force it some time.

Though if it keeps working properly it may buy me some time to further optimize my load sequence as there's a few things I can do to avoid having to load ALL the strings into memory at once, which may possibly be part of the problem.

[Mutant_Fruit]

Have your process crash with an OOM isn't necessary to see if it's a garbage collection issue. Just run the app through a normal startup sequence (or even better, a really stressful one - load everything three times in a loop) and then look at the results in the memory profiler. If you see a lot of data retained in memory, then you've found an issue. If you don't, then it's not the loading sequence causing the issue.