Program crashes when working with large array

[RichDrummer]

My program loops through an array of null elements, assigning each element to a new object (an instance of a class I made). The array gets quite large - about 60,000 elements in size. On occasion, this loop will work, but generally the program will crash, giving a generic microsoft error "x.exe has encountered a problem and needs to close...". I have concluded that this is not a memory <i>size</i>issue, but could be a memory management issue of some sort...

Here is the Point class:

public partial class Point
    {
        private string m_ID; // Theoretically to be used only for the transmitter point locations... Could be used for other points too depending on application...
        public double x, y, power;
        public string m_time;

        public Point()
        {
            power = 0;
        }

        public Point(double x, double y)
        {
            this.x = x; this.y = y; power = 0;
        }

        public Point(double x, double y, int column)
        {
            this.x = x; this.y = y; power = 0;
        }

        public Point(double x, double y, int column, string time)
        {
            m_time = time; this.x = x; this.y = y; power = 0;
        }

        public Point(double x, double y, int column, string time, double power)
        {
            m_time = time; this.x = x; this.y = y; this.power = power;
        }

        /// <summary>
        /// THE LABEL OR NAME OF THE GIVEN POINT.
        /// </summary>
        public string ID
        {
            get
            {
                return m_ID;
            }
            set
            {
                m_ID = value;
            }
        }

        public string time
        {
            get
            {
                return m_time;
            }
            set
            {
                m_time = value;
            }
        }
    }

And here is the function which builds the array of Point's:

public Point[] getPoints(ExcelInterface excelData)
        {
            int longColumn = excelData.FindColumnNumber("Longitude");
            int latColumn = excelData.FindColumnNumber("Latitude");
            int timeColumn = excelData.FindColumnNumber("Time");
            int powerColumn = excelData.FindColumnNumber("Power_dBm");

            Point[] dataPoints = new Point[excelData.rows];

            string hh, mm, ss, MM, dd;
            hh = string.Empty; mm = string.Empty; ss = string.Empty; MM = string.Empty; dd = string.Empty;
            int hour, minute, second;
            double power;
            DateTime time;
            double[] thresholds = getPowerThresholds(numThresholds, excelData);
            string dateTime, timeString;

            if (System.DateTime.Now.Month < 10)
                MM = "0" + System.DateTime.Now.Month.ToString();
            else MM = System.DateTime.Now.Month.ToString();

            if (System.DateTime.Now.Day < 10)
                dd = "0" + System.DateTime.Now.Day.ToString ();
            else dd = System.DateTime.Now.Day.ToString();

            string date = System.DateTime.Now.Year + "-" + MM + "-" + dd;
            double lat, lon;

            // array stop between 1900 and 2000 in size...
            for (int i = 2; i < excelData.rows; i++)
            {
                if (i % 400 == 0)
                {
                }

                time = new DateTime(); // DEBUG
                lat = (double)excelData.GetCellValue(i, latColumn, 1);
                lon = (double)excelData.GetCellValue(i, longColumn, 1);
                time = (DateTime)excelData.GetCellValue(i, timeColumn, 1);
                power = (double)excelData.GetCellValue(i, powerColumn, 1);

                hour = time.Hour;
                if (hour < 10)
                    hh = "0" + hour.ToString();
                else hh = hour.ToString();

                minute = time.Minute;
                if (minute < 10)
                    mm = "0" + minute.ToString();
                else mm = minute.ToString();

                second = time.Second;
                if (second < 10)
                    ss = "0" + second.ToString();
                else ss = second.ToString();

                timeString = "T" + hh + ":" + mm + ":" + ss + "Z";
                dateTime = date + timeString;

                try
                {
                    dataPoints[i - 2] = new Point(lon, lat, i, dateTime);  // dataPoints[i - 2] = new Point(lon, lat, i, dateTime);
                }
                catch (Exception ex)
                {

                }
            }

I have also tested this by simplifying the loop where the error occurrs, by using only the constructor:

   for (int i = 2; i < excelData.rows; i++)
            {
                try
                {
                    dataPoints[i - 2] = new Point(0, 0, 0, "");  
                }
                catch (Exception ex)
                {
                }
}

The program still crashed with this test scenario. I have also built a simple test program which solely creates a point array and assigns each array element to a new Point. That program worked just fine with over 1million elements...

I would really appreciate any help - Ive been wrestling with this for days!

Rich

[Arjay]

Two things:

1) Declare your Point class as a struct.
2) Instead of a raw array, try out whether a List<Point> will improve performance or reduce the memory footprint.

[[email protected]]

unless your system is from the 1980's there is no way your computer is crashing creating 60,000 points. The problem is something else. As a simple test bring up the task manager and watch your memory as the program runs.

[RichDrummer]

Yea this array doesnt even make a dent in the memory. I am aware that its not a memory size issue.

[RichDrummer]

2) Instead of a raw array, try out whether a List<Point> will improve performance or reduce the memory footprint.

Arjay, that did the trick . What a simple solution. Thanks.

[RichDrummer]

Ok, so, now i've encountered another issue. Errors seem to occurr in the following code. I never had issues with this code in the past, when working with smaller data sets. What is happening is the program crashes randomly within the foreach loop (could be any line). I have stepped through this and the code seems to be doing everything functionally correctly. The same error message as was ocurring with the array is ocurring here.

        private void plotPoints(List<Point> points, double[] thresholds)
        {
            string node;
            string style = "";

            System.Xml.XPath.XPathNavigator navigator = xDoc.CreateNavigator();
            navigator.MoveToChild("kml", "http://www.opengis.net/kml/2.2");
            navigator.MoveToFirstChild();

            if (isNewKml())
                insertFolder("Points", navigator);

            else // delete folder
            {
                deleteFolder("Points");
                insertFolder("Points", navigator);
            }

            navigator = getNavigator(navigator, "Points");

            foreach (Point p in points)
            {
                if (p.ID != null) // if (p != null)
                {
                    // Errors occur here at random places...
                    // Check power thresholding and apply colors accordingly
                    for (int n = 0; n < thresholds.Length; n++) // Error here
                    {// error here
                        if (p.power < thresholds[n])
                        {
                            style = "CustomStyle" + (n + 1).ToString();
                            break;
                        }
                        else if (n == (thresholds.Length - 1)) // Gets the case where a power measurement is above the average max (20 highest measurements)
                            style = "CustomStyle" + (n + 1).ToString();
                    }

                    node = "<name>Point" + pointIdex + "</name>" + System.Environment.NewLine + "<TimeStamp> <when>" + p.time + "</when> </TimeStamp>" + System.Environment.NewLine + "<styleUrl>" + "#" + style + "</styleUrl>" + System.Environment.NewLine + "<Point> <coordinates>" + p.x + ", " + p.y + ", 0</coordinates> </Point>";

                    navigator.MoveToFirstChild();
                    try
                    {
                        navigator.InsertElementAfter(navigator.Prefix, "Placemark", navigator.LookupNamespace(navigator.Prefix), null); // "Point" + pointIdex
                    }
                    catch (Exception ex)
                    {
                        textBox3.Text = ex.Message;
                    }

                    navigator.MoveToNext();
                    // navigator.MoveToFirstChild(); // not used?
                    navigator.AppendChild(node);
                    pointIdex++;
                    navigator.MoveToParent();
                }
            }
        }

You may mention to use string.format. I know .

Rich

[Arjay]

You may mention to use string.format. I know .

Actually, what I'd like to mention is that you use try/catch more often.

That way, instead of your program crashing, you'll be able to capture a meaningful error.

[Arjay]

Btw, can you give me an idea of the xml points snippet you need to insert at the selected node?