read/write text file (log parsing)

[jkessous]

hi all,
i am a newbie in all the C# world. i have to create an application that will take a certain log and will do parsing to it. first of all it needs to read the lines that contains a certain string and write it to another log with just the hour and part of the line.

the logfile lines that need to be parsed look like this.

Thread#96(               ...............)<<<2011-02-06 19:09:33.648<<<[    INFO]EF2:EFDevice.ValidateConnection calling Connect() ...
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.148<<<[    INFO]EF2:EFDevice.m_Communication_Connected.
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.148<<<[    INFO]EF2:EFDevice.Reset BEGIN Reset of the fence controller 
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.148<<<[    INFO]EF2:EFDevice.Reset Sent: 0xF 
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.258<<<[    INFO]EF2:EFDevice.Reset Sent: 0xF 
Thread#96(               ...............)<<<2011-02-06 19:09:49.367<<<[    INFO]EF2:EFDevice.ValidateConnection Connect() took:15.71875seconds
Thread#96(               ...............)<<<2011-02-06 19:09:49.367<<<[    INFO]EF2: SND 0x0  took:0
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.602<<<[    INFO]EF2:Default sector will be associated with the DefaultPresetName:Preset1
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.883<<<[    INFO]EF17:EFDevice.m_Communication_Connected.
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.883<<<[    INFO]EF17:EFDevice.Reset BEGIN Reset of the fence controller 
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.883<<<[    INFO]EF17:EFDevice.Reset Sent: 0xF 
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.992<<<[    INFO]EF17:EFDevice.Reset Sent: 0xF 
Thread#04(4C Server Main                          )<<<2011-02-06 19:09:50.102<<<[    INFO]EF17:Default sector will be associated with the DefaultPresetName:Preset1
Thread#96(               ...............)<<<2011-02-06 19:09:50.367<<<[    INFO]EF2: SND 0x0  took:0
Thread#22(ConfigurationSetManager                 )<<<2011-02-06 19:09:50.727<<<[    INFO]EF17:Default sector will be associated with the DefaultPresetName:Preset1
Thread#99(               ...............)<<<2011-02-06 19:09:50.727<<<[    INFO]EF17:EFDevice.m_Communication_Disconnected

what i need is to write these kind of lines to a new log which should look like this.

hour EF__ SND hex number

currently i have a code that reads the text file and writes down to another log only the lines that contains "SND". this is currently my code.
I have no clue how to write to the new log from a certain place in the line (EF___)
this is my code

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.IO;

namespace Log_Parser
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            
        }
        private System.IO.StreamWriter theWriter = null;
        private void Form1_Load(object sender, EventArgs e)
        {


        }

        private void button1_Click(object sender, EventArgs e)
        {
            string FilePath = @"C:\weather.txt";

            if (!File.Exists(FilePath))
            {
                MessageBox.Show("The Text File could not be found", "File Missing");
                return;
            }
            else
            try 
        {
            // Create an instance of StreamReader to read from a file.
            // The using statement also closes the StreamReader.
            using (StreamReader sr = new StreamReader(FilePath)) 
            {
                string line;
                // Read and display lines from the file until the end of 
                // the file is reached.
                while ((line = sr.ReadLine()) != null) 
                {

                    if (line.Contains("SND"))
                    {
                        theWriter = new System.IO.StreamWriter(@"C:\maniplog.txt", true);
                        string sYear = DateTime.Now.Year.ToString();
                        string sMonth = DateTime.Now.Month.ToString();
                        string sDay = DateTime.Now.Day.ToString();
                        string sHour = DateTime.Now.Hour.ToString();
                        string sMinutes = DateTime.Now.Minute.ToString();
                        string sSeconds = DateTime.Now.Second.ToString();

                    theWriter.WriteLine(sYear + " " + sMinutes + " " + sSeconds + " " + line);
                    theWriter.Close();
                    }
                    else
                    {
                        Console.WriteLine(line);
                    }
                }
            }
        }
        catch
        {
            // Let the user know what went wrong.
            Console.WriteLine("The file could not be read:");
            //Console.WriteLine(e.Message);
        }
        }
        }

}

thanks alot for the help
Jonathan

[ThermoSight]

perhaps something like the following ? .....

I have taken the liberty of placing your sample log data in an array of strings to avoid the heartache of working with IO. (I wanted to concentrate on the parsing rather than taking the time to write the IO code as well).

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Windows.Forms;

namespace logParser
{
    public partial class Form1 : Form
    {

        static string[] logStgs = new string[] 
            { "Thread#96(...............)<<<2011-02-06 19:09:33.648<<<[    INFO]EF2:EFDevice.ValidateConnection calling Connect() ...",
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.148<<<[    INFO]EF2:EFDevice.m_Communication_Connected.",
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.148<<<[    INFO]EF2:EFDevice.Reset BEGIN Reset of the fence controller", 
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.148<<<[    INFO]EF2:EFDevice.Reset Sent: 0xF", 
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.258<<<[    INFO]EF2:EFDevice.Reset Sent: 0xF", 
"Thread#96(               ...............)<<<2011-02-06 19:09:49.367<<<[    INFO]EF2:EFDevice.ValidateConnection Connect() took:15.71875seconds",
"Thread#96(               ...............)<<<2011-02-06 19:09:49.367<<<[    INFO]EF2: SND 0x0  took:0",
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.602<<<[    INFO]EF2:Default sector will be associated with the DefaultPresetName:Preset1",
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.883<<<[    INFO]EF17:EFDevice.m_Communication_Connected.",
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.883<<<[    INFO]EF17:EFDevice.Reset BEGIN Reset of the fence controller ",
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.883<<<[    INFO]EF17:EFDevice.Reset Sent: 0xF", 
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:49.992<<<[    INFO]EF17:EFDevice.Reset Sent: 0xF", 
"Thread#04(4C Server Main                          )<<<2011-02-06 19:09:50.102<<<[    INFO]EF17:Default sector will be associated with the DefaultPresetName:Preset1",
"Thread#96(               ...............)<<<2011-02-06 19:09:50.367<<<[    INFO]EF2: SND 0x0  took:0",
"Thread#22(ConfigurationSetManager                 )<<<2011-02-06 19:09:50.727<<<[    INFO]EF17:Default sector will be associated with the DefaultPresetName:Preset1",
"Thread#99(               ...............)<<<2011-02-06 19:09:50.727<<<[    INFO]EF17:EFDevice.m_Communication_Disconnected"  };

        static private string _sndPresence = "EF\\d{1,3}:\\s*SND\\s*(?<hexValue>0x\\d{1,3})";
        static private Regex sndPresence = new Regex(_sndPresence);
        static private string _timeEntry = "\\s*(\\d{2}:){2}\\d{2}\\.\\d{3}";
        static private Regex timeEntry = new Regex(_timeEntry);

        static Match sndStringHexNo;

        static string[] sndStrings;


        public Form1()
        {
            InitializeComponent();
        }



        private void button1_Click(object sender, EventArgs e) {
            sndStrings = new string[0];
            List<string> tmp = new List<string>();
            string tmpString;

            // **************************************************************
            // * because I chose to implement this in a button_click handler
            // * I wish to retain the 'void' signature, so the result is a
            // * Form static array. In the real world, a similar function 
            // * would simply return a string array.
            // *
            // * Test each string in the array for the presence of the "SND"
            // * entry. If it is found, we've "captured" the hex value but
            // * must now capture the hour. 
            // *
            // * Get the whole time entry, then split on the colon and take 
            // * the zeroth element (the hour). Then assemble the result 
            // * string and add it to the list.
            // *
            // * When all strings have been examined, convert the list to
            // * an array of strings and kinda-sorta "return" it to the
            // * pseudo-caller via the static array 'sndStrings'.
            // **************************************************************

            foreach (string stg in logStgs) {
                if ((sndStringHexNo = sndPresence.Match(stg)).Success) {
                    tmpString = (Regex.Split(timeEntry.Match(stg).Value, ":"))[0];
                    tmp.Add(tmpString + " EF__ SND " + sndStringHexNo.Groups["hexValue"].Value);
                }// end IF this is a 'SND' log string
            }// end FOREACH string in the log strings array
            if (tmp.Count > 0)
                sndStrings = tmp.ToArray();
        }// end button1_Click handler
    }// end this segment, partial declaration class 'Form1'
}// end this segment, namespace 'logParser'

[jkessous]

Hi,
thanks for the quick reponse.
i had some issues loading this code (i am using 2005 version of visual studio). don thave the System.linq.
so i could not see the result of the output log itself. i will try to keep working on it.
in any case i will need to use he IO as the input log file is a huge log aprrox 400MB.

thanks

[ThermoSight]

Hi Jonathan.

Regarding the System.Linq issue ... not a problem: I'm not doing anything that requires linq. I think you can just delete that using statement. I didn't place it there ... VS2010 did.

Regarding the fact that I used an array of strings ... irrelevant (I think): how the data is supplied to the parsing algorithm seems irrelevant to me.

One strategy (not necessarily the best, but just a strategy) for operating with such a huge file is to read in and process subsets of the data and provide those subsets to the parsing algorithm. Then, after each subset has been parsed, convert the list to an array and write that out.

Will you need some help with writing the IO ? I think it would be rather easily accomplished. At first glance, I don't see anything particularly difficult about it.

Perhaps you could supply a larger subset of the data than what you've posted here. You can reach me at (email address removed) if you're interested in help with the IO. I have some free time.

OldFool

[jkessous]

Thanks for the help.
this is the code i have until now. i need assistance with the following:
how can i make this code to create a new file for each of the EF's that are in the log file.

thanks alot.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.IO;
using System.Text.RegularExpressions; // need this to use regex


namespace Log_Parser
{    
    public partial class Form1 : Form    
    {       public Form1()        
            {            
                InitializeComponent();        
            }

        private System.IO.StreamWriter theWriter = null;        
        private void Form1_Load(object sender, EventArgs e)        
        {  
      
        }

        private void button1_Click(object sender, EventArgs e)        
        {
            string FilePath = @"C:\renewing weather.txt";
            
            if (!File.Exists(FilePath))            
            {                
                MessageBox.Show("The Text File could not be found", "File Missing");                
                return;            
            }

            else            
            {                
                // Create a list to keep a record of lines not written to new log file                
                List<string> list = new List<string>();       
         
                //if the file exist delete it
                string FileName = @"C:\maniplog.csv";
                if (File.Exists(FileName))
                {
                    File.Delete(FileName);
                }


                // Create an instance of StreamWriter to write to the new log file 
                theWriter = new System.IO.StreamWriter(FileName, true);
                
                // Create an instance of StreamReader to read from a file.
                StreamReader sr = new StreamReader(FilePath);

              //  string regExp = @"(EF\d+)\s*:\s*SND\s+(0x\d+)"; //This shows SND messages with the hex
                string regExp = @"(EF\d+)\s*:\s*RCV\sData";
              //  string regExp2 = @"\d{4}-\d{2}-\d{2} (\d{2}):"; //shows only the hour.
                string regExp2 = @"\d{4}-\d{2}-\d{2} (\d{2}:\d{2}:\d{2})."; //will show the entire hour

                try                
                {                    
                    string line;              
      
                    // Read and display lines from the file until the end of                     
                    // the file is reached.                    
                    while ((line = sr.ReadLine()) != null)
                    {                        
                        Match m = Regex.Match(line, regExp);
                        if (m.Success)                        
                        {                           
                            Match m2 = Regex.Match(line, regExp2);
                       //     string hour = (m2.Success) ? m2.Groups[1].Value : "??"; //shows only the hour.
                            string hour = (m2.Success) ? m2.Groups[1].Value : "??:??:??."; //will show the entire hour
                            string EF = m.Groups[1].Value;
                            string hex = m.Groups[2].Value;
                            // theWriter.WriteLine(hour + " " + EF + "," + "SND" + "," + hex); //This shows SND messages with the hex
                            theWriter.WriteLine(hour + " " + EF + "," + "SND");
                        }
                        else                        
                        {                            
                            list.Add(line);                        
                        }                    }                    
             //       string msg = "The following lines did not contain SND:\r\n\r\n";                    
             //       msg += String.Join("\r\n", list.ToArray());
             //       MessageBox.Show(msg); // display lines which were not written
                }
                catch (Exception ex)
                {                    
                    // Let the user know what went wrong.
                    MessageBox.Show("The file could not be read: " + ex.Message); // no console in a Windows Forms app 
                }                
                finally                
                {                    
                    // close writer and reader whatever happens
                    theWriter.Close();                    
                    sr.Close();
                }
            }
        }
    }
}