[RESOLVED] Reading a .txt file in real time

hey guys, I've been trying to find a way to read a .txt file in real time. I've developed a client/server console app program in Visual Studio 2010. I can open and send txt files to my sever program with my client program (tcp, netstream). However, the file that I am calling is constantly being updated with new data. I want to sent that updated data straight to my File Sharing Watcher function but this merely tells me if my file or directory has changed not the actual string of data. Does anyone know of any other function or way of achieving this?

It would help to know how the file may be changed. If data is being appended then it is simple just keep track of the filesize and read to the end starting at the position of the size of the file at the last read. If data is being changed within the file then it is a bit more complicated.

Your server program could write each incoming block of text to a new file. Eg: 1.txt, 2.txt, 3.txt (or even get the system to generate a temp file name for you, so you don't have to keep a counter). Then your FileSystemWatcher could just look for files being created. These files will just have the new text and of course they can be deleted afterwards or appended to a main file and deleted.

I've done some work with FileSystemWatcher in the past and when opening one of these new files, you may find that it is still locked by the process writing to it and have to catch the exception (only way to do this) wait a second or 2 and then open it again.

I appreciate the replies. DataMiser, I basically want to mimic the function of the Tail application. I am working with sensor readings. These readings are sent to a file in intervals of 8 seconds. My client program reads the .txt file with these sensor readings and sends them to my server. Therefore, I want my program to read the .txt file until the end and wait for the next string to be added. rliq, I understand what you are saying about writing new data to a temp file. However, FileSystemWatcher would not be able to tell me the exact string that was updated right?

So new data will always be appended to the end of the file which makes it fairly simple to know what was written.

Basically what you could do is read the file initially and store the size of the file at that point.
When the file changes look at the stored file size then read the file starting at that location to the end of the file. Which will in effect be reading only the new data. Update your variable to reflect the new file size as of this read.

I would use a filestream and binaryreader to do this since it is simple to read based on byte position and number of bytes to read but there are other options.

If you write each block of data to a different temp file each time, then the whole of its contents is the string that was written. Then append the whole of that text to the end of another file and delete the temp file. It's not quite the UNIX "tail -f" command, but it may serve your purpose.

I was under the impression that the file is being written by another program. If you are writing it yourself then the suggestion above is a good one.

Thank you both for your suggestions, they have helped me big time. Yes Datamiser, the file is actually being written by another program. The sensor readings I was telling you guys about actually send the data to the computer via serial communication. Therefore, I figured it would be easier to develop a COM port listener and have my client program read that data as the event happens. I'm having some issues with my COM port listener. Could you help? Thanks.

What kind of issues are you having?

Ok so as I mentioned before, I have some sensors sending data to a file. I want to listen to the port directly (COM 4). I set all the parameters and when I try to listen for incoming data, my application shuts down. It doesn't listen at all. Its a small program. I have attached my entire COM listener program. I know the problem is in the part where I receive data, I just dont know why? it seems like everyone is using the DataReceive Method.

I have not tried writing this kind of app as a console app and I am far from an expert in C# but I do not see any thing in your program to keep it from just dropping through main and exiting.

Am I missing something that allows the program to stay running?

Normally I would do this with a windows forms app and the form being open would stop the project from closing while I wait for events on the port.

Im new at C# as well. I'll try doing this same project with a windows form app. I am assuming the program will keep running or listening since I open the port and never close it. It's actually what I what, but Im not sure if I still need that (close port) to have a working program. Thanks for your insight. I appreciate it.

In a forms app the program keeps running until you close the form. The port will keep receiving data so long as the port is open and data is coming into the port.

How are you intercepting the data from the port? Are you tied into the cable with a splitter or something so you can get the data and the other program still get it as well?

So the sensors are transmitting to a device and this device is connected to my pc via rs 232. The data that is being transmitted by this device is in the for of strings. I have a little third party terminal app that listens to the COM port and it is receiving data so I know its transmitting. My app is the one not receiving anything.

I was wondering how you are connected. Have you disabled the other program and now using yours to monitor the port or is the other program still running at the same time? Have you converted to a forms app and still not receive anything? Have you set the receive threshold to a value >0 so as to trigger the data received event when data arrives?

Look up SignalIR on CodePlex. It's designed for real-time communications

DataMiser, the sensor receiver is connected to my pc via rs232. So once I run my program I am listening to COM port 4 which is where the receiver is configured to send. I did not convert the program to a windows form app but my program is working now. I set the receive threshold to a value >0 and I coded the serial parameters myself instead of asking for input. Now my program runs a loop that listens for incoming traffic and receives it. Now I cant get the formatting right. The receiver is sending two lines of data and I am receiving chunks of it and my handler is being fired about 8 times per reading. I know this is common when working with serial ports, but it reacts the same even when I try to convert the bytes to string. Check out my handler method:

//retrieve number of bytes in the buffer
int bytes = _serialPort.BytesToRead;
//create a byte array to hold the awaiting data
byte[] dataBuffer = new byte[bytes];
//Read the data and store it
_serialPort.Read(dataBuffer, 0, bytes);
//convert to string.
string s = System.Text.ASCIIEncoding.ASCII.GetString(dataBuffer);
//display the data
Console.WriteLine("Data" + s + " was received at " + DateTime.Now);

Well if I remember correctly isn't there a readline() method? If so you could use this to read a line at a time. Or you could use another var that will que up the incoming data until it contains one or more CR characters then parse it as needed and write to disk.

What exactly is causing you a problem?

Yes, I used ReadLine() but that didnt work for me. I used:

string data = _serialPort.ReadLine();
Console.WriteLine(data);

I don't know if there is more to it than that but when I used that my program would only tell me that I had received data, not show the actual data. I guess whats causing my problem is my

string s = System.Text.ASCIIEncoding.ASCII.GetString(dataBuffer); line.

The data that is being sent to me has the following format:
ID:34242 State:NY
Zip:12345 StreetType:Ave

when it is received by my program it displays it like this:

ID:34242
State:N
Y Zip:12
345 Street
Type:Ave

So not only does some data get pushed to the next line, but it doesn't get formatted into one string.

So you need to que up the data and print it when you have enough. Create a variable with a larger scope so that it holds it's value. For example if you create it at the top of the class.

Rather than displaying s append s to this que var and then check to see what this var holds and display when it has the data you want.

Or if you simply want to update a file as I thought you were trying to do you just append whatever you receive to the file either once several bytes have been received or anytime something is received. Just be sure not to add a CRFL to the end of the data where it is not needed.

Yes, I do want to send the data to a file. I am simply displaying it on the console app so that I can see the string as its coming to through the port and monitoring the rest of the readings. I've tried what you have suggested. It has improved the formatting to a certain degree but the spacing is off and im still getting several iterations of the same reading. Ive attached my code so you can take a look at the modified DataReceive Event Handler.

so this is what the actual data from the sensors look like in its raw form:

6001:0 Temp=66
S -158 N -125 C 89

This will help you see why I made the decision about the regex statement. This is the the form I want it in: 6001:0 Temp=66 S -158 N -125 C 89. My reason for having the string in this form is to than split the string and send each field to a certain column in a db table.

Why would you create a new event handler inside this loop?

Code:

---

while (_serialPort.IsOpen)
                        {
                            Thread.Sleep(1000);
                            _serialPort.DataReceived += new SerialDataReceivedEventHandler(Port_DataReceived);
                        }

---

You are also writing the data you received more than once, could be why you are seeing the same data? Or perhaps the device is actually sending it more than once.

Code:

---

Console.WriteLine(spL.Read(dataBuffer, 0, bufSize));
            string s = System.Text.ASCIIEncoding.ASCII.GetString(dataBuffer);
            receiveBuffer.Append(s);
            var regex = new Regex(@"(\d{4} Temp=\d{2} S\d{3} N\d{3} C\d{2} )");
            Match match;
            do
            {
                match = regex.Match(receiveBuffer.ToString());
                if (match.Success)
                {
                    //"Process" the significant chunk of data
                  Console.WriteLine(match.Captures[0].Value);

---

I am not familar with regex so I can not make heads or tails from what you are doing there.

I would just look for the newline characters and replace the odd ones with a space.

I finally found a solution to my problem. The reason I had put my Event Handler in the loop was because I was under the impression that this handler had to be fired as long as the program was listening to the port (I was thinking inside the loop). I put the line above the loop and it is now showing one reading as a reading comes in so that is working perfectly. I also changed my Handler method a bit. I first used the _serialPort.Read() function to actually tell me how many bytes the whole packet contained. Once I knew that, I set the data buffer to that number. The new handler looks like this:

private static void Port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{

int i = 0;
string dtString="";
Byte [] dataBuffer = new Byte[37];


Thread.Sleep(500); // wait data ready
while (_serialPort.BytesToRead > 0) // if data ready in buffer
{
dataBuffer[i] = Convert.ToByte(_serialPort.ReadByte()); // read data serial & saving to array byte
i++; // counter buffer
}

dtString = ASCIIEncoding.ASCII.GetString(dataBuffer); // convert array byte to string


Console.Write("Data" + dtString + " " + "was received at:" + " " + DateTime.Now);
}
So what I did differently here is set the buffer to the bytes I wanted to receive. Create a while loop with a counter that counted up to my buffer's indicated byte number , read those bytes, translate to string and than display them. Thank you very much for all your help DataMiser, it was very useful and appreciated.

You're welcome :)

Glad you got it working.

Don't forget to mark your thread resolved. Use the thread tools at the top of the page to do so.