Need Advice - Opening a file once and reading data every time a method is called

[MrProsser]

Hi,
I was not sure what section to post this but I thought this was the most appropriate. I have a bit of a strange programming problem because I am forced to use a number of closed formats and tools and only have access to my own code that hooks into their system as well as some dlls that allow me to read their data files. I should also say I am fairly new to C# programming and programming in .NET so I am often not sure what tools I have at my disposal.

I have a C# project that has in it a method that I am working on. This method is called regularly (once every few seconds) by the company's system. When my method is called I need to open a data file using the company's file IO API, read some of the data, once again using their API, and then do some interpretation of this and pass back to their software some information which it then uses, and after this the process repeats itself, calling my method once again where I get new data from the file and do my processing of it. The problem I am having is that obtaining the file object occasionally fails. In my discussion with an employee at the company he suggests that I might not be able to open the file safely that often, as it is being opened every time the method is called, and suggested I try to come up with some method where the data file is opened once and kept open and during each iteration of my method I can read from this open file.

The problem I have is that I am not really sure how I can accomplish this when the only access I have is modifying this method that is constantly being called. Is it possible to somehow start a thread the first time my method is called to provide access to the file, opening it and then doling out new data each time the method is called after this?

[Brad Jones]

Moved thread. Better chance of getting an answer here.

[DataMiser]

Without info on what the api is doing it is hard to say. If you can open the file and assign a variable to the file handle which can then be resused then you could open the file in a seperate routine, save the handle in a variable and then process the file as needed with this variable. Of course this assumes that the api has seperate functions to open, read and close the file.

[BigEd781]

We can't really help you if we don't know the API that you're using. However, I assume there is some way to copy the contents of the file into memory. If the file isn't changing then just copy it and keep it in memory for subsequent access. If it is changing then you have a problem as you will eventually try to open a locked file. This sounds like a horrible design to me, and I would hate t write code that reads from disk every second if I can avoid it (and I would find a way...)

[MrProsser]

DataMiser,
I am able to open the file and get a file handle so I believe an approach like this is what I need. But sadly the only code I can modify myself is my method that is regularly called. So I would need to find a way to open the file the first time the method is called and keep it open, otherwise I get back to my current problem where I am constantly opening it.

BigEd781,
Sadly in this case the device is constantly gathering information and I am trying to get this new data from the file at each iteration. Sadly it is not possible to have the data itself directly passed to my method, I need to access it from the file. If I had my choice there would be many design changes as the software is a bit of a disaster and poorly documented but sadly I am constrained by it.

I know that it must be possible in some fashion to do this as the software itself has a mode where the data file being written to can be opened and the data displayed as it is obtained by the device. However the exact method uses is hidden from view as the code is not available. It is painful to use this API because the details are poorly documented.

[fcronin]

DataMiser,
I am able to open the file and get a file handle so I believe an approach like this is what I need. But sadly the only code I can modify myself is my method that is regularly called. So I would need to find a way to open the file the first time the method is called and keep it open, otherwise I get back to my current problem where I am constantly opening it.

Is this method you have the ability to modify in your own project? I'm not sure I understand why you would have access to modify only a single method, unless this third party API merely executes a single block of code that you provide in a text file or something every xx seconds...

If you indeed have your own application, what is being suggested I think is that you create a file reading object (StreamReader etc...) with a scope that persists outside your method, in your method you would then check to see if the file has already been opened and ready for reading, if not, open it (shared and for read) and set that object to the stream/handle... if it has, simply read from it... when your method will no longer be hit, close the file.

[MrProsser]

fcronin,

I'm not sure I understand why you would have access to modify only a single method, unless this third party API merely executes a single block of code that you provide in a text file or something every xx seconds...

This is basically what is going on. During a run of the experiment on this device there are periodic times when a decision has to be made about what data should next be looked at. Normally this is done in their own software. The maker of the software has slightly modified this so that rather than using there method to do this, I can hook into it with my method. So basically it does its stuff, calls a block of my code, then goes back to its normal operation with a modified list of data to examine.

So the problem is that I want to have something created with a scope greater the method. Let me post some code fragments. I am trying not to post much as I do not want to run afoul of NDA agreements about sharing the works. I have cut out most of the code that I do not make use of.

namespace Analyst_UserIDA
{
    [Guid("76F452FF-7A89-11d4-8A2C-00B0D023C6A0")]
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    public unsafe interface IUserIDA
    {
        [DispId(7)]
        void OnInitIDA();

        [DispId(9)]
        void OnPrepareNextScan(double* selectedMasses, double* SelectedIntensities, long* selectedCharges, int itemCount);
    }

    [Guid("5B2DBDD4-B763-428a-B48F-2E148138E7A4")]
    [ClassInterface(ClassInterfaceType.None)]
    [ProgId("Analyst_UserIDA.UserIDAObject")]
    public unsafe class UserIDAObject : IUserIDA
    {
        const long S_OK = 0;
        const long S_FALSE = 1;

        public UserIDAObject()
        {
        }

        #region IUserIDA Members

        public void OnInitIDA()
        {
            System.Diagnostics.EventLog.WriteEntry("UserIDA", "OnInitIDA");

        }

        public void OnPrepareNextScan(double* selectedMasses, double* selectedIntensities, long* selectedCharges, int itemCount)
        {

            const string FilePath = @"C:\Test.wiff";
            
            FMANFileControl FileControl = null;
            try
            {
                FileControl = new FMANFileControl();
            }
            catch
            {
                System.Diagnostics.EventLog.WriteEntry("FMAN", "problem creating FMANControl");
            }

            FMANFile wFile = new FMANFile();
            try
            {
                wFile = FileControl.GetFileObject(FilePath, 1);
            }
             catch (Exception ex)
             {
                   System.Diagnostics.EventLog.WriteEntry("FMAN", "GetFileObject " + ex.Message);
                    return;
                }
             .
             .
             .
        }

        #endregion
    }
}

This method OnPrepareNextScan is called regularly. I would like to not have to open the file in this method but in some way with a scope greater than it. Could I possible make use of a thread created upon the creation of the userIDA object?

[fcronin]

Could you do this... ?

    [Guid("5B2DBDD4-B763-428a-B48F-2E148138E7A4")]
    [ClassInterface(ClassInterfaceType.None)]
    [ProgId("Analyst_UserIDA.UserIDAObject")]
    public unsafe class UserIDAObject : IUserIDA
    {
        const long S_OK = 0;
        const long S_FALSE = 1;

        FMANFile _wFile;   

        public UserIDAObject()
        {
        }

        #region IUserIDA Members

        public void OnInitIDA()
        {
            System.Diagnostics.EventLog.WriteEntry("UserIDA", "OnInitIDA");

        }

        public void OnPrepareNextScan(double* selectedMasses, double* selectedIntensities, long* selectedCharges, int itemCount)
        {
            
            const string FilePath = @"C:\Test.wiff";
            
            if (_wFile == null)
            {
              FMANFileControl FileControl = null;
              try
              {
                 FileControl = new FMANFileControl();
               }
               catch
              {
                System.Diagnostics.EventLog.WriteEntry("FMAN", "problem creating FMANControl");
               }

              try
              {
                 _wFile = FileControl.GetFileObject(FilePath, 1);
              }
               catch (Exception ex)
               {
                   System.Diagnostics.EventLog.WriteEntry("FMAN", "GetFileObject " + ex.Message);
                    return;
               }

             }
             .
             .
             .
        }

...it may have to be cleaned up a bit depending on what your needs are further into the method and all... but shouldn't this give you a FMANFile of the scope you would need?

[MrProsser]

fcronin,
Actually I had the same thought just a little while ago. I got so stuck looking at my method (because normally I have not had to look at anything else) that I did not even think about trying that. But when I was formatting those code snippets it came to mind.

Tunnel-vision can be a very bad thing.

[fcronin]

It happens...

The next questions that would come to my mind would be...

Can other application(s) access that file while my class has it 'open' (not sure what state FileControl.GetFileObject() puts the file in...
and...
When/where will I be able to close the file.

Hope all works out for you!

[Mutant_Fruit]

You have two options:

1) Forget about your API and just use something like:

FileStream stream = new FileStream (@"C:\Test.wiff", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);

Open the file once as shown above. This will let any other application open and file for reading or writing at the same time as you have it open, assuming the other application doesn't require and try to obtain exclusive access to the file. Then just have a loop where you constantly try to read more data from it. You'll have to keep looping when you get to the end of the file (signified by reading 0 bytes from the stream).

2) Do something similar to what was already recommended.

To be honest, I think using the standard IO APIs are probably your best option If approach 1 works, use it! After all, it is just a file, there's absolutely no reason why you can't just open it like I demonstrated.

[MrProsser]

Mutant_Fruit,
Thanks, I am going to try something like #2 first, even though it might be problematic as well. The problem I have with #1 is that I have no idea what the format of the file is. The format is a proprietary one and the API was designed to allow people to make use of them (maybe not me though for I am doing something a little unusual that they likely never expected).

[Mutant_Fruit]

Actually i should probably expand on my idea What I was thinking is you can open the file in read/write sharing mode using the regular File APIs and then keep reading data from it and appending it to your own file which is 100% under your control. You can easily control when you open the file for reading and when you open it for writing, so you can 100% safely use that crazy API you've been describing as you know you'll never be reading/writing concurrently from your temp file.

[Arjay]

From the first post...

This method is called regularly (once every few seconds) by the company's system. When my method is called I need to open a data file using the company's file IO API, read some of the data, once again using their API, and then do some interpretation of this and pass back to their software some information which it then uses, and after this the process repeats itself, calling my method once again where I get new data from the file and do my processing of it. The problem I am having is that obtaining the file object occasionally fails.

Imo, you're going to need to work with the vendor to get something better than the api they're offering now.

The problem with working outside of the framework that they've provided is that you have no guarantee of file locking or when they've finished writing data, so you could get into a race condition and end up reading partially written data.

Maybe that isn't a concern, but if it is, it sounds like you need to work with them to provide a event handler that passes the updated data to you directly (in a thread safe manner).