How to know when a file is finished copying?

OK I have a folder watcher type program which uses a simple timer to check for new files being added to a directory every 30 seconds. The problem is that there is no way, at least that I can find, to tell for sure if the file is ready to be processed. These are big video files, so if the user is copying them from another drive or a network share it could take several minutes from the time they show up in the directory to the time they're actually ready to be processed.

I've tried using a few tricks to determine if they're ready like opening the file in exclusive mode or checking the file size to see if it's growing, but these have been unreliable. Is there any way to tell for sure when a file has finished copying? Or are these tricks my only option?

Thanks,
Dan

Obtaining Directory Change Notifications

Quote:

---

I've tried using a few tricks to determine if they're ready like opening the file in exclusive mode or checking the file size to see if it's growing, but these have been unreliable.

---

Okay, what's wrong with opening exclusively? Which way that may be unreliable?

Quote:

---

Is there any way to tell for sure when a file has finished copying?

---

Nope, as the concept of copying file is somewhat vague. Not everybody uses native CopyFile call, you know... That's why there's no such a thing like "copying complete" notification in Windows. :)

Quote:

---

Is there any way to tell for sure when a file has finished copying?

---

Sure, why not. The copying program should notify your app that it positively finished with the copy operation. :D

Quote:

---

Originally Posted by Igor Vartanov

Okay, what's wrong with opening exclusively? Which way that may be unreliable?

---

I just ran into an issue today where I was testing the program with this setup. I cut and paste 6 files into the watch folder. For some reason the program hung up on one of them at the point where it tries to get exclusive access. It got stuck in the loop for several minutes until finally I just killed it. Maybe it was a fluke, but I figured if I could make it happen then so could the customer.

Anyway if this is the only way to do it then so be it. I just wanted to make sure I wasn't missing some simple method built into Windows that I just didn't know about.

Thanks,
Dan

Unfortunately, polling is the only way to detect when copying finished. But there are some problems with this approach. Consider the situation when file is copied, and immediately after this renamed or deleted. In this case the watching program enters endless loop. It must always test the failure reason, and continue, only if the reason is "Access denied".
Another situation: file is copied, and immediately after this opened by another program. In this case even testing for "Access denied" code doesn't prevent an endless loop.
File also may be read-only, don't try to open it with write access.
So, the answer is polling, but do this very careful.

So, the summary is: try to avoid any dependence on the copy completion detection as much as possible, otherwise try to establish an explicit communication with the app which copy function you depend on. :)

Guys, this is quite possible and Windows Explorer uses it all the time.

Quote:

---

Originally Posted by Dan203

I just wanted to make sure I wasn't missing some simple method built into Windows that I just didn't know about.

---

VictorN, actually gave you a link to how you need to do it:

1. You call FindFirstChangeNotification() when your program starts. There you can specify what change you need to watch: change in the whole directory, just a single file, change in file name, in size, attributes, etc. Read MSDN from the link he gave you above.

2. Then you can use the handle returned by FindFirstChangeNotification() pretty much like an event object that will be signaled when your requested change occurs. Then you'll need to call FindNextChangeNotification() to reset it though:

Code:

---

//If used from a worker thread:
::WaitForSingleObject(hHandleChangeNotification, INFINITE);
FindNextChangeNotification(hHandleChangeNotification);

---

Code:

---

//If used for polling from a GUI thread:
if(::WaitForSingleObject(hHandleChangeNotification, 0) == WAIT_OBJECT_0)
    FindNextChangeNotification(hHandleChangeNotification);

---

3. Also read MSDN from the link above on how to see what change has actually occurred. As an example, you can use ReadDirectoryChangesW() to track changes to a folder.

4. When your app closes, remember to clean everything up by calling FindCloseChangeNotification().

Oh, and to see if your file is ready (say, after a copy operation) for whatever you want to do with it, you can use a simple non-invasive file opening technique:

Code:

---

HANDLE hFile = CreateFile(FilePath, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);
if(hFile != INVALID_HANDLE_VALUE)
{
    //Yes, it is ready for reading
    CloseHandle(hFile);
}

---

You have to know at this point, what access to a file your program needs to watch. In the sample above it tracks the reading access only (i.e. if the file can be opened).

Thanks guys, this is all very helpful information.

Dan

Quote:

---

Originally Posted by ahmd

actually gave you a link to how you need to do it

---

Nope, he did not, as the original query was about copy completion notification. :D

Quote:

---

Originally Posted by Igor Vartanov

Nope, he did not, as the original query was about copy completion notification. :D

---

It's not a one-to-one example but it pretty much clues the OP right into how to do it:

1. You use Directory Change Notifications to get a point when the file is added, or changed (when copying has begun).

2. Then you start polling that file with my method from post #8 until it succeeds or the file is no more there (use PathFileExists for that). Obviously this has to be done in a worker thread, or with the use of WM_TIMER message so that not to block the GUI thread. And a "graceful" handling of error situations, or app's shutdown should be also implemented. I'd poll it with a second or 500ms frequency. The exact flags for the CreateFile call should also be tailored for the OP's needs.

That will pretty much accomplish the OP's original goal. I know that it will because I have a working app that does just that.

The original post was about a hundred percent proof method, while yours eventually is just a sort of same meditation on results of opening file which was already stated unreliable by the thread's originator. And listening for file system events could give you nothing more in this direction.

Besides, a little comment to your post #8: "You have to know at this point, what access to a file your program needs to watch." The only reasonable access type is exclusive access, but again it seems already tried. :)

Quote:

---

Originally Posted by Igor Vartanov

The original post was about a hundred percent proof method, while yours eventually is just a sort of same meditation on results of opening file which was already stated unreliable by the thread's originator. And listening for file system events could give you nothing more in this direction.

---

Igor, what exactly is unreliable for you here?

FindFirstChangeNotification and FindNextChangeNotification are parts of the OS kernel (and I'm obviously talking about Windows NT-based OS). They provide 100% accurate results, well unless the OS is unstable itself. So in this case, one can have a guaranteed result when a new file is added into a folder, when it's renamed, resized, deleted, its attributes change, etc.

Here we don't know what the OP's intentions are once the file is done copying. He mentioned big video files, so I assumed that his program may show a notification of a new file added to his watched folder, or maybe his program would start playing it automatically. Whatever the purpose, all of it could be recapped in one term -- when the file is ready for opening for reading. That's what my method in post 8 checks for. That will also signify when the copy operation is complete. So again, what is not reliable for you in this approach?

Quote:

---

Originally Posted by Igor Vartanov

The only reasonable access type is exclusive access

---

This is not true at all. The rule of thumb is that you don't open someone else files with exclusive access.

Quote:

---

Originally Posted by Dan203

I cut and paste 6 files into the watch folder. For some reason the program hung up on one of them at the point where it tries to get exclusive access.

---

A program that might be working with those files doesn't expect some other process to be opening them exclusively and may lead to it hanging up or working incorrectly. Any watch program has to be as unobtrusive as possible.

On the side note, it is also a bad practice to open your own files with exclusive access. In my book the only reason you'd open a file exclusively is if you have something to "hide" it in.

I'll give you a real-life example that I've encountered with exclusive access to a file as a user of another software. I don't know if you know what Adobe After Effect is? It's a video compositing software. When it renders video, the process might take up to several hours. In the meantime it outputs any possible warnings/error messages into a text file. For some reason the programmers that made that software decided to open such a text file with exclusive access (when simple write-exclusive would suffice). As a result I can't open that file to see what errors the rendering might have produced until the whole process is done (again it takes several hours to complete).

Quote:

---

Igor, what exactly is unreliable for you here?

---

Human factor. As I said, not everybody uses CopyFile function, and custom copying might appear to be a beast you never met before.

Quote:

---

So in this case, one can have a guaranteed result when a new file is added into a folder, when it's renamed, resized, deleted, its attributes change, etc.

---

Neither of that is the subject. The question was about detection of copy completion, so reading change notifications appears hardly relevant, except maybe size change, which is again unreliable.

Quote:

---

The rule of thumb is that you don't open someone else files with exclusive access.

---

Do you mind if I never agree with the rule? :)

Quote:

---

I'll give you a real-life example

---

And I'll give you an imaginary example back. Led by your rule, some guy creates a program that does custom copy operation with FILE_SHARE_READ + FILE_SHARE_WRITE access. Again, led by your post #8 I create a part of my app that verifies the guy's app copy completion. Oops, I'm gonna have false positive situation.

Quote:

---

Originally Posted by Igor Vartanov

And I'll give you an imaginary example back. Led by your rule, some guy creates a program that does custom copy operation with FILE_SHARE_READ + FILE_SHARE_WRITE access. Again, led by your post #8 I create a part of my app that verifies the guy's app copy completion. Oops, I'm gonna have false positive situation.

---

Well, what if some guy creates a program that crashes the whole OS, my app will crash with it. I cannot imagine a real-life legitimate situation when someone may create a file with FILE_SHARE_WRITE access while performing a write operation into it. I truly can't think of a single one, can you?

But if the OP wants to make a fail-safe (or fool-proof in this case) program, he can look at the
WhoUses code sample and employ NtQuerySystemInformation() calls to try to see if a file handle is opened by any process. The downside of such approach is that the abovementioned API will most certainly require elevated privileges in Vista/Windows 7.

Quote:

---

I cannot imagine a real-life legitimate situation when someone may create a file with FILE_SHARE_WRITE access while performing a write operation into it. I truly can't think of a single one, can you?

---

Seems we're digressed alot, but yes, I can imagine that several apps write to the same file "simultaneously", or better say, opened the same file for writing. This is only the matter of design, same as use exclusive access or not. The fact itself that such "shared write" access flag exists (and not only in Windows :)) tells me that I'm not the only person who can imgine such situation.

Copy two files, instead of only one.

When the second file arrives, you can presume the first file is finished copying, because files are copied sequentially, as far as I know on the systems I have used.

You could also use a third file that would be sent first, and which would contain the size of your main file.

Quote:

---

Originally Posted by olivthill2

... because files are copied sequentially, as far as I know on the systems I have used.

---

Why "sequentially"? :confused:
What prevents OS to make a "parallel" copying? :confused:

Quote:

---

Originally Posted by olivthill2

Copy two files, instead of only one.

When the second file arrives, you can presume the first file is finished copying, because files are copied sequentially, as far as I know on the systems I have used.

You could also use a third file that would be sent first, and which would contain the size of your main file.

---

If the copying program copies one file, waits for it to complete then starts a 2nd file... Then sure.

If you mark 2 files from explorer and make it copy them elsewhere. The order they start copying isn't defined, even the order they finish isn't defined, not even to be identical to the starting order. It may SEEM to be that way for you in the few tests you done, but it isn't guarantee to work that way. We had to explicitely move away from using SHFileOperation() in one project and write our own because we couldn't rely on internal ordering. If you have reparsepoints in your directory structure, you can get some really weird results.

Quote:

---

Originally Posted by ahmd

I cannot imagine a real-life legitimate situation when someone may create a file with FILE_SHARE_WRITE access while performing a write operation into it. I truly can't think of a single one, can you?

---

Shared access databases ? I'm using them quite frequently. There is internal organisation to keep 2 treads/processes from putting the database or the indexes in a corrupt state ofc, that's why you still have file range locks to arrange something like that.

Even without this. Windows itself is using this in the pagefile.

Or if your program has a shared data segment (or shared filemapping which is basically the same).

I have even seen video editing software that allows multiple authors to modify parts 'on the fly' while other users are watching.

Quote:

---

Originally Posted by Igor Vartanov

Seems we're digressed alot, but yes, I can imagine that several apps write to the same file "simultaneously", or better say, opened the same file for writing. .. The fact itself that such "shared write" access flag exists (and not only in Windows ) tells me that I'm not the only person who can imgine such situation

---

OK you have a point. So, I guess, if they keep persistent handles to that file ... but that is clearly asking for trouble... Well, in that case one needs to use NT sys core APIs to determine if a file handle is still in use (my previous post). Again, we don't know the exact use the OP was intending for this. I just hope he doesn't use it to monitor database files.

Quote:

---

Originally Posted by OReubens

Shared access databases ? I'm using them quite frequently. There is internal organisation to keep 2 treads/processes from putting the database or the indexes in a corrupt state ofc, that's why you still have file range locks to arrange something like that.

Even without this. Windows itself is using this in the pagefile.

Or if your program has a shared data segment (or shared filemapping which is basically the same).

I have even seen video editing software that allows multiple authors to modify parts 'on the fly' while other users are watching.

---

And you're sure that all that software opens files you mentioned with a FILE_SHARE_WRITE access while writing into it at the same time??? Please note that I'm not talking about FILE_SHARE_READ access mode, at which one can open a file being written to.

Quote:

---

Originally Posted by ahmd

And you're sure that all that software opens files you mentioned with a FILE_SHARE_WRITE access while writing into it at the same time??? Please note that I'm not talking about FILE_SHARE_READ access mode, at which one can open a file being written to.

---

No. I'm actually sure that NOT all software opens files with share deny write and then writes to it.
But you also can't say it has no use at all.

Most database engines these days are services running on the machine where the database is stored, and all other processses that want to access the database send requests to that service (via a pipe or socket usually), so in this case, most databases will be exclusive access (by the service), but allow simultaneous access to the database via some API of the DB service. This is usually a more stable, reliable and more performant approach. But it does have the disadvantage that you need to install a database manager on the server, and that can be a complicated issue for small networks. Or for networks without a dedicated server.

But the alternative is still possible. You can open a file and have 2 applications (or more) write to it at the same time, you'll need code to make sure your database and the indexes stay valid, but it can be done. Before high bandwidth networks (and dedicated servers) and fast CPU's became common place, this approach was quite common for ISAM based databases. With the move towards RDBMS databases, dedicated servers, faster networks, it's a bit of a dinosaur.

Quote:

---

with a FILE_SHARE_WRITE access while writing into it at the same time???

---

Just a little note: simultaneous opening file for writing is not equal to or stipulates simultaneous writing.