How to know when a file is finished copying?

[Dan203]

OK I have a folder watcher type program which uses a simple timer to check for new files being added to a directory every 30 seconds. The problem is that there is no way, at least that I can find, to tell for sure if the file is ready to be processed. These are big video files, so if the user is copying them from another drive or a network share it could take several minutes from the time they show up in the directory to the time they're actually ready to be processed.

I've tried using a few tricks to determine if they're ready like opening the file in exclusive mode or checking the file size to see if it's growing, but these have been unreliable. Is there any way to tell for sure when a file has finished copying? Or are these tricks my only option?

Thanks,
Dan

[VictorN]

Obtaining Directory Change Notifications

[Igor Vartanov]

I've tried using a few tricks to determine if they're ready like opening the file in exclusive mode or checking the file size to see if it's growing, but these have been unreliable.

Okay, what's wrong with opening exclusively? Which way that may be unreliable?

Is there any way to tell for sure when a file has finished copying?

Nope, as the concept of copying file is somewhat vague. Not everybody uses native CopyFile call, you know... That's why there's no such a thing like "copying complete" notification in Windows.

Is there any way to tell for sure when a file has finished copying?

Sure, why not. The copying program should notify your app that it positively finished with the copy operation.

[Dan203]

Okay, what's wrong with opening exclusively? Which way that may be unreliable?

I just ran into an issue today where I was testing the program with this setup. I cut and paste 6 files into the watch folder. For some reason the program hung up on one of them at the point where it tries to get exclusive access. It got stuck in the loop for several minutes until finally I just killed it. Maybe it was a fluke, but I figured if I could make it happen then so could the customer.

Anyway if this is the only way to do it then so be it. I just wanted to make sure I wasn't missing some simple method built into Windows that I just didn't know about.

Thanks,
Dan

[Alex F]

Unfortunately, polling is the only way to detect when copying finished. But there are some problems with this approach. Consider the situation when file is copied, and immediately after this renamed or deleted. In this case the watching program enters endless loop. It must always test the failure reason, and continue, only if the reason is "Access denied".
Another situation: file is copied, and immediately after this opened by another program. In this case even testing for "Access denied" code doesn't prevent an endless loop.
File also may be read-only, don't try to open it with write access.
So, the answer is polling, but do this very careful.

[Igor Vartanov]

So, the summary is: try to avoid any dependence on the copy completion detection as much as possible, otherwise try to establish an explicit communication with the app which copy function you depend on.

[ahmd]

Guys, this is quite possible and Windows Explorer uses it all the time.

I just wanted to make sure I wasn't missing some simple method built into Windows that I just didn't know about.

VictorN, actually gave you a link to how you need to do it:

1. You call FindFirstChangeNotification() when your program starts. There you can specify what change you need to watch: change in the whole directory, just a single file, change in file name, in size, attributes, etc. Read MSDN from the link he gave you above.

2. Then you can use the handle returned by FindFirstChangeNotification() pretty much like an event object that will be signaled when your requested change occurs. Then you'll need to call FindNextChangeNotification() to reset it though:

//If used from a worker thread:
::WaitForSingleObject(hHandleChangeNotification, INFINITE);
FindNextChangeNotification(hHandleChangeNotification);

//If used for polling from a GUI thread:
if(::WaitForSingleObject(hHandleChangeNotification, 0) == WAIT_OBJECT_0)
    FindNextChangeNotification(hHandleChangeNotification);

3. Also read MSDN from the link above on how to see what change has actually occurred. As an example, you can use ReadDirectoryChangesW() to track changes to a folder.

4. When your app closes, remember to clean everything up by calling FindCloseChangeNotification().

[ahmd]

Oh, and to see if your file is ready (say, after a copy operation) for whatever you want to do with it, you can use a simple non-invasive file opening technique:

HANDLE hFile = CreateFile(FilePath, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);
if(hFile != INVALID_HANDLE_VALUE)
{
    //Yes, it is ready for reading
    CloseHandle(hFile);
}

You have to know at this point, what access to a file your program needs to watch. In the sample above it tracks the reading access only (i.e. if the file can be opened).

[Dan203]

Thanks guys, this is all very helpful information.

Dan

[Igor Vartanov]

actually gave you a link to how you need to do it

Nope, he did not, as the original query was about copy completion notification.

[ahmd]

Nope, he did not, as the original query was about copy completion notification.

It's not a one-to-one example but it pretty much clues the OP right into how to do it:

1. You use Directory Change Notifications to get a point when the file is added, or changed (when copying has begun).

2. Then you start polling that file with my method from post #8 until it succeeds or the file is no more there (use PathFileExists for that). Obviously this has to be done in a worker thread, or with the use of WM_TIMER message so that not to block the GUI thread. And a "graceful" handling of error situations, or app's shutdown should be also implemented. I'd poll it with a second or 500ms frequency. The exact flags for the CreateFile call should also be tailored for the OP's needs.

That will pretty much accomplish the OP's original goal. I know that it will because I have a working app that does just that.

[Igor Vartanov]

The original post was about a hundred percent proof method, while yours eventually is just a sort of same meditation on results of opening file which was already stated unreliable by the thread's originator. And listening for file system events could give you nothing more in this direction.

Besides, a little comment to your post #8: "You have to know at this point, what access to a file your program needs to watch." The only reasonable access type is exclusive access, but again it seems already tried.

[ahmd]

The original post was about a hundred percent proof method, while yours eventually is just a sort of same meditation on results of opening file which was already stated unreliable by the thread's originator. And listening for file system events could give you nothing more in this direction.

Igor, what exactly is unreliable for you here?

FindFirstChangeNotification and FindNextChangeNotification are parts of the OS kernel (and I'm obviously talking about Windows NT-based OS). They provide 100% accurate results, well unless the OS is unstable itself. So in this case, one can have a guaranteed result when a new file is added into a folder, when it's renamed, resized, deleted, its attributes change, etc.

Here we don't know what the OP's intentions are once the file is done copying. He mentioned big video files, so I assumed that his program may show a notification of a new file added to his watched folder, or maybe his program would start playing it automatically. Whatever the purpose, all of it could be recapped in one term -- when the file is ready for opening for reading. That's what my method in post 8 checks for. That will also signify when the copy operation is complete. So again, what is not reliable for you in this approach?

The only reasonable access type is exclusive access

This is not true at all. The rule of thumb is that you don't open someone else files with exclusive access.

I cut and paste 6 files into the watch folder. For some reason the program hung up on one of them at the point where it tries to get exclusive access.

A program that might be working with those files doesn't expect some other process to be opening them exclusively and may lead to it hanging up or working incorrectly. Any watch program has to be as unobtrusive as possible.

On the side note, it is also a bad practice to open your own files with exclusive access. In my book the only reason you'd open a file exclusively is if you have something to "hide" it in.

I'll give you a real-life example that I've encountered with exclusive access to a file as a user of another software. I don't know if you know what Adobe After Effect is? It's a video compositing software. When it renders video, the process might take up to several hours. In the meantime it outputs any possible warnings/error messages into a text file. For some reason the programmers that made that software decided to open such a text file with exclusive access (when simple write-exclusive would suffice). As a result I can't open that file to see what errors the rendering might have produced until the whole process is done (again it takes several hours to complete).

[Igor Vartanov]

Igor, what exactly is unreliable for you here?

Human factor. As I said, not everybody uses CopyFile function, and custom copying might appear to be a beast you never met before.

So in this case, one can have a guaranteed result when a new file is added into a folder, when it's renamed, resized, deleted, its attributes change, etc.

Neither of that is the subject. The question was about detection of copy completion, so reading change notifications appears hardly relevant, except maybe size change, which is again unreliable.

The rule of thumb is that you don't open someone else files with exclusive access.

Do you mind if I never agree with the rule?

I'll give you a real-life example

And I'll give you an imaginary example back. Led by your rule, some guy creates a program that does custom copy operation with FILE_SHARE_READ + FILE_SHARE_WRITE access. Again, led by your post #8 I create a part of my app that verifies the guy's app copy completion. Oops, I'm gonna have false positive situation.

[ahmd]

And I'll give you an imaginary example back. Led by your rule, some guy creates a program that does custom copy operation with FILE_SHARE_READ + FILE_SHARE_WRITE access. Again, led by your post #8 I create a part of my app that verifies the guy's app copy completion. Oops, I'm gonna have false positive situation.

Well, what if some guy creates a program that crashes the whole OS, my app will crash with it. I cannot imagine a real-life legitimate situation when someone may create a file with FILE_SHARE_WRITE access while performing a write operation into it. I truly can't think of a single one, can you?

But if the OP wants to make a fail-safe (or fool-proof in this case) program, he can look at the
WhoUses code sample and employ NtQuerySystemInformation() calls to try to see if a file handle is opened by any process. The downside of such approach is that the abovementioned API will most certainly require elevated privileges in Vista/Windows 7.