Copying big files over 4GBytes with the percentage of the copy display.

[Mark Agius]

I was dragging a file 5.6 GBytes from my C: drive to a second hard drive with 73 GBytes of free space.
but Windows Explorer would display the 'Not enough free disc space' error.

So I am now writing a program to copy big files.
As copying a big file will take some time, the code should be fast and display a percentage of the data copied as it copies it.

The Microsoft Scripting Runtime FileSystemObject is fast but if file is over 4GB's then no file is created (and no error message)
The FileCopy CopyThisFile, AsThisFile will copy the first 4GB's then displays a 'No disc space' error.
Using 'Open files as binary' you can display the percentage but is very very slow and so can't check if it works with files over 4GB's

Any ideas on how this could be done.

[Code 1]
  Dim fso As FileSystemObject
  Set fso = New FileSystemObject
  fso.CopyFile CopyThisFile, AsThisFile
  
  No AsThisFile created and no error message if file is over 4GB's
  My fastest method.

[Code 2]
  FileCopy CopyThisFile, AsThisFile

  Copyes the first 4GB's only (with error message)
  Second fastest method.

[Code 3]
  Dim copyFromHandel As *******, copyToHandel As Integer
  Dim MovingData As String
  copyFromHandel = FreeFile
  Open CopyThisFile For Binary Access Read Lock Read As #copyFromHandel
  copyToHandel = FreeFile
  Open AsThisFile For Binary Access Write As #copyToHandel
  Do While (Not EOF(copyFromHandel)) And CopyingData
    Get #copyFromHandel, , MovingData
    Put #copyToHandel, , MovingData
  Loop
  Close #copyToHandel
  copyToHandel = 0
  Close #copyFromHandel
  copyFromHandel = 0

  Only tested upto 212,950 KBytes due to a very slow time to copy data.
  This is a very slow method. (1 day to copy 280,000 KBytes)

[Comintern]

I think the CopyFileEx API will let you copy files bigger than 4GB (not sure as I don't have a file to test it on). Check out the sample code on AllAPI.net. You need to control the progress display (via a callback function) instead of using the API one though.

[Shuja Ali]

Also looking at SHFileOperation API would be even more helpful. This is the one that Windows uses internally to copy folders and files. You can lookup this example at http://allapi.mentalis.org/apilist/S...peration.shtml.

[WizBang]

It can help greatly to read the file in chunks. VB's Binary file operations are indeed fast, but not if you read the file in small pieces. However, if I'm not mistaken, the 4GB limitation still applies. So it may end up being fastest with API, combined with reading chucks. If so, you may be able to have a progress indicator without a callback too.

[GremlinSA]

Wizbang has it right...

Using API's and the correct data types you can open, read and write files of upto 922GB in size..

For a full explination on how to program this have a look at this article on CG ..

Theres even a module you can download and add to your project !!!!

Gremmy....

[Mark Agius]

Here are my results, using Windows XP (Hopefully newer operating systems will support bigger file values)
[Code 1]
Dim fso As FileSystemObject : fso.CopyFile "CopyThisFile", "AsThis"
= Ok upto 4 GB's
Over 4 GB's = no file created and no error message.

[Code 2]
FileCopy "CopyThisFile", "AsThisFile"
= copies the first 4 GB's then displays error message.

[Code 3]
Open "File" as binary : Get/Put#handel,data
= 2 GB's then crash (after 1 day and 19 hours)

[Code 2]
SHFileOperation API (http://allapi.mentalis.org/apilist/SHFileOperation.shtml)
Private Declare Function CopyFile Lib "kernel32" Alias "CopyFileA" (ByVal lpExistingFileName As String, ByVal lpNewFileName As String, ByVal bFailIfExists As Long) As Long
Copying file from C: to E:
Under 4 GB's = Ok
Over 4 GB's = no file created and no error displayed
Copying file to a different location on the same drive
Under 4 GB's = Ok
Over 4 GB's = Ok
(Displays file in seconds, but takes a long time to copy)

Strange that copying (yes I do mean copying and not moving) a big file on the same drive works
but not to a different drive!

[Code 5]
CreateFile Lib "kernel32" (http://www.codeguru.com/vb/controls/vb_file/directory/article.php/c12917/)
Still testing but seams to be the best and only way to copy big files between 4GB's and 922GB's
Full marks to Richard Newcombe code.


WATCH THIS SPACE IN A FEW YEARS TIME WHEN SOMEBODY WANTS TO USE FILES OVER 922 GIGA-BYTES (989,989,961,728 bytes)

[GremlinSA]

I wont say that it's the fastest or even the best method, but it works,

Using drive compresion, and loose data, i've created files up to 400GB easily, and read from it with ease.. (just to test the module)

And when you look at how i put the module together, I tried to keep the calls simple....

Thank you for the compliment Mark.......

Gremmy....

[Mark Agius]

I've clicked menu on my E: hard drive icon in the 'My Computer' and clicked on Format.
This listed 'NTFS' as the only type of filing system format allowed.
So I thought that my drive was NTFS.

I have now reformatted the drive and guess what.
I can now drag files bigger than 4 GBytes.
I think there must have been a problem with the drive or was it a FAT or FAT32 drive that told the OS that it was NTFS.

So some of the code above will now work.

CreateFile Lib "kernel32" (http://www.codeguru.com/vb/controls/...le.php/c12917/)
is my favourite code.
But I have fount two small bugs.

The program will crash when FileSize=2,147,483,647

FileSize=2147483647
Call Size2Long( 2147483647 ,LongLow,LongHigh)

So adder a line to handel any errors
and replaced
LongLow = -CLng(Cutoff - (FileSize - 1))
with
LongLow = -CLng(Cutoff - FileSize)
LongLow = LongLow - 1

Private Sub Size2Long(ByVal FileSize As Currency, ByRef LongLow As Long, ByRef LongHigh As Long)
  '&HFFFFFFFF unsigned = 4294967295
  On Error GoTo ErrorCodeTrap ' (Error code added)
  Dim Cutoff As Currency
  Cutoff = 2147483647
  Cutoff = Cutoff + 2147483647
  Cutoff = Cutoff + 1 ' now we hold the value of 4294967295 and not -1
  LongHigh = 0
  Do Until FileSize < Cutoff
    LongHigh = LongHigh + 1
    FileSize = FileSize - Cutoff
  Loop
  If FileSize > 2147483647 Then
    ' LongLow = -CLng(Cutoff - (FileSize - 1)) ' (Code edited)
    LongLow = -CLng(Cutoff - FileSize)         ' (Code edited)
    LongLow = LongLow - 1                      ' (Code edited)
  Else
    LongLow = CLng(FileSize)
  End If
  ' ErrorLog = ErrorLog + "FileSize=" + CStr(FileSize) + vbTab + " (L-Long=" + CStr(LongLow) + ", H-Long=" + CStr(LongHigh) + ") OK" + vbCrLf
Exit Sub
ErrorCodeTrap:
  ' Error code starts here.
  ' ErrorLog = ErrorLog + "FileSize=" + CStr(FileSize) + vbTab + " (L-Long=" + CStr(LongLow) + ", H-Long=" + CStr(LongHigh) + ") Error; " + Error$ + vbCrLf
Exit Sub
End Sub

Also you can't crate a new file (only open an existing one) with
Sub API_OpenFile(...)

so added code;

Public Function API_CreateNewFile(ByVal FileName As String) As Long
  ' This function will create a new blank/empty file.
  API_CreateNewFile = -2
  On Error GoTo ErrorCodeTrap
  ' Open/create a new file.
  API_CreateNewFile = CreateFile(FileName, GENERIC_WRITE, FILE_SHARE_READ Or FILE_SHARE_WRITE, 0&, CREATE_NEW, 0, 0)
  ' Close the newley opened file. (File handel is API_CreateNewFile)
  API_CreateNewFile = CloseHandle(API_CreateNewFile)
  API_CreateNewFile = -1
Exit Sub
On Error GoTo ErrorCodeTrap:
Exit Function

' If API_CreateNewFile(..)=-2 then error. Eg. Path not found. Old locked file found etc.
' If API_CreateNewFile(..)=-1 then file created and closed
' If API_CreateNewFile(..)>0  then file is still open. File handel is..

[GremlinSA]

The module is complete and only the 'CreationDisposition' need to be adjusted accordingly..

I did set it to 'Open_Allways' but there are other vaules that can be used..

Const CREATE_NEW As Short = 1
Const CREATE_ALWAYS As Short = 2
Const OPEN_EXISTING As Short = 3
Const OPEN_ALLWAYS As Short = 4

Gremmy...