-
Reading chunks from large files
Hi, the code below works fine with files smaller than 2GB, but it also needs to be abe to handle larger files. I need to change the code to API but I have never really worked with those API before, only to get the size of files bigger than 2GB.
EOF doesn't work and I think I also need to use SetFilePointer, but it turns into a mess.
Can anyone please have a look at the code and tell what to change exactly? Thank you.
Original code:
Code:
Public Function HashFile(ByVal strFilename As String)
Dim fFile As String, sTheString As String, sTheHash As String, sFinalHash As String
Dim FF As Long, cCount As Long, cFile As Long, i As Integer
Dim b() As Byte
Const CHUNK_SIZE As Long = 9728000
Dim cCrypto As CryptKci.clsCryptoAPI
Set cCrypto = New CryptKci.clsCryptoAPI
sTheHash = ""
cFile = FileLen(strFilename)
FF = FreeFile()
ReDim b(0 To CHUNK_SIZE - 1)
Open strFilename For Binary Access Read Lock Write As #FF
Do While Not EOF(FF)
i = i + 1
If cFile < CHUNK_SIZE Then
ReDim b(0 To cFile - 1)
Get #FF, , b()
sTheString = StrConv(b(), vbUnicode)
sTheHash = sTheHash & cCrypto.CreateHash(sTheString, 2)
frmMain.txtHashSet.Text = frmMain.txtHashSet.Text & i & " - " & cCrypto.CreateHash(sTheString, 2) & " - " & cFile & vbCrLf
Exit Do
Else
Get #FF, , b()
sTheString = StrConv(b(), vbUnicode)
sTheHash = sTheHash & cCrypto.CreateHash(sTheString, 2)
frmMain.txtHashSet.Text = frmMain.txtHashSet.Text & i & " - " & cCrypto.CreateHash(sTheString, 2) & " - " & "9728000" & vbCrLf
End If
cFile = cFile - CHUNK_SIZE
DoEvents
Loop
Close #FF
sFinalHash = cCrypto.ConvertStringFromHex(sTheHash)
frmMain.txtHash.Text = cCrypto.CreateHash(sFinalHash, 2)
Set cCrypto = Nothing
End Function
New code:
Code:
Const FILE_BEGIN = 0
Const FILE_SHARE_READ = &H1
Const FILE_SHARE_WRITE = &H2
Const OPEN_EXISTING = 3
Const GENERIC_READ = &H80000000
Private Declare Function ReadFile Lib "kernel32" (ByVal hFile As Long, lpBuffer As Any, ByVal nNumberOfBytesToRead As Long, lpNumberOfBytesRead As Long, ByVal lpOverlapped As Any) As Long
Private Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, ByVal lpSecurityAttributes As Any, ByVal dwCreationDisposition As Long, ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
Private Declare Function SetFilePointer Lib "kernel32" (ByVal hFile As Long, ByVal lDistanceToMove As Long, lpDistanceToMoveHigh As Long, ByVal dwMoveMethod As Long) As Long
Private Declare Function GetFileSize Lib "kernel32" (ByVal hFile As Long, lpFileSizeHigh As Long) As Long
Public Function HashFile(ByVal strFilename As String)
Dim sTheString As String, sTheHash As String, sFinalHash As String
Dim i As Integer, b() As Byte
Dim hFile As Long, FileLenght As Long, Result As Long
Dim cCrypto As CryptKci.clsCryptoAPI
Set cCrypto = New CryptKci.clsCryptoAPI
Const CHUNK_SIZE As Long = 9728000
hFile = CreateFile(strFilename, GENERIC_READ, FILE_SHARE_READ Or FILE_SHARE_WRITE, ByVal 0&, OPEN_EXISTING, 0, 0)
sTheHash = ""
FileLenght = GetFileSize(hFile, 0)
SetFilePointer hFile, 0, 0, FILE_BEGIN
ReDim b(0 To CHUNK_SIZE - 1)
Do While Not EOF(hFile)
i = i + 1
If FileLenght < CHUNK_SIZE Then
ReDim b(0 To FileLenght - 1)
ReadFile hFile, b(1), UBound(b), Result, ByVal 0&
sTheString = StrConv(b(), vbUnicode)
sTheHash = sTheHash & cCrypto.CreateHash(sTheString, 2)
frmMain.txtHashSet.Text = frmMain.txtHashSet.Text & i & " - " & cCrypto.CreateHash(sTheString, 2) & " - " & FileLenght & vbCrLf
Exit Do
Else
ReadFile hFile, b(1), UBound(b), Result, ByVal 0&
sTheString = StrConv(b(), vbUnicode)
sTheHash = sTheHash & cCrypto.CreateHash(sTheString, 2)
frmMain.txtHashSet.Text = frmMain.txtHashSet.Text & i & " - " & cCrypto.CreateHash(sTheString, 2) & " - " & "9728000" & vbCrLf
End If
FileLenght = FileLenght - CHUNK_SIZE
DoEvents
Loop
sFinalHash = cCrypto.ConvertStringFromHex(sTheHash)
frmMain.txtHash.Text = cCrypto.CreateHash(sFinalHash, 2)
CloseHandle hFile
Set cCrypto = Nothing
End Function
-
Re: Reading chunks from large files
I notice that in your original code you are using the 'old' method of opening and reading a file.
I'm not totally sure, but it may be worth making a reference to the 'Microsoft Scripting Runtime' and using the methods in there to try and read your file.
-
Re: Reading chunks from large files
Hmm i got your problem in the bag.. ;)..
to use API's to open, read and write files is not that difficult at all.
Attached is a API Module you can include into your project and do calls very simular to OPEN, Read, Write, and they will be doen via Windows API's.. Added is a extra funtion allowing you to cut files shorter (Set a new end of file) Something that VB6 commands lack..
for a working example of this api, follow the Hex-Editor link in my signature..
Hope this Helps...
Gremmy....
----- EDIT ------
Updated API Module on later post...
-
Re: Reading chunks from large files
-
Re: Reading chunks from large files
Thanks for the suggestions. I'm trying GremlinSA's API module first and it makes it easier, but I'm still having some problems with it.
It returns the wrong size of files bigger than 4GB. My test file is 4.617.568.256 bytes (4.29GB), but API_OpenFile returns a file size of 322.597.374 bytes (File_Len in the code below).
Another problem I have is how to loop until the end of the file, because I can't use EOF. The final problem is how to read chunks of 9.728.000 bytes.
I need to read 9.728.000 bytes from the file, hash it, read the next 9.728.000 bytes, hash it, until the end of the file. If the last chunk is smaller than 9.728.000 bytes, then the Chunk_Size needs to be changed to the size of the last chunk or else the hash code will be wrong.
Code:
Option Explicit
Public File_Len As Long
Public File_Num As Long
Public Function HashFile(ByVal strFilename As String)
Dim Seek_Len As Long
Dim sString As String, sHash As String
Dim b() As Byte
Const Chunk_Size As Long = 9728000
API_OpenFile strFilename, File_Num, File_Len
Seek_Len = File_Len
'do until what?
If Seek_Len < Chunk_Size Then
API_ReadFile File_Num, Seek_Len, File_Len - Seek_Len, b()
sString = StrConv(b(), vbUnicode)
'sHash = sHash & cCrypto.CreateHash(sString, 2)
'frmMain.txtHashSet.Text = frmMain.txtHashSet.Text & i & " - " & cCrypto.CreateHash(sTheString, 2) & " - " & cFile & vbCrLf
'Exit Do
Else
API_ReadFile File_Num, Seek_Len, Chunk_Size, b()
sString = StrConv(b(), vbUnicode)
'sHash = sHash & cCrypto.CreateHash(sString, 2)
'frmMain.txtHashSet.Text = frmMain.txtHashSet.Text & i & " - " & cCrypto.CreateHash(sTheString, 2) & " - " & "9728000" & vbCrLf
End If
Seek_Len = Seek_Len - Chunk_Size
'Loop
End Function
-
Re: Reading chunks from large files
Okay My bad on this one...
I originaly wrote the module without using the High values in the API's mostly in these two
Code:
Private Declare Function SetFilePointer Lib "kernel32" (ByVal hFile As Long, ByVal lDistanceToMove As Long, lpDistanceToMoveHigh As Long, ByVal dwMoveMethod As Long) As Long
Private Declare Function GetFileSize Lib "kernel32" (ByVal hFile As Long, lpFileSizeHigh As Long) As Long
Private
so we just need to update them to use these values.
The filesize sub could be updated as follows..
Code:
Public Sub API_FileSize(ByVal FileNumber As Long, ByRef FileSize As Currency)
Dim FileSizeL As Long
Dim FileSizeH As Long
FileSizeH = 0
FileSizeL = GetFileSize(FileNumber, FileSizeH)
FileSize = FileSizeL + (FileSizeH * &H10000 * &H10000) ' &H100000000 is not valid in the VB IDE so we just split it..
End Sub
and the readfile sub can be updated as follows
Code:
Public Sub API_ReadFile(ByVal FileNumber As Long, ByVal Position As currency, ByRef BlockSize As Long, ByRef Data() As Byte)
Dim PosL As Long
Dim PosH As Long
Dim SizeRead As Long
Dim Ret As Long
PosL = Position And &HFFFFFFFF
PosH = (Position / &H10000 / &H10000) And &HFFFFFFFF
Ret = SetFilePointer(FileNumber, PosL, PosH, FILE_BEGIN)
Ret = ReadFile(FileNumber, Data(0), BlockSize, SizeRead, 0&)
BlockSize = SizeRead
End Sub
the writefile sub will need the same update.
this should give you an effective 922,337,203,685,477 Max filesize.
It's a bit late now, so i will test it better in the morning, and repost a corrected API Module...
Gremmy..
-
Re: Reading chunks from large files
Thanks. I tried the changes so far, but I get a "ByRef argument type mismatch" error on the line (API_FileSize FileH, FileSize) in the API_OpenFile sub.
-
Re: Reading chunks from large files
Did you remember to change the Def-types in the calling subs to "Curency" - Filesize and position are now Currency types
Gremmy...
-
1 Attachment(s)
Re: Reading chunks from large files
I've changed them and the error message is gone, but now it returns 0 for the 4.29GB test file. With smaller files below 2GB it returns the correct filesize.
I removed API_SetEOF and API_WriteFile, because I don't need them.
-
Re: Reading chunks from large files
VB has a very serious problem working with any number larger than a long, we looking at some real workarounds to get past the 4G limit..
Given enough time i'm sure that we could work out something...
theres even a module that our uncle Jonny wrote to handle this sort of poblem, i'm going to look into incorperating it into this module to get it to work for you..
Gremmy
-
Re: Reading chunks from large files
Quote:
Originally Posted by Cimperiali
I just looked at this very closely...
It's essentially the same code as in my Module, with a little difference. Your still left with the task of splitting the file position variable into 2 - 32 bit words, and this is where most of us get stuck, considering that VB does not have a Unsigned 4 byte number Type....
As soon as a file passes the 2gig limit(2147483647 bytes) pasing the hex value to a Long would return a negative value (-2147483648), And this is the limitation that we have to work around...
Gremmy...
-
Re: Reading chunks from large files
Well i think ive done it, And the wife is a little peeved that i'v sitting infront of the PC at 2:00 in the morning, but anyways here it is...
I've added two small subs to convert a Currency value into two Long's and back, that will allow you to work with large files.. (tested in 1gig steps right up to 15 gig)...
Enjoy..
Please just let us know if it works 2 your satisfaction...
Gremmy....
--- Edit ----
New attachment on next post..
-
Re: Reading chunks from large files
Yes, the filesize works fine now. Thank you very much for your help :)
The problem I have now is that I get an overflow error when I use the code below as a test and reach the 2GB.
I get the error on the line: "PosL = Position And &HFFFFFFFF" in the API_ReadFile sub. I tried to change some things myself, but I keep getting the overflow error.
Code:
Private b(9727999) As Byte
Private Sub Command1_Click()
Dim sString As String, cCount As Double, i As Long
Do Until i = 470
i = i + 1
API_ReadFile File_Num, cCount, 9728000, b()
cCount = CDbl(cCount) + CDbl(9728000)
DoEvents
Loop
End Sub
-
1 Attachment(s)
Re: Reading chunks from large files
Oops i updated the write API, but not the Read API...
Fixed it...
Gremmy....
-
Re: Reading chunks from large files
Excellent. Thank you :)
Is the test code in my previous post the fastest way to read a file in 9728000 byte chunks or do you maybe have any tips to speed it up?
-
Re: Reading chunks from large files
This method is just about the fastest. Using the API is noticebly faster that the Get #1, or Input #1.
using the 'Microsoft Scripting Runtime' is also a bit faster but i'm not too sure how it compares to the API..
But you may have to tweek it a little to read the full file..
Gremmy
-
Re: Reading chunks from large files
Ok, thanks a lot for all your help :)
-
Re: Reading chunks from large files
Quote:
Originally Posted by GremlinSA
This method is just about the fastest. Using the API is noticebly faster that the Get #1, or Input #1.
using the 'Microsoft Scripting Runtime' is also a bit faster but i'm not too sure how it compares to the API..
But you may have to tweek it a little to read the full file..
Gremmy
Using ONLY API Calls is the fastest. MSR only calls these functions, built into Windows, so it takes a bit longer to process the instruction.
-
Re: Reading chunks from large files
GremlinSA, I'm working on a program that copies 40GB backup files from one server to another on my network. The APIs look like they would get the job done, but I must admit that it is a bit above my skill level.
The standard VB6 file copy works, but gives no update to the user as to the status of the copy. It just appears to lock up the program until the operation is complete (approx. 70 minutes per file).
Could you show me how to use the API to copy a file in chunks?
-
Re: Reading chunks from large files
Check out Powershell 2.0 (it's FREE)
It should be in every network admin's toolbox. You can INVOKE a command on a CLIENT, or run a background task. (anything that the Framework can do)
It's built into most of the new Server Operating Systems and Programs
-
Re: Reading chunks from large files
Quote:
Originally Posted by
tasrock
GremlinSA, I'm working on a program that copies 40GB backup files from one server to another on my network. The APIs look like they would get the job done, but I must admit that it is a bit above my skill level.
The standard VB6 file copy works, but gives no update to the user as to the status of the copy. It just appears to lock up the program until the operation is complete (approx. 70 minutes per file).
Could you show me how to use the API to copy a file in chunks?
After the work done in this thread i did write a article covering the basic's on using the API's, and how the module works...
Breaking the 2gig file barrier is the article, and it also has details on how to use it in several other languages...
That should get you going nicely...
-
Re: Reading chunks from large files
GremlinSA,
I've looked over both the OPers code, your code, and your tutorial. But I'm still significantly confused on how to copy larger then 2 gig files.
I want to be able to simply do something similiar to:
CopyHugeFile(strSource, strDestination)
Then within the CopyHugeFile subroutine I would make it update a progress bar. I've created several different sets of code and can't seem to understand how to use your module properly.
Please help,
Knightofoldcode.
-
Re: Reading chunks from large files
You don't have to use his module. You can create a breakpoint on a line, and run the app up to that point. Then, pressing F8, you can execute a line at a time, and hold the mouse over a variable to see it's temp value.
(You can also use print Len(mystring) in the Debug Windows at any point that it's stopped.)
Doesn't quite work that easily in VS2008 (64bit) but that's a different story.
-
Re: Reading chunks from large files
Quote:
Originally Posted by
dglienna
You don't have to use his module. You can create a breakpoint on a line, and run the app up to that point. Then, pressing F8, you can execute a line at a time, and hold the mouse over a variable to see it's temp value.
(You can also use print Len(mystring) in the Debug Windows at any point that it's stopped.)
Doesn't quite work that easily in VS2008 (64bit) but that's a different story.
I appreciate your help, but I don't understand how the debug tools are going to help me understand what/how to use the module properly to copy files over 2 gb size.
I need to use his module in order to copy the files that are in excess of 2 gb.
Thanks,
Knight.
-
Re: Reading chunks from large files
You step thru the code, see that it picks up one chunk, adds to a counter, and then sends it along, until the counter is finished. If you see how he passes the file to his module, you'll know how to substitute your own.
He'll be here later.
-
Re: Reading chunks from large files
Quote:
Originally Posted by
dglienna
You step thru the code, see that it picks up one chunk, adds to a counter, and then sends it along, until the counter is finished. If you see how he passes the file to his module, you'll know how to substitute your own.
He'll be here later.
Okay, yeah, I see what your saying, but I don't see any working examples... though, it wouldn't surprise me if it's staring me in the eye. ;)
Knight.
-
Re: Reading chunks from large files
Well, there is an attachment of a .bas module at one of Gremmy's recent posts.
But I should like to point out, that this thread here is 3 years old, and not so recent at all. :)
-
Re: Reading chunks from large files
Quote:
Originally Posted by
WoF
Well, there is an attachment of a .bas module at one of Gremmy's recent posts.
But I should like to point out, that this thread here is 3 years old, and not so recent at all. :)
There is an attachment of the module, however that module only has some sub routines... this doesn't show me any examples of how to use the subroutines, and I'm seriously confused. And yes, I have read the tutorial presented by GremlinSA.
Duly noted that the original thread is 3 years old, however, the most recent post was March 5th, 2009, 05:42 AM, by GremlinSA.
I'm just trying to get more information on how to use the subroutines to copy a file. I'm seriously confused on how to do this.
Knight.
-
Re: Reading chunks from large files
Put a breakpoint at the START of the subroutine, and execution will stop THERE when you run the app. You can see what he passes (and the values)
Of course, things might have changed since then. It may NOT work anymore...
VB6 has been known to do that. Look up my old CAPICOM posts. Used to be the greatest...
-
Re: Reading chunks from large files
Quote:
Originally Posted by
dglienna
Put a breakpoint at the START of the subroutine, and execution will stop THERE when you run the app. You can see what he passes (and the values)
Of course, things might have changed since then. It may NOT work anymore...
VB6 has been known to do that. Look up my old CAPICOM posts. Used to be the greatest...
Alright... I have been trying easily for 2-3 days trying to get this right. I can not copy a file larger then 2 gb!
I have went back over the tutorial located at http://www.codeguru.com/vb/controls/...le.php/c12917/ I have tried to download GremmlinSA's demonstration program, Hex Editor, however with no modifications it errors (ByRef Argument type Mismatch) on the following line:
Code:
API_OpenFile Tmp_File, File_Num, File_Len
And highlights the "File_Len" portion of the above line. This is in the File -> Open menu subroutine.
If I can't see any code/examples, I'm left blindly trying... to that end, here is what I have, taken from the OPers post and modified to add copying the file:
Code:
Private Declare Function WriteFile Lib "kernel32" (ByVal hFile As Long, lpBuffer As Any, ByVal nNumberOfBytesToWrite As Long, lpNumberOfBytesWritten As Long, ByVal lpOverlapped As Any) As Long
Private Declare Function ReadFile Lib "kernel32" (ByVal hFile As Long, lpBuffer As Any, ByVal nNumberOfBytesToRead As Long, lpNumberOfBytesRead As Long, ByVal lpOverlapped As Any) As Long
Private Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, ByVal lpSecurityAttributes As Any, ByVal dwCreationDisposition As Long, ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
Private Declare Function SetFilePointer Lib "kernel32" (ByVal hFile As Long, ByVal lDistanceToMove As Long, lpDistanceToMoveHigh As Long, ByVal dwMoveMethod As Long) As Long
Private Declare Function GetFileSize Lib "kernel32" (ByVal hFile As Long, lpFileSizeHigh As Long) As Long
Private Declare Function SetEndOfFile Lib "kernel32" (ByVal hFile As Long) As Long
Private Function CopyHugeFile(strSource As String, strDestination As String)
Const FILE_BEGIN = 0
Const FILE_SHARE_READ = &H1
Const FILE_SHARE_WRITE = &H2
Const OPEN_EXISTING = 3
Const GENERIC_READ = &H80000000
Dim i As Integer
Dim b() As Byte
Dim hFileSource As Long
Dim hFileDestination As Long
Dim FileLenght As Long
Const CHUNK_SIZE As Long = 9728000
hFileSource = CreateFile(strSource, GENERIC_READ, FILE_SHARE_READ Or FILE_SHARE_WRITE, ByVal 0&, OPEN_EXISTING, 0, 0)
hFileDestination = CreateFile(strDestination, GENERIC_READ, FILE_SHARE_READ Or FILE_SHARE_WRITE, ByVal 0&, OPEN_EXISTING, 0, 0)
FileLenght = GetFileSize(hFileSource, 0)
ProgressBar1.Max = FileLenght
SetFilePointer hFileSource, 0, 0, FILE_BEGIN
SetFilePointer hFileDestination, 0, 0, FILE_BEGIN
ReDim b(0 To CHUNK_SIZE - 1)
Do While Not EOF(hFileSource)
i = i + 1
If FileLenght < CHUNK_SIZE Then
ReDim b(0 To FileLenght - 1)
ReadFile hFileSource, b(1), UBound(b), Result, ByVal 0&
WriteFile hFileDestination, b(1), UBound(b), Result, ByVal 0&
Exit Do
Else
ReadFile hFileSource, b(1), UBound(b), Result, ByVal 0&
WriteFile hFileDestination, b(1), UBound(b), Result, ByVal 0&
End If
FileLenght = FileLenght - CHUNK_SIZE
ProgressBar1.Value = FileLenght
DoEvents
Loop
CloseHandle hFileSource
CloseHandle hFileDestination
End Function
However this code is flawed. I make sure to emphasize that to anyone who might try to use this code first, it's wrong. When I figure it out, I'll post a correct working version. The code doesn't work, when ran it gives me the following error: "Bad filename or number." And it hightlights the following line:
Code:
Do While Not EOF(hFileSource)
I'm not even sure this is the proper way to copy files larger then 2gb.
Please advise,
Knight.
-
Re: Reading chunks from large files
If Gremmy said it copies large files, the I believe that something is flawed on your end. I did a quick search, and found this thread (kind of old, but might be fitting?)
http://www.msfn.org/board/copy2gb-t81200.html
-
Re: Reading chunks from large files
Gremmy said it can handle large files, he never said it can copy large files. Simply because the routine wasn't made to do that. I need to develop that routine. Furthermore, I can't try his program since it fails before I can use it, as posted in my previous post.
Your post is interesting, but I believe it relates to the operating system directly. My OS can copy large files, but my VB6 program can't. I need to know how to do it in VB6. I want a status bar so I can't use the FileSystemObject method of copying files, besides on a 2+ gig file it would appear to have locked....not what I want.
Knight.
-
Re: Reading chunks from large files
Your code looks quite good and should work with a little modification.
The EOF() function expects a VB filenumber, but you give it the filehandle hFileSource you received from the API call to CreateFile(). This is not a valid VB filenumber, so you cannot use the EOF() function here.
Either there is an API GetEndOffile(hFileHandle) you can use (I don't know, didn't look it up), or you simply determine the file's end through your counting down from the filesize. If it reaches zero (or underruns zero) the last chunk is reached.
-
Re: Reading chunks from large files
I've not been around much lately .. Between the Job and my 4th grandchilds birth last week, I've been batteling to keep up ... how ever i did get your msg, Knightofoldcode...
Quote:
Gremmy,
I've read one of your posts about breaking the 2 gb limit, and I've read your tutorial. However, I'm still confused as to how to copy a file larger then 2 gb. I tried reposting on the same thread, however no one else was helping, just telling me to go through debug tools and look at previous code. I've looked at previous code.. to that end I downloaded your hex editor code, however it fails when I try to use it, on the line:
API_OpenFile Tmp_File, File_Num, File_Len
inside the File -> Open menu code.
Please help me figure out how to create a subroutine to simply copy a large 2gb+ file from one place to another!
Much MUCH thanks,
Knightofoldcode.
You will have to write a special function to take care of the copy ... The Module handles the file open, read, write and close. You need to handle the bits between the read and write...
Psudo code (Using the Functions from the article module)
Code:
Public Sub API_CopyFile (Source as string, Dest as String)
Dim FileSize as currency
Dim FileWSize as currency
Dim Pos as Currency
Dim FreadNum as integer
Dim FwrteNum as integer
Dim Block() as byte
Dim BlockSize as integer
Dim Blockread as integer
'Set Block size according to personal pref.
blocksize = 2048 ' 2K
redim Block(blocksize)
API_Openfile (source, FreadNum, Filesize)
API_Openfile (Dest, FwrteNum, FileWsize)
If FileWsize>0 then
'Dest has data .. lets clear it
API_SetEndOfFile (FwrteNum,0)
End If
pos = 0
while Pos < FileSize
Blockread = BlockSize
API_ReadFile(FReadNum, Pos, Blockread, Block)
API_WriteFile(FWrteNum, Pos, Blockread, Block)
Pos = Pos + Blockread
'show whatever progress you want here ..
Wend
API_CloseFile(FreadNum)
API_CloseFile(FwrteNum)
End Sub
The Above sub is a very good start on how to do a copy for files over 2 gig..
I've looked over the code you posted and it is almost the same way that Windows copies files, and 'WILL NOT' work with 2Gig or larger files..
If you took carefull notice in the article the third para' describes why .. Long's have a MAX value of 2,147,483,647 (2gig), and your code uses only one of the two longs for size, limiting you to 4gig if you managed to get it to work. The Module uses the Currency type for the file size and position variables, these now have a max of ~970gig, far exceeding any std hard drive size.... and has some nifty code to convert the currency to the two longs needed for the API's...
hope this helps ...
Gremmy.....
-
Re: Reading chunks from large files
Gremmy,
I appreciate your help, and wasn't trying to be rude to anyone. All help is appreciated.
I think I was getting confused on the very many different ways of doing it. I've got easily 10 subroutines that all either didn't work properly, or didn't copy above 2 gig files.
I don't know how many times I've read your tutorial, yet still never been able to understand how to use the module.
Again, thank you, all.
Gremmy, doesn't the subroutine have to tell it where to advance the pointer? I thought I read somewhere that I'd need to advance the pointer since that method doesn't do it for you? I haven't checked the posted code yet, I guess I should do that.... ;)
Knight.
-
Re: Reading chunks from large files
oops ... you are right ... i did leave out code to advance the pointer.. ( i fixed it in the code above ) .. the line Pos = Pos + BlockRead ... slight over sight when typing in the psudo code .. also posibly tiredness ...
API usage is slightly advanced and this module does take it a little over the top .... the thing is, once you understand API's, its easier to understand how the module works...
also i'm not sure as to why the Hex Editor does not want to run for you but i suspect that it could be that its trying t o pass a Long instead of currency type to the Sub.....
Gremmy..
-
Re: Reading chunks from large files
GremlinSA or any other member,
I apologize for constantly bringing back up this dead thread... But I am still having problems getting large files to copy, for that matter, to generate md5sums for large files also.
I keep getting soo discouraged that I give up, then the project that I'm working on needs to be done, so I give it another go...
Using the most recent code posted, I get error of
"ByRef argument type mismatch"
It highlights the line: Call API_OpenFile(Source, FreadNum, FileSize)
And in particular it highlights the "FreadNum" of that line.
When I download the hex editor example, and click, "open" I get a error of the same type!
Is there anyway I can get a little more help? I've tried everything I can think of, and nothing gets me further, otherwise I'd be willing to post what I've done, but nothing is helping....
Knight.
-
Re: Reading chunks from large files
Sir Knight of the old code, how are you calculating the MD5 hash? Are you using the API or using something you found over at PSC?
As for other large file reading/writing, please see this thread... http://www.vbforums.com/showthread.php?t=531321
Good Luck
-
Re: Reading chunks from large files
You better show more code. How is API_OpenFile() declared? What data types are the variables you pass to it?
-
Re: Reading chunks from large files
Hi,
To read the large files size (more than 2Gb) in VB6 I use the following way:
Dim fs As Object
Set fs = CreateObject("Scripting.FileSystemObject")
Set fi = fs.GetFile(CStr(FileName))
FileSize = fi.Size
then I open file:
hFile= FreeFile
Open FileName For Binary As hFile
last, it is required to read a chunk string with given BlockSize and from some Position of this large file.
So I would appreciate if you help me with this API. I.e. inside for the following function:
strCurr$=GetDataStringFromPosition (hFile, Position, BlockSize)
Thanks in advance for your help and prompt response.
-
Re: Reading chunks from large files
Read every post, starting from page #1. If that doesn't answer your question, then START A NEW THREAD.