How to read bytes in chunks to avoid out of memory exception? Or something else...
Hello,
after searching and trying to find a solution for 2 days i finnaly decided to post here to ask for help...
I have a code to read and write bytes of a file (like an hex editor) working very good taken from here: http://www.novahq.net/forum/showthread.php?t=42592
Here goes the code:
Code:
Public Class Form1
Private Shared Function HexStringToByteArray(ByRef strInput As String)
As Byte()
Dim length As Integer
Dim bOutput As Byte()
Dim c(1) As Integer
length = strInput.Length / 2
ReDim bOutput(length - 1)
For i As Integer = 0 To (length - 1)
For j As Integer = 0 To 1
c(j) = Asc(strInput.Chars(i * 2 + j))
If ((c(j) >= Asc("0")) And (c(j) <= Asc("9"))) Then
c(j) = c(j) - Asc("0")
ElseIf ((c(j) >= Asc("A")) And (c(j) <= Asc("F"))) Then
c(j) = c(j) - Asc("A") + &HA
ElseIf ((c(j) >= Asc("a")) And (c(j) <= Asc("f"))) Then
c(j) = c(j) - Asc("a") + &HA
End If
Next j
bOutput(i) = (c(0) * &H10 + c(1))
Next i
Return (bOutput)
End Function
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button1.Click
Dim myFile As String = "..\..\Test.jpg"
Dim myBytes As Byte() =
My.Computer.FileSystem.ReadAllBytes(myFile)
Dim txtTemp As New System.Text.StringBuilder()
For Each myByte As Byte In myBytes
txtTemp.Append(myByte.ToString("X2"))
Next
RichTextBox1.Text = txtTemp.ToString()
End Sub
Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button2.Click
Dim myFile As String = "..\..\Test2.jpg"
Dim myBytes As Byte() = HexStringToByteArray(RichTextBox1.Text)
My.Computer.FileSystem.WriteAllBytes(myFile, myBytes, False)
End Sub
End Class
This is working very good, BUT only with small/medium size files. If i try to load a 512MB or a bigger file i will get "System.OutOfMemoryException" error.
I know that in order to make this work (read bigger files without error) i have to read files in small pieces (or chunks whatever you wanna name it), or maybe read it byte by byte i suppose...
The only problem is that i couldn't find a way to do it. I did find some codes but for some reason i wasn't able to make that codes work :(
I apreciate if someone could give me a push on this.
...However, i've thinking and maybe i don't need to read the whole file because i just need to add some bytes to the end of the file and then remove those again.
I'll try to explain what i need:
I only need to add this "504B0506000000000000000000000000000000000000" bytes at the end of a zip in onder do blind/hide contents of zip and then i need to read in again and remove those bytes.
a) Append bytes to the end of file
b) Remove those bytes again
This sounds really simple but from all codes and examples i have found i couldn't adapt them to my code.
Thank you
Re: How to read bytes in chunks to avoid out of memory exception? Or something else..
Adding to a ZIP will invalidate it. The CHECKSUM becomes wrong. Plus, you'd have to know the ZIP file format.
Re: How to read bytes in chunks to avoid out of memory exception? Or something else..
To read a file in chunks just define a chunk size to use then check the filesize to make sure there is at least that many bytes remaining in the file before each read. If less bytes the read the remaining bytes as a chunk.
In the case of appending you do not need to read anything from the file, just jump to the end and read. In the case of removing just use a chunk read/write to create a new file stopping when you get to the end of the data you want to save. FileSize-BytesToRemove
What chunk size to use depends on your needs but always remember larger chunks will be faster. for example reading a file in 1k chunks will be much much faster than reading byte by byte and using 4k 0r 8k will be faster still. Play around with it a bit using different size chunks and see what results you get and the difference it makes in speed on large files.
Re: How to read bytes in chunks to avoid out of memory exception? Or something else..
DM - Adding this to a zip? "504B0506000000000000000000000000000000000000" No problem with that?
Re: How to read bytes in chunks to avoid out of memory exception? Or something else..
Apparently his goal is to effectivly break the zip file then later repair it so..... if that is what he wants to do then that may work.
Re: How to read bytes in chunks to avoid out of memory exception? Or something else..
Its a VB6 project but the Hex editor project here will give you good idea on how to do this... It can handle a file of up to 920 gig, by reading it sector for sector, as needed...
NOTE: it uses API's to open, read and write to files.