How to read bytes in chunks to avoid out of memory exception? Or something else...
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 6 of 6

Thread: How to read bytes in chunks to avoid out of memory exception? Or something else...

Hybrid View

  1. #1
    Join Date
    Nov 2012
    Posts
    1

    Question How to read bytes in chunks to avoid out of memory exception? Or something else...

    Hello,

    after searching and trying to find a solution for 2 days i finnaly decided to post here to ask for help...

    I have a code to read and write bytes of a file (like an hex editor) working very good taken from here: http://www.novahq.net/forum/showthread.php?t=42592


    Here goes the code:

    Code:
    Public Class Form1
        Private Shared Function HexStringToByteArray(ByRef strInput As String)
    As Byte()
            Dim length As Integer
            Dim bOutput As Byte()
            Dim c(1) As Integer
            length = strInput.Length / 2
            ReDim bOutput(length - 1)
            For i As Integer = 0 To (length - 1)
                For j As Integer = 0 To 1
                    c(j) = Asc(strInput.Chars(i * 2 + j))
                    If ((c(j) >= Asc("0")) And (c(j) <= Asc("9"))) Then
                        c(j) = c(j) - Asc("0")
                    ElseIf ((c(j) >= Asc("A")) And (c(j) <= Asc("F"))) Then
                        c(j) = c(j) - Asc("A") + &HA
                    ElseIf ((c(j) >= Asc("a")) And (c(j) <= Asc("f"))) Then
                        c(j) = c(j) - Asc("a") + &HA
                    End If
                Next j
                bOutput(i) = (c(0) * &H10 + c(1))
            Next i
            Return (bOutput)
        End Function
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As
    System.EventArgs) Handles Button1.Click
            Dim myFile As String = "..\..\Test.jpg"
            Dim myBytes As Byte() =
    My.Computer.FileSystem.ReadAllBytes(myFile)
            Dim txtTemp As New System.Text.StringBuilder()
            For Each myByte As Byte In myBytes
                txtTemp.Append(myByte.ToString("X2"))
            Next
            RichTextBox1.Text = txtTemp.ToString()
        End Sub
    
        Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As
    System.EventArgs) Handles Button2.Click
            Dim myFile As String = "..\..\Test2.jpg"
            Dim myBytes As Byte() = HexStringToByteArray(RichTextBox1.Text)
            My.Computer.FileSystem.WriteAllBytes(myFile, myBytes, False)
        End Sub
    End Class
    This is working very good, BUT only with small/medium size files. If i try to load a 512MB or a bigger file i will get "System.OutOfMemoryException" error.

    I know that in order to make this work (read bigger files without error) i have to read files in small pieces (or chunks whatever you wanna name it), or maybe read it byte by byte i suppose...

    The only problem is that i couldn't find a way to do it. I did find some codes but for some reason i wasn't able to make that codes work

    I apreciate if someone could give me a push on this.

    ...However, i've thinking and maybe i don't need to read the whole file because i just need to add some bytes to the end of the file and then remove those again.

    I'll try to explain what i need:

    I only need to add this "504B0506000000000000000000000000000000000000" bytes at the end of a zip in onder do blind/hide contents of zip and then i need to read in again and remove those bytes.

    a) Append bytes to the end of file

    b) Remove those bytes again

    This sounds really simple but from all codes and examples i have found i couldn't adapt them to my code.

    Thank you

  2. #2
    Join Date
    Jan 2006
    Location
    Chicago, IL
    Posts
    14,949

    Re: How to read bytes in chunks to avoid out of memory exception? Or something else..

    Adding to a ZIP will invalidate it. The CHECKSUM becomes wrong. Plus, you'd have to know the ZIP file format.
    David

    CodeGuru Article: Bound Controls are Evil-VB6
    2013 Samples: MS CODE Samples

    CodeGuru Reviewer
    2006 Dell CSP
    2006, 2007 & 2008 MVP Visual Basic
    If your question has been answered satisfactorily, and it has been helpful, then, please, Rate this Post!

  3. #3
    DataMiser is offline Super Moderator Power Poster
    Join Date
    Jul 2008
    Location
    WV
    Posts
    4,813

    Re: How to read bytes in chunks to avoid out of memory exception? Or something else..

    To read a file in chunks just define a chunk size to use then check the filesize to make sure there is at least that many bytes remaining in the file before each read. If less bytes the read the remaining bytes as a chunk.

    In the case of appending you do not need to read anything from the file, just jump to the end and read. In the case of removing just use a chunk read/write to create a new file stopping when you get to the end of the data you want to save. FileSize-BytesToRemove

    What chunk size to use depends on your needs but always remember larger chunks will be faster. for example reading a file in 1k chunks will be much much faster than reading byte by byte and using 4k 0r 8k will be faster still. Play around with it a bit using different size chunks and see what results you get and the difference it makes in speed on large files.
    Always use [code][/code] tags when posting code.

  4. #4
    Join Date
    Jan 2006
    Location
    Chicago, IL
    Posts
    14,949

    Re: How to read bytes in chunks to avoid out of memory exception? Or something else..

    DM - Adding this to a zip? "504B0506000000000000000000000000000000000000" No problem with that?
    David

    CodeGuru Article: Bound Controls are Evil-VB6
    2013 Samples: MS CODE Samples

    CodeGuru Reviewer
    2006 Dell CSP
    2006, 2007 & 2008 MVP Visual Basic
    If your question has been answered satisfactorily, and it has been helpful, then, please, Rate this Post!

  5. #5
    DataMiser is offline Super Moderator Power Poster
    Join Date
    Jul 2008
    Location
    WV
    Posts
    4,813

    Re: How to read bytes in chunks to avoid out of memory exception? Or something else..

    Apparently his goal is to effectivly break the zip file then later repair it so..... if that is what he wants to do then that may work.
    Always use [code][/code] tags when posting code.

  6. #6
    Join Date
    Jun 2005
    Location
    JHB South Africa
    Posts
    3,772

    Re: How to read bytes in chunks to avoid out of memory exception? Or something else..

    Its a VB6 project but the Hex editor project here will give you good idea on how to do this... It can handle a file of up to 920 gig, by reading it sector for sector, as needed...

    NOTE: it uses API's to open, read and write to files.
    Articles VB6 : Break the 2G limit - Animation 1, 2 VB.NET : 2005/8 : Moving Images , Animation 1 , 2 , 3 , User Controls
    WPF Articles : 3D Animation 1 , 2 , 3
    Code snips: VB6 Hex Edit, IP Chat, Copy Prot., Crop, Zoom : .NET IP Chat (V4), Adv. ContextMenus, click Hotspot, Scroll Controls
    Find me in ASP.NET., VB6., VB.NET , Writing Articles, My Genealogy, Forum
    All VS.NET: posts refer to VS.NET 2008 (Pro) unless otherwise stated.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Azure Activities Information Page

Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center