CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 1 of 2 12 LastLast
Results 1 to 15 of 20
  1. #1
    Join Date
    Jan 2014
    Posts
    8

    C++ Memory Mapped Files Trouble

    Hello everybody,

    I have a data buffer project in Windows 7 x64 embedded OS (using Visual Studio 2008), that would work simply like that:

    One writer application will send data (no network protocols, just procedure call on the same machine) to my app like 20 packages per second, each data packages will be approximately 3 MB size and comes with a timestamp tag.

    My application will store each data item for 100 minutes and then remove. (So I can calculate the total size from beginnig no need for dynamic allocation etc...)

    Meanwhile there will be up to 5 reader applications which will query data from my app via Timestamp tag and retreive data (no updates or deletitions on data by reader apps).

    So since the data in my buffer app can grow over 50GB I don't think that shared memory is going to work for my case.

    I'm thinking about using Boost Memory Mapped Files or Windows API Memory Mapped Files.
    So theoratically I will crate a fixed size File on harddisk (lets say 50GB) and then map some portion to RAM write the data to this portion, if the reader applications wants to use the data which is mapped currently on memory, then they will use directly, otherwise they will map some other potion of the file to their address spaces and use from there...

    My problem is I haven't use Map File concept at all and I'm not a C++ developer so I'm not very familiar with the language fundementals. I've searched tutorials of boost and msdn for windows as well but there is not too much code example.

    So I don't know how to map some portion of the data to memory and manage it, also how to search data in file or memory according to the timestamp tag. Yes there are codes for creating files and mapping the whole file to memory but none for dividing it into regions, aligning or padding the data which I need for my case.

    Any help with some code portion or sample projects would help extremely...

    Thank you very much for reading and further helps...

  2. #2
    Join Date
    Jul 2005
    Location
    Netherlands
    Posts
    2,042

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by vecihi View Post
    My problem is I haven't use Map File concept at all and I'm not a C++ developer so I'm not very familiar with the language fundementals. I've searched tutorials of boost and msdn for windows as well but there is not too much code example.
    You failed to mention what you can do. Do you program at all? If so, which language?
    Quote Originally Posted by vecihi View Post
    So I don't know how to map some portion of the data to memory and manage it, also how to search data in file or memory according to the timestamp tag. Yes there are codes for creating files and mapping the whole file to memory but none for dividing it into regions, aligning or padding the data which I need for my case.
    Last time I checked (more than a year ago), the boost library does not support regions for memory mapped files, but the win32 API does. So you can write your own abstraction layer using the win32 API that gives you a more useful interface for your specific problem.
    Cheers, D Drmmr

    Please put [code][/code] tags around your code to preserve indentation and make it more readable.

    As long as man ascribes to himself what is merely a posibility, he will not work for the attainment of it. - P. D. Ouspensky

  3. #3
    Join Date
    Jan 2014
    Posts
    8

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by D_Drmmr View Post
    You failed to mention what you can do. Do you program at all? If so, which language?

    Yes I was doing development on Java and VB 6.0 and VB.NET.

    Last time I checked (more than a year ago), the boost library does not support regions for memory mapped files, but the win32 API does. So you can write your own abstraction layer using the win32 API that gives you a more useful interface for your specific problem.
    As far as I know it is possible in boost as well, at least the tutorials say so. But the problem is they don't explain how. That why I'm asking it, may be some experienced developers on boost may help.

    Do you know how it can be done with Windows API? Any sample code?

  4. #4
    2kaud's Avatar
    2kaud is online now Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    7,822

    Re: C++ Memory Mapped Files Trouble

    Using the windows memory mapped files api requires some familiarity with both c/c++ and with the windows api. Programming your requirements in c/c++ when you are not very familiar with the language fundamentals is a big ask. If you are not familiar with c/c++ why does this program need to written in this language? Are there other programming languages of which you are familiar?

    For the windows memory mapped files api set, have a look at
    http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx and its links.

    Also
    http://www.codeproject.com/Articles/...y-Mapped-Files
    which provides some code as a starter

    http://social.msdn.microsoft.com/For...orum=vcgeneral

    http://www.codeguru.com/cpp/article....using-RAII.htm


    Good luck!
    Last edited by 2kaud; January 27th, 2014 at 07:48 AM. Reason: Updated references
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++23 Compiler: Microsoft VS2022 (17.6.5)

  5. #5
    Join Date
    Jan 2014
    Posts
    8

    Re: C++ Memory Mapped Files Trouble

    Thank you for reply, I wrote it in in my previous reply but I forget it in "QUOTE" tag

    I developed with Java, VB 6.0, VB.NET and a little bit with C# before, but this projects must be in C++ (according to proj specs.)

    I saw the MSDN examples before but they have some general stuff. I mean the codes are just too low level. I wondered if anyone has used dividing a mapped files into regions and manage it in his project, may be a real situation example would more help.

    Thanks anyway.

  6. #6
    2kaud's Avatar
    2kaud is online now Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    7,822

    Re: C++ Memory Mapped Files Trouble

    Have a look at the link to the codeproject site I gave in my previous reply. It provides the code for a generic c++ class for using memory mapped files which might your task easier.
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++23 Compiler: Microsoft VS2022 (17.6.5)

  7. #7
    Join Date
    Jul 2005
    Location
    Netherlands
    Posts
    2,042

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by vecihi View Post
    I developed with Java, VB 6.0, VB.NET and a little bit with C# before, but this projects must be in C++ (according to proj specs.)
    Then either you need to change the project specs or you need to learn C++ if you want to achieve anything. You cannot bluff your way through C++. It is completely unlike the languages you mention, even if the syntax looks familiar.
    Cheers, D Drmmr

    Please put [code][/code] tags around your code to preserve indentation and make it more readable.

    As long as man ascribes to himself what is merely a posibility, he will not work for the attainment of it. - P. D. Ouspensky

  8. #8
    2kaud's Avatar
    2kaud is online now Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    7,822

    Re: C++ Memory Mapped Files Trouble

    For a book covering the windows areas you will need have a look at
    Windows via c/c++ by Jeffrey Richter
    http://www.amazon.co.uk/Windows-Via-...ichter+windows

    Note that earlier versions of this book also cover memory mapped files and can be obtained very cheaply if you can't stretch to the cost of the current new one eg
    Advanced Windows by Jeffrey Richter
    http://www.amazon.co.uk/Advanced-Win...ichter+windows

    These will give you the details you need from the windows side of things, but (as stated by D_Drmmr in post #7) you must become somewhat proficient with c++ before you start with this otherwise you will get into a heap of problems.
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++23 Compiler: Microsoft VS2022 (17.6.5)

  9. #9
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626

    Re: C++ Memory Mapped Files Trouble

    You cannot use memorymapping for a buffer larger than the available virtual memory space.
    For Win32 that means there is a hard limit of 3Gb, although since your program and windows kernel DLL's take part of that, you're Lucky if you can even allocate a contiguous region of 1.5Gb (in practice even 1gb will be an issue on some PC's).

    Making a Win64 app pretty much removes that limit, but it's still not a very good method because you're hogging a lot of Virtual memory.

    I would look at a solution with either an actual database (if you can find one that allows 3Mb data chunks) or alternatively... store the data in regular files, and maintain a database to the files or maintain a list in memory of references to the files (rather than keeping all the data in memory).

  10. #10
    Join Date
    Jan 2014
    Posts
    8

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by OReubens View Post
    You cannot use memorymapping for a buffer larger than the available virtual memory space.
    For Win32 that means there is a hard limit of 3Gb, although since your program and windows kernel DLL's take part of that, you're Lucky if you can even allocate a contiguous region of 1.5Gb (in practice even 1gb will be an issue on some PC's).

    Making a Win64 app pretty much removes that limit, but it's still not a very good method because you're hogging a lot of Virtual memory.

    I would look at a solution with either an actual database (if you can find one that allows 3Mb data chunks) or alternatively... store the data in regular files, and maintain a database to the files or maintain a list in memory of references to the files (rather than keeping all the data in memory).
    Thank you. Yes as you mentioned i know the limitations for 32 bit systems but the project will be definitely 64 bit. So at least for the addressinf there won't be any problems.

    As i said before what i'm trying to implement is instead of mapping the whole file just map a portion (lets say 1 gb) of it. While writing data to that portion the reader applications theoretically can reach the written data immediately (since the data is available on ram)

    But for the older data yes you're right once again they have to map that portion of data to their address spaces and read from there.

    Reading the data can wait a little bit, i mean a little delay is acceptable. But writing is critical because we can not miss any data package.

    As far as i know the operations like flushing and mapping other portions of the file are handled by kernel so i don't need to deal with them.

    Also boost and microsoft offers this method for sharing data between seperate processes especially for extremely large files. That why i'm into this way.

    Using a third party database would create some problems with the customers so i have to develop it myself and unfortunately this is the best way that i can find up to now.

    In every documentation its said that "it can be done very easly" but there is no example or code part about it. Thats the reason that i wrote it here.

    Thank you again for your time and ideas. Further ideas also highly appreciated

  11. #11
    Join Date
    Aug 2000
    Location
    New York, NY, USA
    Posts
    5,656

    Re: C++ Memory Mapped Files Trouble

    May I do a little math?

    20 packages per sec at 3MB a piece is 60MB of data per second.
    This is 3,600MB per minute, or 360GB for a 100 min you want to keep your data (not 50GB).
    What kind of storage do you have on that embedded system? Is it both large and fast?
    How frequent will your 5 readers request the data?
    Writing of 60MB/sec will exhaust throughput of the low end HDD, leaving nothing for reading.
    Vlad - MS MVP [2007 - 2012] - www.FeinSoftware.com
    Convenience and productivity tools for Microsoft Visual Studio:
    FeinWindows - replacement windows manager for Visual Studio, and more...

  12. #12
    Join Date
    Jan 2014
    Posts
    8

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by VladimirF View Post
    May I do a little math?

    20 packages per sec at 3MB a piece is 60MB of data per second.
    This is 3,600MB per minute, or 360GB for a 100 min you want to keep your data (not 50GB).
    What kind of storage do you have on that embedded system? Is it both large and fast?
    How frequent will your 5 readers request the data?
    Writing of 60MB/sec will exhaust throughput of the low end HDD, leaving nothing for reading.
    Yeah you're right, these numbers are the maximum limits in theory, but in real application these would be a little less than these.

    I will definitely use a Solid State Drive whose size is big enough for storing the data in real application. But I guess the real file size would vary between 50GB - 150GB.

    And for the reader applications frequency, there is nothing much to say, we have to try and see the performance in real system with the real equipment.

  13. #13
    Join Date
    Aug 2000
    Location
    New York, NY, USA
    Posts
    5,656

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by vecihi View Post
    Yeah you're right, these numbers are the maximum limits in theory, but in real application these would be a little less than these.

    I will definitely use a Solid State Drive whose size is big enough for storing the data in real application. But I guess the real file size would vary between 50GB - 150GB.
    So you plan to write variable-length records to a file? Then recycling becomes tricky. If you want to avoid copying entire multi-gig file, you would have to replace records in-place, dealing with the new record covering more than one. The maintenance becomes a chore.
    Also, to search that kind of file for your time stamp is not trivial.
    I would certainly use fix-length records, that allows easy random access to the file. Then the question is - why would you need memory-map it? Your writer would just write sequentially, keeping a pointer to the next record. Let Windows worry about caching recently accessed records.

    One more question - you don't have 150GB of RAM on that system, do you?
    Vlad - MS MVP [2007 - 2012] - www.FeinSoftware.com
    Convenience and productivity tools for Microsoft Visual Studio:
    FeinWindows - replacement windows manager for Visual Studio, and more...

  14. #14
    Join Date
    Jan 2014
    Posts
    8

    Re: C++ Memory Mapped Files Trouble

    Quote Originally Posted by VladimirF View Post
    So you plan to write variable-length records to a file? Then recycling becomes tricky. If you want to avoid copying entire multi-gig file, you would have to replace records in-place, dealing with the new record covering more than one. The maintenance becomes a chore.
    Also, to search that kind of file for your time stamp is not trivial.
    I would certainly use fix-length records, that allows easy random access to the file. Then the question is - why would you need memory-map it? Your writer would just write sequentially, keeping a pointer to the next record. Let Windows worry about caching recently accessed records.

    One more question - you don't have 150GB of RAM on that system, do you?
    Oh, I'm sorry I thought I wrote it in the first post but I think I forgat.

    This application is going to be a DLL. For each different data type we're going to built another DLL. So to your question; the Data packages sizes are exactly same for one DLL and known from the compile time. There is no variable-length reacords, they're all same size.

    When I said, the file size is going to vary from 50 -150GB, I mean for example for one DLL, the data packages will be 1 MB each and 10 packages per second and 100 minutes to store the data. For another instance, the packages will be 3MB and 3 packages per second but lets say 60 minutes for storing data and so on...

    So these will be different applications that will be built seperately. So in our case the file size, frequency of the data, each data item size and time to store each data item is fixed & known from the compile time, no doubt about that.

    My purpose to use mermory map files is to increase the I/O performance since we're trying to implement a performance critical app. As far as I know memory mapped files are the fastest way for sharing data like huge files between multiprocess systems.

    Am I wrong?

    And yes, I don't have that much RAM. The hardware would be something like Intel I7 processor, with 8GB - 16GB RAM and a SSD which has enough capacity for the real numbers.

    Thanks.

  15. #15
    2kaud's Avatar
    2kaud is online now Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    7,822

    Re: C++ Memory Mapped Files Trouble

    My purpose to use mermory map files is to increase the I/O performance since we're trying to implement a performance critical app. As far as I know memory mapped files are the fastest way for sharing data like huge files between multiprocess systems.

    Am I wrong?
    Why don't you write some simple programs with files on this hardware configuration and find out? Why have all the data packages in one file? Why not a file per data package with the name as the time stamp?
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++23 Compiler: Microsoft VS2022 (17.6.5)

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured