dcsimg
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 6 of 6

Thread: size of allocated shared memory

  1. #1
    Join Date
    May 2009
    Location
    Boston
    Posts
    356

    size of allocated shared memory

    Hello,

    I have two c++ binaries that perform some IPC using shared memory. I didn't write this code and the comments are a little unclear. I need to develop a unambiguous formula for determining the total required amount of shared memory. The data that is passed is the output, basically an array of column headers and a second array of values. It sometimes becomes necessary to change the amount of shared memory if the total amount of output changes significantly over time.

    The SHMEM is allocated as an integer number of pages of size 4096
    Code:
    #define SHMEMSIZE (4096 * 15)
    // used as third argument in the call to memset
    memset(lpvMem, '\0', SHMEMSIZE);
    The comments say that this is the number of kibibit pages. I thought that a kibibit was 1024 bits, so this comment doesn't make sense to me. Perhaps I am thinking of an older definition from 8-bit operating systems or something like that.

    At any rate, the following are passed through shared memory,
    Code:
    - an array of characters with sizex elements of 11 characters per element
      a character in 32-bit takes 8-bits, so the size of this array should be
      sizex * (8 * 11)
    - an array of float with sizex elements
      a float in 32-bit takes 32 bits, so the size of this array should be
      sizex * 32
    At 4096 bits per page, I should need
    Code:
    pages = ( (sizex * (8 * 11)) + (sizex * 32) ) / 4096
    the size of sizex changes from time to time and this is where the shared memory size may also need to be increased.

    At the moment, I am doing integer arithmetic in bash like,
    Code:
    # example of number of outputs
    sizex=305
    # calculate the required number of bits
    total_output = (sizex * (8 * 11)) + (sizex * 32)
    # calculate the number of pages, round up
    num_pages = (( ($total_output + (4095)) / 4096 ))
    # add an extra page
    num_pages=$(($num_pages + 1))
    The bash script is called by make and modifies the appropriate locations in the src code before compilation.

    I realize that this is nowhere near ideal but it works ok in what is a complicated situation with no easy fix.

    My main question is a math check here. Am I calculating this correctly? Am I missing anything? I do build this in 64-bit as well, does that change things? I have not noticed that a value will work in 32-bit but not 64-bit, but I am not very confident about that assessment.

    As always, suggestions are appreciated.

    LMHmedchem
    Last edited by LMHmedchem; October 7th, 2019 at 09:24 PM.

  2. #2
    2kaud's Avatar
    2kaud is offline Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    6,877

    Re: size of allocated shared memory

    1 kibibit is indeed defined as 1024 bits.

    Why are the calculations based on bits and not on bytes?

    For 64-bit you need to check for your compiler the size of float. The size of char will still be 1 byte (8 bits).

    Code:
    #define SHMEMSIZE (4096 * 15)
    // used as third argument in the call to memset
    memset(lpvMem, '\0', SHMEMSIZE);
    memset() requires the 3rd argument to be the number of characters to set. So in this case it is setting 4096 * 15 = 61440 characters. How is memory for lpvMem allocated? What does the 15 represent? Is this the number that is changed to be the value of num_pages from the bash script?
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++17 Compiler: Microsoft VS2019 (16.3.5)

  3. #3
    Join Date
    May 2009
    Location
    Boston
    Posts
    356

    Re: size of allocated shared memory

    Quote Originally Posted by 2kaud View Post
    Why are the calculations based on bits and not on bytes?
    I am not sure since I didn't write this section.

    I think the code I posted earlier was not helpful, meaning it looks like that was just the statement that initialized the shared memory block to 0. In that context, SHMEMSIZE, is used to know the size of the area to be written with 0. SHMEMSIZE is used earlier by CreateFileMapping() which is I believe where the memory is actually allocated. I think this gives a better picture.

    Code:
    // defined in a header file
    #define LPVOID void*
    
    // declare a pointer for the shared memory, null the pointer
    static LPVOID lpvMem = NULL;
    
    // create file mapping object
    hMapObject = CreateFileMapping( (HANDLE) 0xFFFFFFFF, 
                                    NULL,
                                    PAGE_READWRITE,
                                    0,
                                    SHMEMSIZE,
                                    "arrayShmem4P" );
    
    // make sure the object was created
    if( !hMapObject ) { cerr << "failed to create file mapping in parent." << endl;  exit(1); }
    
    //FILE_MAP_READ, set the address of the pointer lpvMem to the object
    lpvMem = MapViewOfFile( hMapObject,     
                            FILE_MAP_WRITE,
                            0,            
                            0,           
                            0 );
    
     // make sure lpvMem is not still NULL         
    if( !lpvMem ) { cerr << "failed map file in parent." << endl;  exit(1); }
    
    // initalize the shared memory to 0s
    memset(lpvMem, '\0', SHMEMSIZE);
    Quote Originally Posted by 2kaud View Post
    Code:
    #define SHMEMSIZE (4096 * 15)
    // used as third argument in the call to memset
    memset(lpvMem, '\0', SHMEMSIZE);
    The memory is allocated using the CreateFileMapping() 4th and 5th arguments, dwMaximumSizeHigh and dwMaximumSizeLow which are 0 and SHMEMSIZE respectively. The documentation for CreateFileMapping() is very unclear to me. I guess at the moment I am passing 0 and 61440 for those two parameters. I really have no idea how much memory I am allocating. At runtime, the application has a rather small memory footprint, but I don't know where the shared memory would show up.

    Quote Originally Posted by 2kaud View Post
    memset() requires the 3rd argument to be the number of characters to set. So in this case it is setting 4096 * 15 = 61440 characters. How is memory for lpvMem allocated? What does the 15 represent? Is this the number that is changed to be the value of num_pages from the bash script?
    At the moment, the interer 15 is the number I am changing using $num_pages from the script. I guess that was ment to be a straightforward way to modify the ammount of memory that was allocated.

    Quote Originally Posted by 2kaud View Post
    For 64-bit you need to check for your compiler the size of float. The size of char will still be 1 byte (8 bits).
    I guess I will need to add a script to determine the sizes for the current environment and then add a compiler directive of some kind to adjust where necessary.

    LMHmedchem
    Last edited by LMHmedchem; October 8th, 2019 at 11:07 AM.

  4. #4
    2kaud's Avatar
    2kaud is offline Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    6,877

    Re: size of allocated shared memory

    Code:
    (HANDLE) 0xFFFFFFFF
    For CreateFileMapping(), this would be easier to understand if replaced by INVALID_HANDLE_VALUE. If this value is used, then the dwMaxmimumSizeHigh and dwMaxmimumSizeLow parameters are used.

    The memory is allocated using the CreateFileMapping() 4th and 5th arguments, dwMaximumSizeHigh and dwMaximumSizeLow which are 0 and SHMEMSIZE respectively. The documentation for CreateFileMapping() is very unclear to me. I guess at the moment I am passing 0 and 61440 for those two parameters. I really have no idea how much memory I am allocating.
    The 4th argument (dwMaximumSizeHigh) is only used if the file mapping is more than 2 ^ 32 bytes - otherwise this is 0 and the 5th (dwMaximumSizeLow) is used to specify the maximum size in bytes of the mapped object. Which, using 15 for SHMEMSIZE, is indeed 61440 bytes.

    MapViewOfFile() simply says that the mapping used starts from the beginning and extends to the end of the specified file mapping.

    Effectively after MapViewOfFile(), lpvMem points to the beginning of some allocated memory that is SHMEMSIZE in bytes - and can be used like 'ordinary memory'. Like ordinary memory, accessing memory beyond its limits can/will cause problems.
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++17 Compiler: Microsoft VS2019 (16.3.5)

  5. #5
    Join Date
    May 2009
    Location
    Boston
    Posts
    356

    Re: size of allocated shared memory

    Quote Originally Posted by 2kaud View Post
    The 4th argument (dwMaximumSizeHigh) is only used if the file mapping is more than 2 ^ 32 bytes - otherwise this is 0 and the 5th (dwMaximumSizeLow) is used to specify the maximum size in bytes of the mapped object. Which, using 15 for SHMEMSIZE, is indeed 61440 bytes.
    So at the moment, I am allocating 61440 bytes and not 61440 bits like I thought?

    Based on the data that is being passed,

    Code:
    C number of elements in each array
    PARAMETER( SIZEX = 476 )
    C number of characters in each element of character array
    PARAMETER( SIZEY = 11 )
    C character array with 11 characters per element
    CHARACTER*11 NAMES( SIZEX )
    C array of real
    REAL VALUES( SIZEX )
    C yes, this part is in FORTRAN
    where a character is 8-bits and a real is 32-bits.

    This comes to (476*(11*8)) + (476*32) = 57120 bits or 7140 bytes.

    In my current bash code,
    Code:
    # total from above
    total_size=57120
    # integer math for the number of 4096 byte pages, round up
    num_pages=$(( ($total_size + (4095)) / 4096 ))
    # add one page just in case
    num_pages=$(($num_pages + 1))
    # num_pages = 15
    it looks like I am allocating way more memory than I need. It seems as if I should divide total_size by 8 to get the number of bytes and keep the rest the same, though I can probably dispense with the extra page.

    Code:
    # total from above
    total_size_bits=57120
    total_size_bytes=7140
    # integer math for the number of 4096 byte pages, round up
    num_pages=$(( ($total_size_bytes + (4095)) / 4096 ))
    # num_pages = 2
    Does this make sense?

    I am not sure why the original programmer chose to allocate in blocks of 4096 bytes. The size is probably not all that important in that in practice, it will never approach a value that will stress resources. It only needs to be big enough to avoid overflow, etc.

    LMHmedchem

  6. #6
    2kaud's Avatar
    2kaud is offline Super Moderator Power Poster
    Join Date
    Dec 2012
    Location
    England
    Posts
    6,877

    Re: size of allocated shared memory

    4096 (4k) could be the sector (or cluster?) size of the original disk that used this program - hence file memory was allocated in whole sectors. Back when, this could have given a performance enhancement.

    I suggest that you change the code to always work in bytes. Also, why bother with your bash code? You have to run it before you compile if you want to change values. Why not just #define SIZEX in a separate .h file (and any other compile time constants that may change between compilations) and include that first with the code? Then everything else is then calculated from that. So before you compile, just change that value in one place if you need to. Note that sizeof() will give the number of bytes used by the specified type at compile time (so in your case sizeof(float) will give 4), so there is no need to use constants of 8, 32 etc. Work in bytes and get the required number for a type using sizeof().

    Yes, based upon 476 elements, you only need 2 pages. Basically, each entry is 11 (char array) bytes + 4 (float) bytes = 15 bytes. You have 476 entries so 7140 bytes which gives 1.7 pages which when rounded up gives 2.
    All advice is offered in good faith only. All my code is tested (unless stated explicitly otherwise) with the latest version of Microsoft Visual Studio (using the supported features of the latest standard) and is offered as examples only - not as production quality. I cannot offer advice regarding any other c/c++ compiler/IDE or incompatibilities with VS. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/ and can be used without reference or acknowledgement. Also note that I only provide advice and guidance via the forums - and not via private messages!

    C++17 Compiler: Microsoft VS2019 (16.3.5)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width




On-Demand Webinars (sponsored)