As macabre13 suggested, you need to implement global 'new' and 'delete' operators. They will allocate the memory (via malloc, for instance), and would keep track of number of bytes allocated. new-operator will add to global counter, and delete will decrease the global counter.

But for all this to work you also need bookkeeping of each allocation, since when 'delete' is called you need to identify how much to reduce from global counter. vector or map would help.

So, in new operator you can reject memory allocation if it is going to exceed the maximum.

Overloading 'new' and 'delete' may cause issues with other libraries. So you may need to provide custom functions (like AllocateMem, DeleteMem). These can be global functions, or implemented in a (singleton) class. Multithreading should also be taken care of, if application is MT.