CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 2 of 2 FirstFirst 12
Results 16 to 29 of 29
  1. #16
    Join Date
    Sep 2002
    Location
    Maryland - Fear The Turtle!
    Posts
    7,537
    Originally posted by dimm_coder


    I remember that, if it'll be something like showed by the next link, then nooo thanks :
    http://www.levenez.com/unix/
    I didn't open your link before I responded...silly me, but they do have dynix listed...now that's funny...what was the harris OS called? Or ICL's or tectronix (sp?) what was the precusor to Decnet called???

    /It was the early 90's after all....

  2. #17
    Join Date
    Sep 2002
    Location
    Belarus - Tirol, Austria
    Posts
    647
    Originally posted by Mick
    Ohh dimm you didn't just say that did you??? I kinda agree, to make commerical software, who cares...but for you knowledge...I care...are you a better coder because you know the internals of windoze??? yes or no???
    Oh noo.. don't say I got a bit of sh*t and so now I'm trying to giggle out of it
    Just I wanted to say: it's important to know API names, of course too, but in addition U can know what under its cover, but donot simply use it stupidly.
    "UNIX is simple; it just takes a genius to understand its simplicity!"

  3. #18
    Join Date
    May 2003
    Location
    Pakistan
    Posts
    223
    Originally posted by Mick
    Try not to insert MFC into the sockets picture...

    Try to understand the 'E' in economics...and you'll find linux isn't powerful.....nor is UN*X these days...
    ohhhhhhh thanx Mike for ure valueable suggestion, i forgot this forum is for Non Visual C++ issues.
    thanx a lot.

  4. #19
    Join Date
    May 2003
    Location
    Pakistan
    Posts
    223

    Coding is just the 13 % of overall project development.

    well well why u ppl r against MFC , no dought MFC socksts are full of bugs , but do u think Win32 API's dont have bugs in it.
    Whole Windows , even VC IDE is full of bugs. so there is no point or sense of superiority if one use MFC and other WIn32 API.
    There are thousand ways to solve out a problem , but what is the best one , it is upto you.
    I would say that i have always choosen the best design , and it is what software eniginnering all about, finding out each possible or alternate snenario , before u write even a single piece of code .

    Coding is just the 13 % of overall project development.

  5. #20
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626
    I seem to have missed this in my usual looking around here, and I just picked this up on the 'hot topics' :-) Allow me to share some of my experiences on this matter...

    On 'regular' Windows, each process you start, has at it's disposal a memoryspace of 2^32 bytes. Part of this memory is reserved for windows kernel code and data (Usually everything from 0x80000000 to 0xFFFFFFFF) giving you 2 GB of breathing room for your application.

    On Win2K server there is a setting which will change this slightly to give applications 3Gb of breathing room, leaving 'only' 1Gb for kernel code/data. Not all programs will run properly if you enable the setthing however. Look around on MSDN for more details on this matter.

    Now... Having 2Gb of adressable memory space doesn't mean you will be able to allocate that much memory. Since part of that memory will be occupied with your programs code (usually starting at 0x00400000), stack, and the code of DLL's you've loaded. Most of the time, the 2gb will be fragmented to some degree and this may prevent you from allocating large chunks of memory at once.
    It's entirely possible that while you have 1.5Gb of free memory, you won't be able to allocate a continuous chunk of say 1Gb, although you may be able to allocate 4 chunks of 256Mb. Especially for a server program that needs to run non-stop, this may end up being problematic if you do lots of allocates and frees. Fragmentation may end up preventing you from allocating memory, even though there's no leak, and there's enough 'free' memory. Just not continuous.
    In one service I wrote, I ended up having to write my own low level memory allocation to prevent fragmentation from causing too much problems.

    If you're allocating memory, you're also going to need either enough physical RAM or virtual RAM. Note that under WinNT/2000/XP, there is usually a limit to the effective pool. If your programs NEEDS large quantities of memory, you may have to change the settings. to allow for more virtual RAM.
    Find this under: Control Panel, System, Advanced, Performance, Advanced (Obviously for extra advanced users :-)), Virtual Memory.
    You should have the pool set sufficiently high to allow your program AND OTHER programs (and the Windows system itself) to run.
    Note that If you end up using huge amounts of virtual memory, you can end up with a VERY slow program because of lots of paging.
    If you're going to be needing lots of memory in your program, you're usually going to be wanting to have as much of that being actual RAM :-)

    There are ways around the 2Gb (or 3Gb) limit...:
    - Try using files for the memory, reading/writing the portions you need, when you need them. (basically, make your own 'virtual ram'/paging system). Quite often, if your program seems to need large amounts of RAM, it means you really need to use a database :-)
    - Spawn multiple processes which will do nothing more than allocate and maintain additional memory for the 'main' program, each process will have 2Gb memoryspace. You can use some form of Interprocess communications (mailslots, pipes, sockets, Read/WriteProcessMemory(), DDE, ...) from your main program to the helper programs to obtain the memory you need.
    - Switch to the new Win64. This gives processes a whopping 2^64 bits of address space (16 Exabyte, or 16 billion gigabytes), Again, half of that is reserved for the system, but even 8 exabyte should be more than enough. You're going to need a room full of RAM and/or harddisks to even HAVE that much in your machine ;-)
    Of course, Win64 and a Win64 capable machine don't come cheap...

    Each thread you create 'reserves' (not allocate/commit) a portion of memory for it's stack, the default is 1mb, but you can change the limit at the call to CreateThread(), After 2008 threads, each with 1mb of reserved stack, you will have more or less exhausted your 2Gb memory space, there is memory left, but no more continuous chunks of 1Mb.

    Making this many threads is generally a bad idea anyway (although it's easy), you'll be having a lot of overhead (thread synchronising, thread context switching). It's better to have a limited number of working threads (if you can, try 1 per actual CPU), and use an internal queue of of requests/jobs.

  6. #21
    Join Date
    May 2003
    Location
    Pakistan
    Posts
    223
    Originally posted by OReubens
    I seem to have missed this in my usual looking around here, and I just picked this up on the 'hot topics' :-) Allow me to share some of my experiences on this matter...

    On 'regular' Windows, each process you start, has at it's disposal a memoryspace of 2^32 bytes. Part of this memory is reserved for windows kernel code and data (Usually everything from 0x80000000 to 0xFFFFFFFF) giving you 2 GB of breathing room for your application.

    On Win2K server there is a setting which will change this slightly to give applications 3Gb of breathing room, leaving 'only' 1Gb for kernel code/data. Not all programs will run properly if you enable the setthing however. Look around on MSDN for more details on this matter.

    Now... Having 2Gb of adressable memory space doesn't mean you will be able to allocate that much memory. Since part of that memory will be occupied with your programs code (usually starting at 0x00400000), stack, and the code of DLL's you've loaded. Most of the time, the 2gb will be fragmented to some degree and this may prevent you from allocating large chunks of memory at once.
    It's entirely possible that while you have 1.5Gb of free memory, you won't be able to allocate a continuous chunk of say 1Gb, although you may be able to allocate 4 chunks of 256Mb. Especially for a server program that needs to run non-stop, this may end up being problematic if you do lots of allocates and frees. Fragmentation may end up preventing you from allocating memory, even though there's no leak, and there's enough 'free' memory. Just not continuous.
    In one service I wrote, I ended up having to write my own low level memory allocation to prevent fragmentation from causing too much problems.

    If you're allocating memory, you're also going to need either enough physical RAM or virtual RAM. Note that under WinNT/2000/XP, there is usually a limit to the effective pool. If your programs NEEDS large quantities of memory, you may have to change the settings. to allow for more virtual RAM.
    Find this under: Control Panel, System, Advanced, Performance, Advanced (Obviously for extra advanced users :-)), Virtual Memory.
    You should have the pool set sufficiently high to allow your program AND OTHER programs (and the Windows system itself) to run.
    Note that If you end up using huge amounts of virtual memory, you can end up with a VERY slow program because of lots of paging.
    If you're going to be needing lots of memory in your program, you're usually going to be wanting to have as much of that being actual RAM :-)

    There are ways around the 2Gb (or 3Gb) limit...:
    - Try using files for the memory, reading/writing the portions you need, when you need them. (basically, make your own 'virtual ram'/paging system). Quite often, if your program seems to need large amounts of RAM, it means you really need to use a database :-)
    - Spawn multiple processes which will do nothing more than allocate and maintain additional memory for the 'main' program, each process will have 2Gb memoryspace. You can use some form of Interprocess communications (mailslots, pipes, sockets, Read/WriteProcessMemory(), DDE, ...) from your main program to the helper programs to obtain the memory you need.
    - Switch to the new Win64. This gives processes a whopping 2^64 bits of address space (16 Exabyte, or 16 billion gigabytes), Again, half of that is reserved for the system, but even 8 exabyte should be more than enough. You're going to need a room full of RAM and/or harddisks to even HAVE that much in your machine ;-)
    Of course, Win64 and a Win64 capable machine don't come cheap...

    Each thread you create 'reserves' (not allocate/commit) a portion of memory for it's stack, the default is 1mb, but you can change the limit at the call to CreateThread(), After 2008 threads, each with 1mb of reserved stack, you will have more or less exhausted your 2Gb memory space, there is memory left, but no more continuous chunks of 1Mb.

    Making this many threads is generally a bad idea anyway (although it's easy), you'll be having a lot of overhead (thread synchronising, thread context switching). It's better to have a limited number of working threads (if you can, try 1 per actual CPU), and use an internal queue of of requests/jobs.

    Thanks a lot ,this really helped me understanding the scenario.

    As you mentioned overhead invoved in creating thread for each user.......I used this technique in Instant Messaaging server , It is working quite fine , but certainly with the increasing user , it will become overhead ....

    cauz software is released so it is not posible to change it now , so Can I optimize this design may be using Sleep , SuspendThread, ResumeThread techniques , so that if a client is not communicating for some time , its thread goes into suspended mode , May be this technique could optimize things
    Any other Idea

    Atif Mushtaq

  7. #22
    Join Date
    Sep 2002
    Location
    Belarus - Tirol, Austria
    Posts
    647
    Originally posted by atif_ilm
    cauz software is released so it is not posible to change it now , so Can I optimize this design may be using Sleep , SuspendThread, ResumeThread techniques , so that if a client is not communicating for some time , its thread goes into suspended mode , May be this technique could optimize things
    Any other Idea
    But what your threads do now when client is not communicating for some time. I guess, them must be blocked by some blocked call (select, recv, ...) and them do not use any CPU time. So there is no any special additional reason for using Suspend/Resume only for that purpose. And its using cannot resolve a problem with memory.
    May be I lost something here, but can U give a more complecated description of your project and purposes it wants to gain.
    "UNIX is simple; it just takes a genius to understand its simplicity!"

  8. #23
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626
    Well, I was really talking about WORKER threads. Communication threads are something of another matter. You can't as easily queue what will be arriving on the sockets One thread per socket is pretty much the only way out there. Try to restrict the tasks of those threads to actually receiving data however, and have (as few as possible) worker threads actually do something with data that has been received.

    Windows allocates 1Mb stack by default for each thread, but in my personal experience, I've found this to be much much more than my programs ever need if you don't have recursion, no large buffers placed on the stack, and no extremely deep nesting of function calls, you can make do with significantly less than 1Mb. The comm. threads in one service I made have only 32Kb stack, and even that is more than what they actually need.

  9. #24
    Join Date
    Sep 2002
    Location
    Belarus - Tirol, Austria
    Posts
    647
    Originally posted by OReubens
    Well, I was really talking about WORKER threads. Communication threads are something of another matter. You can't as easily queue what will be arriving on the sockets One thread per socket is pretty much the only way out there. Try to restrict the tasks of those threads to actually receiving data however, and have (as few as possible) worker threads actually do something with data that has been received.
    Yes, agree. As I've mentioned above, it is a good idea to have a one read thread to get requests and a pool of worker threads (number of threads can be increased/decreased depends on the number of requests) to handle thouse requests.

    Windows allocates 1Mb stack by default for each thread, but in my personal experience, I've found this to be much much more than my programs ever need if you don't have recursion, no large buffers placed on the stack, and no extremely deep nesting of function calls, you can make do with significantly less than 1Mb. The comm. threads in one service I made have only 32Kb stack, and even that is more than what they actually need.
    Yes, had the same situation with Linux Pthreads. A thread there has 1-2 Mb of reserved vm for stack (depends on kernel version), so I ussualy decrease it.
    "UNIX is simple; it just takes a genius to understand its simplicity!"

  10. #25
    Join Date
    May 2003
    Location
    Pakistan
    Posts
    223
    Originally posted by dimm_coder
    But what your threads do now when client is not communicating for some time. I guess, them must be blocked by some blocked call (select, recv, ...) and them do not use any CPU time. So there is no any special additional reason for using Suspend/Resume only for that purpose. And its using cannot resolve a problem with memory.
    May be I lost something here, but can U give a more complecated description of your project and purposes it wants to gain.
    Actually i wanted to prevent CPU thread switching time , say there are 2000 client connected to my server means 2000 threads.so OS will have to switch among these 2000 threads , they all will be getting there time slice. so most of CPU time will be wasted on this.

    As far as my project concerns , it is a messenger service , just like

    MSN or Yahoo Messenger .

    And I was involved in the development of its server , it is released
    and working fine for quite larger users.

    But i am now worried , if user extends to thousands , what will happen , we are using 64 Bit OS .That is why i asked if we use 20 -40 GB of physical RM and processor in GHZ .and we continue increase this hardware with new increasing users , will it be ok.



    Originally posted by OReubensWindows allocates 1Mb stack by default for each thread, but in my personal experience, I've found this to be much much more than my programs ever need if you don't have recursion, no large buffers placed on the stack, and no extremely deep nesting of function calls, you can make do with significantly less than 1Mb. The comm. threads in one service I made have only 32Kb stack, and even that is more than what they actually need.
    yehh its a good idea to limit stack size i think 200 k stack would be large enough for my purpose.

  11. #26
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626
    But i am now worried , if user extends to thousands , what will happen , we are using 64 Bit OS .That is why i asked if we use 20 -40 GB of physical RM and processor in GHZ .and we continue increase this hardware with new increasing users , will it be ok.
    If you've written your program as a Win32 program, but running it on Win64, it will still be a program restricted to the limitations of a Win32 program (2Gb memoryspace), to make advantage of the 8ExaByte memory space thinghy, you need to compile your program as a native 64bit program. If your program is 32bit now, you may have to make alterations to the code in order to port it to win64.

  12. #27
    Join Date
    Sep 2002
    Location
    Belarus - Tirol, Austria
    Posts
    647
    Originally posted by atif_ilm
    Actually i wanted to prevent CPU thread switching time , say there are 2000 client connected to my server means 2000 threads.so OS will have to switch among these 2000 threads , they all will be getting there time slice. so most of CPU time will be wasted on this.
    Hum... Of xourse, I donot know your project internals, but it is a possible decision to have a thread wich handle some clients (not a one). So U can have a set of worker threads and each of them can handle requests of some clients. That's first.
    About the memory limit per process. Like it has been mentioned above, a common decision is to have some processes for your task (like I 've described for *NIX fork()). Every process has own worker threads (each thread handle some clients). Here U'll have some virtual process spaces, so U'll be not boundared by limitation of one process virtual space.
    And that is a right decision for such kind of a project.
    In additional, U should have a power server (some CPU), or even better to have some power servers.
    "UNIX is simple; it just takes a genius to understand its simplicity!"

  13. #28
    Join Date
    May 2003
    Location
    Pakistan
    Posts
    223
    Originally posted by OReubens
    If you've written your program as a Win32 program, but running it on Win64, it will still be a program restricted to the limitations of a Win32 program (2Gb memoryspace), to make advantage of the 8ExaByte memory space thinghy, you need to compile your program as a native 64bit program. If your program is 32bit now, you may have to make alterations to the code in order to port it to win64.
    What kind of modification i will have to do to move from win32 ro win64 , can VC7 IDE do it autometically.

  14. #29
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626
    For starters, you need to have a compiler that can target Win64.

    For VS.NET, this means you need to install this Win64 support as an add-on. Depending on what version/edition of VS.NET you've purchase, this may or may not already be in your posession. If it's installed, then you can select 'Win64' as platform in the solution configuration of your project, if it's not installed, then you will have only Win32 as possible platform.

    Just recompiling probably won't do either. You will have to make some modifications to the code... While you can use a single codebase to target both Win32 and Win64, it does require you to follow certain guidelines to make this possible. How easy the port will be depends on how 'clean' your code is.

    The most important thing about Win64 is that all pointers are 64bit. If you've used pointer arithmatic, or stored pointers in non-pointer types (int, DWORD, LONG), this may need some rewrites
    Last edited by OReubens; October 31st, 2003 at 07:47 AM.

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured