Silent crash in Visual Studio 2008 c++ application

[kmsingh792]

Hi

We have a unmanaged C++ TCP server application running as a Windows service that is silently crashing after few days of run on Win2003 server. There is no drwtsn log file getting generated (no issue with drwtsn log as it catches other crashes in same application). Due to lack of drwtsn log file, we are unable to progress on how to debug this further..

The TCP server application is sort of HTTP processor. It connects to wide range of webserver and processes data.

Can someone please guide me what can be done to debug the silent crashes.. There are 1000+ users connected to this server at any instant and thus its not possible to run the same in debug mode. The crash is not reproducible and happens once in 5-10 days on one of the 6 servers..



Any tool that can help to debug these silent crashes. The application is a pure C++ application without any MFC or STL..



Thanks in Advance.

Krishna

[Alex F]

This may be result of memory and resource leaks. Using Task Manager, check whether memory usage, threads, handles and GDI objects count is constantly growing when this program runs.

[kmsingh792]

Hi

I have checked the memory, resource, handles etc.. All is good.. Silent crash most probably is due to stack corruption.. Same release is running on some machines for 20+ days and crashes on some machine randomly within few days. So our gut feeling is some particular website data is causing stack corruption in our processing resulting in silent crash..

thanks for your response.

regards
Krishna

[UnfitElf]

If thats your gut feeling start stripping it bare bit by bit, starting with the processing section.

i.e. still retrieve the information, handle the connections etc however remove the processing logic. Run it on the server(s), does it still crash? If not start adding the logic back in bit at a time, testing at each point.. If it does start checking over other sections you have a suspicion over with the same steps.

Its tedious, slow and painful but it will get you the result. :)

[kmsingh792]

Appreciate your help. The problem is we have two different kind of services. Both are used by users to access the internet. The same server build is running on one set of 100 servers without any issues for 4+ months now. But on the second set, we are getting problem that too once in 4-5 days on a server or so with no drwtsn logs.

The code is spread across might be 100+ classes with sloc of around 200,000 lines. For some website, something causes stack corruption like when parsing a webpage or javascript or a css.

I want to understand what ways we can debug a service on Windows.
http://msdn.microsoft.com/en-us/library/ff539106%28v=VS.85%29.aspx
gives few methods..

WinDbg on local machine is too heavy for such load. Does anyone has any idea whether remote debugging eases the load on the debuggee service machine ?. I am thinking of
""
Kernel-controlled user-mode debugging. A user-mode debugger running on the same computer as the service, being controlled from a kernel debugger on a second computer.
""
Has anyone done this earlier. Is this lighter than using Windbg locally for that service?.

thanks in advance
Krishna

[Paul McKenzie]

Hi

We have a unmanaged C++ TCP server application running as a Windows service that is silently crashing after few days of run on Win2003 server. There is no drwtsn log file getting generated (no issue with drwtsn log as it catches other crashes in same application). Due to lack of drwtsn log file, we are unable to progress on how to debug this further..]1) Learn how to remotely debug.

2) Once you know how to do 1), then all you need to do is attach the Visual Studio debugger on your development machine to the running process on the remote machine. Any process can be debugged by attaching to it (unless it has code in there to watch for debuggers trying to attach to it). When the crash occurs, the debugger will pop up, and then you can inspect what happens, i.e. inspect the call stack, look at the registers, etc. If you have the PDB and source created from the release build of your app, you can look at the source code that caused the crash to occur.

Also, when you say "slient crash", what do you mean by that? Do you get the message that an exception occurred in your application?
The application is a pure C++ application without any MFC or STL..So why did you post your question in the Managed C++ forum?

Secondly, you don't consider a C++ application that uses the C++ standard library (STL) "pure"? STL is part of C++, just as fopen(), fclose(), sqrt(), pow(), cin, cout, are part of C++ language and library. MFC is proprietary, so I understand that part of not being "pure", but certainly not STL..

Regards,

Paul McKenzie

[emailus]

1) Learn how to remotely debug.

2) Once you know how to do 1), then all you need to do is attach the Visual Studio debugger on your development machine to the running process on the remote machine. Any process can be debugged by attaching to it (unless it has code in there to watch for debuggers trying to attach to it). When the crash occurs, the debugger will pop up, and then you can inspect what happens, i.e. inspect the call stack, look at the registers, etc. If you have the PDB and source created from the release build of your app, you can look at the source code that caused the crash to occur.

Also, when you say "slient crash", what do you mean by that? Do you get the message that an exception occurred in your application?
So why did you post your question in the Managed C++ forum?

Secondly, you don't consider a C++ application that uses the C++ standard library (STL) "pure"? STL is part of C++, just as fopen(), fclose(), sqrt(), pow(), cin, cout, are part of C++ language and library. MFC is proprietary, so I understand that part of not being "pure", but certainly not STL..

Regards,

Paul McKenzie

Mr. Paul McKenzie I love your replies as well as rhetorical questions :hihi:

[Alex F]

You can try post-mortem debugging, which helps to solve most complicated and irregular crashes.
http://www.codeproject.com/KB/debug/postmortemdebug_standalone1.aspx
To use it, it is necessary to add minidump generating feature to the program. The question is, whether the program has chance to generate minidump during this silent crash, but it is impossible to say without trying first.

[kmsingh792]

Thanks for response Paul and Alex. I will try the codeproject.. The issue with this is I don't have physical access of the machines on which the server is crashing. I just access them using the network..
Can we do remote debugging on network.. I was seeing USB, serial interface etc but not the Ethernet..
I am in meantime trying to get physical access to the same..

I agree STL is pure. My fault..

Managed C++ and C++/CLI
-> I am new fellow to these forums and thus thoughts its for both as I interpreted it like "Managed C++" and "C++/CLI".

[kmsingh792]

I missed to mention by Silent crash i mean, there is no drwtnsn log, no pop-ups.. There is another application monitoring this server that starts it again after a minute or so of its crash.. So we just know the server has crashed from the traces.

[Alex F]

You don't need physical access to destination computer for post-mortem debugging. You only need to get minidump from this computer, after program crash. Debugging itself is done on developer computer. Again, if dump file was created...

[kmsingh792]

Alex

I guess I am unable to explain problem clearly. Main problem is that there is no dump. Had the dump been there, all would be a cakewalk. But without dump, I can't debug a silent crash..Next steps i am planning to do is
- Use different tools like ntsd, userdump to catch the crash as default debugger in AeDebug registry.
- Run the process under ntsd by ntsd -p <pid>
- Run the process in WinDbg by attaching windbg to the running process.

I heard such thing occurs due to double stack fault or stack corruption.. Anyone has any idea on these?