Bad news that it is impossible to define exactly what is Console and GUI appllication. GUI application may have Console window. Console application can show modal dialog. Application may be non-GUI and non-Console, for example, service or Windows application which doesn't show any window. Program may work with DirectX full-screen mode, in this case it doesn't have any window.
Regarding GetGuiResources - program can create GDI objects without GUI, for example, memory DC.
Assuming that GUI application is one that has at least one window:
Call EnumWindows,
For every window handle, call GetWindowThreadProcessId to detect whether it belongs to the process. If at least one window belongs to the process - this is possibly GUI application.
Assuming that GUI application is one that has at least one window:
Call EnumWindows,
For every window handle, call GetWindowThreadProcessId to detect whether it belongs to the process. If at least one window belongs to the process - this is possibly GUI application.
Will work for many, but not all.
As you said yourself, a Console application can have a messagebox, but it can also have any of the common dialogs, and an application can even start life as a console application, then make a full fledged GUI anyway.
There's no easy solution here, and in fact a proper answer is really dependant on what you want to know.
If you want to know how the application was built, you can use the processID to open a process handle, use that to get the process full filename, open this filename and read the PE header to check for the IMAGE_SUBSYSTEM_WINDOWS_GUI or IMAGE_SUBSYSTEM_WINDOWS_CUI flags in the optional header.
But you can make an exe having the GUI flag that never ever makes a window, and you can have a CUI one that does makes windows and displays a GUI. Without being more specific as to what you're after, there is no single right answer.
Bookmarks