Subsystem:Windows

[John E]

When I'm building a Windows GUI app I usually set the exe's linker option /SUBSYSTEM:WINDOWS. If it's a console app I set /SUBSYSTEM:CONSOLE. So far, so good. BUT...

When I build a DLL (i.e. code only - no GUI) I usually just make sure it isn't set to SUBSYTEM:CONSOLE. Sometimes it can be /SUBSYSTEM:WINDOWS. Other times it can be Not set. I've never found it makes any difference - until today....

I've spent the whole morning chasing a weird problem where _close() was occasionally failing in Debug mode. The line that failed is in red:-

int __cdecl _close (
        int fh
        )
{
        int r;                          /* return value */

        /* validate file handle */
        _CHECK_FH_CLEAR_OSSERR_RETURN( fh, EBADF, -1 );
        _VALIDATE_CLEAR_OSSERR_RETURN((fh >= 0 && (unsigned)fh < (unsigned)_nhandle), EBADF, -1);
        _VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);

        _lock_fh(fh);                   /* lock file */

        __try {
                if ( _osfile(fh) & FOPEN )
                        r = _close_nolock(fh);
                else {
                        errno = EBADF;
                        r = -1;
                        _ASSERTE(("Invalid file descriptor. File possibly closed by a different thread",0));
                }
        }
        __finally {
                _unlock_fh(fh);         /* unlock the file */
        }

        return r;
}

Basically, the file to be closed was deemed be not open (FOPEN flag not set). I could literally open a file and close it straight away and it would fail. I could step through _open() and validate that the flag was getting correctly set - but by the time I got to _close() it was corrupted.

Eventually I found that one of my VC solutions had about 10 projects which all built DLLs. All were set to /SUBSYSTEM:CONSOLE except for one, which was set to Not set. Setting it the same as all the others seems to have fixed the problem

I loathe bugs like this because I don't understand them. Does this make sense to anyone here?

[VladimirF]

...Does this make sense to anyone here?

No, not really. What is "/SUBSYSTEM:NONE" ?
I can't find this option in MSDN, and it is not listed in my VS 2010.
When I just type it in, I get an error:

LINK : fatal error LNK1117: syntax error in option 'SUBSYSTEM:NONE'

[John E]

No, not really. What is "/SUBSYSTEM:NONE" ?
I can't find this option in MSDN, and it is not listed in my VS 2010.
When I just type it in, I get an error:

Oops, I beg your pardon...

The actual selection was "Not set". I don't know where I got the idea about /SUBSYSTEM:NONE.

I'll re-edit my post because that's bound to confuse everyone else, too !

[BigEd781]

That just means "Figure it out for me depending on the entry point(s) which is present". From the docs:

CONSOLE
Win32 character-mode application. Console applications are given a console by the operating system. If main or wmain is defined, CONSOLE is the default.

WINDOWS
Application does not require a console, probably because it creates its own windows for interaction with the user. If WinMain or wWinMain is defined, WINDOWS is the default.

So, the default for a DLL should be SubSystem:Windows.

[John E]

Interesting... I wonder what happens in the case of a static lib (which could be used with either a console app or a GUI app)? Maybe that's what the "Not Set" option is for.

[S_M_A]

I guess that in that case it doesn't matter. Libs can be used for both targets

[Igor Vartanov]

I wonder what happens in the case of a static lib

Subsystem is a linker option. Static lib is built by librarian. No relation to lib.