Subsystem:Windows

[John E]

When I'm building a Windows GUI app I usually set the exe's linker option /SUBSYSTEM:WINDOWS. If it's a console app I set /SUBSYSTEM:CONSOLE. So far, so good. BUT...

When I build a DLL (i.e. code only - no GUI) I usually just make sure it isn't set to SUBSYTEM:CONSOLE. Sometimes it can be /SUBSYSTEM:WINDOWS. Other times it can be Not set. I've never found it makes any difference - until today....

I've spent the whole morning chasing a weird problem where _close() was occasionally failing in Debug mode. The line that failed is in red:-

int __cdecl _close (
        int fh
        )
{
        int r;                          /* return value */

        /* validate file handle */
        _CHECK_FH_CLEAR_OSSERR_RETURN( fh, EBADF, -1 );
        _VALIDATE_CLEAR_OSSERR_RETURN((fh >= 0 && (unsigned)fh < (unsigned)_nhandle), EBADF, -1);
        _VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);

        _lock_fh(fh);                   /* lock file */

        __try {
                if ( _osfile(fh) & FOPEN )
                        r = _close_nolock(fh);
                else {
                        errno = EBADF;
                        r = -1;
                        _ASSERTE(("Invalid file descriptor. File possibly closed by a different thread",0));
                }
        }
        __finally {
                _unlock_fh(fh);         /* unlock the file */
        }

        return r;
}

Basically, the file to be closed was deemed be not open (FOPEN flag not set). I could literally open a file and close it straight away and it would fail. I could step through _open() and validate that the flag was getting correctly set - but by the time I got to _close() it was corrupted.

Eventually I found that one of my VC solutions had about 10 projects which all built DLLs. All were set to /SUBSYSTEM:CONSOLE except for one, which was set to Not set. Setting it the same as all the others seems to have fixed the problem

I loathe bugs like this because I don't understand them. Does this make sense to anyone here?