When I'm building a Windows GUI app I usually set the exe's linker option /SUBSYSTEM:WINDOWS. If it's a console app I set /SUBSYSTEM:CONSOLE. So far, so good. BUT...
When I build a DLL (i.e. code only - no GUI) I usually just make sure it isn't set to SUBSYTEM:CONSOLE. Sometimes it can be /SUBSYSTEM:WINDOWS. Other times it can be Not set. I've never found it makes any difference - until today....
I've spent the whole morning chasing a weird problem where _close() was occasionally failing in Debug mode. The line that failed is in red:-
Basically, the file to be closed was deemed be not open (FOPEN flag not set). I could literally open a file and close it straight away and it would fail. I could step through _open() and validate that the flag was getting correctly set - but by the time I got to _close() it was corrupted. :(Code:int __cdecl _close (
int fh
)
{
int r; /* return value */
/* validate file handle */
_CHECK_FH_CLEAR_OSSERR_RETURN( fh, EBADF, -1 );
_VALIDATE_CLEAR_OSSERR_RETURN((fh >= 0 && (unsigned)fh < (unsigned)_nhandle), EBADF, -1);
_VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);
_lock_fh(fh); /* lock file */
__try {
if ( _osfile(fh) & FOPEN )
r = _close_nolock(fh);
else {
errno = EBADF;
r = -1;
_ASSERTE(("Invalid file descriptor. File possibly closed by a different thread",0));
}
}
__finally {
_unlock_fh(fh); /* unlock the file */
}
return r;
}
Eventually I found that one of my VC solutions had about 10 projects which all built DLLs. All were set to /SUBSYSTEM:CONSOLE except for one, which was set to Not set. Setting it the same as all the others seems to have fixed the problem :confused:
I loathe bugs like this because I don't understand them. Does this make sense to anyone here?
