Problem with AllocConsole and child processes Windows 7

[Rich C]

I have a GUI window that allocates a console using AllocConsole and then executes a child console process which should write to the console window I allocated. When I execute the program from the Start/Run menu or from a command prompt window,it works as expected. If I execute the program via the Windows Explorer or a desktop shortcut, the console window appears but nothing is written to it. I am using Windows 7 64bit. This problem did not occur under Windows XP.

I am setting the bInheritHandles to TRUE, setting STARTF_USESTDHANDLES flag in the startupifno dwFlags, and am setting hStdOutput to GetStdHandle (STD_OUTPUT_HANDLE).

[Igor Vartanov]

It would be good to see your code that replicates the problem, compilable and runnable.

[Rich C]

It would be good to see your code that replicates the problem, compilable and runnable.

Hope this is enough for you.

int execute_command(char *command)
{
HANDLE outp;
COORD coord;
STARTUPINFO si;
PROCESS_INFORMATION pi;
DWORD exitcode;
BOOL ret;

if(AllocConsole())
{
outp = GetStdHandle(STD_OUTPUT_HANDLE);
coord.X = 80;
coord.Y = 500;
SetConsoleScreenBufferSize(outp,coord);
SetConsoleTitle("Text Output");
}

GetStartupInfo(&si);
memset(&pi,0,sizeof(PROCESS_INFORMATION));
si.hStdInput = GetStdHandle (STD_INPUT_HANDLE);
si.hStdOutput = GetStdHandle (STD_OUTPUT_HANDLE);
si.hStdError = GetStdHandle (STD_ERROR_HANDLE);
si.dwFlags |= STARTF_USESTDHANDLES;

ret = CreateProcess(NULL,command,NULL,NULL,TRUE,0,NULL,NULL,&si,&pi);
if (ret != TRUE)
return(-1);

exitcode = STILL_ACTIVE;
if (pi.hProcess)
{
MSG msg;
ret = WAIT_TIMEOUT;
while (ret == WAIT_TIMEOUT || exitcode == STILL_ACTIVE)
{
ret = WaitForSingleObject(pi.hProcess,10);
if (!GetExitCodeProcess(pi.hProcess,&exitcode))
MessageBox(NULL, "GetExitCodeProcess Failure", NULL, MB_OK);
if (PeekMessage(&msg,NULL,0,0,PM_REMOVE))
DispatchMessage(&msg);
}
}
if (pi.hProcess)
CloseHandle(pi.hProcess);
if (pi.hThread)
CloseHandle(pi.hThread);

return(exitcode);
}

[Igor Vartanov]

This worked fine in my tests, Win XP and 7 64-bit:

int execute_command(char *command)
{
	HANDLE outp;
	COORD coord;
	STARTUPINFO si = {sizeof(si)};
	PROCESS_INFORMATION pi = {0};
	DWORD exitcode;
	BOOL ret;

	if(AllocConsole())
	{
		outp = GetStdHandle(STD_OUTPUT_HANDLE);
		coord.X = 80;
		coord.Y = 500;
		SetConsoleScreenBufferSize(outp,coord);
		SetConsoleTitle("Text Output");
	}

	ret = CreateProcess(NULL,command,NULL,NULL,TRUE,0,NULL,NULL,&si,&pi);
	if (ret != TRUE)
		return(-1);

	exitcode = STILL_ACTIVE;
	if (pi.hProcess)
	{
		MSG msg;
		ret = WAIT_TIMEOUT;
		while (ret == WAIT_TIMEOUT || exitcode == STILL_ACTIVE)
		{
			ret = WaitForSingleObject(pi.hProcess,10);
			if (!GetExitCodeProcess(pi.hProcess,&exitcode))
				MessageBox(NULL, "GetExitCodeProcess Failure", NULL, MB_OK);
			if (PeekMessage(&msg,NULL,0,0,PM_REMOVE))
				DispatchMessage(&msg);
		}
	}
	if (pi.hProcess)
		CloseHandle(pi.hProcess);
	if (pi.hThread)
		CloseHandle(pi.hThread);

	return(exitcode);
}

[Rich C]

Did your child process write to stdout and the output was shown in the console window?
Did you try to run your program by double-clicking on the executable file in the windows explorer or running it from a desktop shortcut that has as its target the executable file?

[Igor Vartanov]

Did your child process write to stdout and the output was shown in the console window?
Did you try to run your program by double-clicking on the executable file in the windows explorer or running it from a desktop shortcut that has as its target the executable file?

Rich, why don't you just try my code out yourself?

[Rich C]

Thanks Igor. Your code worked. When I first saw the code, I thought you were just repeating my code.

The difference is in STARTUPINFO. You essentially zeroed it out (except for the size); I used GetStartupInfo and then set some fields in it. Looking at my code, do you know what fields I set incorrectly?

Thanks again for the help.

[Igor Vartanov]

I'm pretty sure that there's something with proper handles duplication/security/inheritance, as your code gave me an 'Invalid handle' message been launched from explorer. (Sorry, I don't remember the details anymore, as I delved into this too many years ago. When I have some time for this, I'll take a more intent look.)

When you provide no explicit startup info, Windows itself does that for you, and of course it does that proper way.

[Rich C]

What's interesting to me is that this worked in Windows XP and prior versions of Windows.

Thanks again,
Rich

[Paul McKenzie]

What's interesting to me is that this worked in Windows XP and prior versions of Windows.

There are a lot of code examples that work with older OSes but will not work or work erratically as soon as you go to a later operating system. Most of this due to programmer fault, and the other OSes just happen to not exercise the bad code. Other times it is documented that the OS handles things differently, but the programmer didn't do their research thoroughly.

Regards,

Paul McKenzie

[Rich C]

I have a feeling its the misuse of the GetStartupInfo function when spawing child processes.

Rich

[Paul McKenzie]

I have a feeling its the misuse of the GetStartupInfo function when spawing child processes.

Rich

If you read the documentation for the STARTUPINFO struct, you see the first member has to have the number of bytes set (cb). Your example didn't do this (next time, use code tags when posting code), but Igor's does.

All the code I've ever written that uses STARTUPINFO has set the initial member to the number of bytes that the struct takes up. This goes back to Windows NT (maybe before that). So in reality, you were getting away with this mistake until now.

Regards,

Paul McKenzie

[2kaud]

Is the problem with CreateProcess? Is the console being created OK but CreateProcess is failing? When you use exeute_command in your program, do you test the return value for -1 and display an error message box or something as you don't show how you are calling execute_command? What commands are you passing to execute_command? What is the value of GetLastError() if the CreateProcess function fails?

For every command you want executed, you are creating and destroying a new process. This takes time. Why not just create a new process once with a console window and have it execute cmd to get a command prompt window?