COM base Logger (inproc)

[Deepak_Sharma]

hello,

i am working on the COM base logger (inproc server).
i want to load the COM DLL by various exes (com clients )and then call its logging API.

i want to do all logging into same file.

for this i have COM object which would return me the Logger interface.
inside the COM logger i am returning the same COM objects. if it is already created by

Result = CComObject<Logger>::CreateInstance( &g_pLogger );

i am guarding it with a static member variable like binitialized.

but static and global variables are mapped different for each process.
how should i model it so that i could use the same COM interface to write into same file

if ( !m_bInitialized )
	{
		Result = CComObject<Logger>::CreateInstance( &g_pLogger );
		if ( SUCCEEDED( Result ) )
		{
			g_Logger->AddRef();
			m_bInitialized = true;

		}
		else
		{
			g_pLogger = NULL;
			return E_UNEXPECTED;
		}
	}

regards
Deepak

[OReubens]

An out of process com server would seem to be ideal.

If that's not feasible, it's down to some form of inter process communication... Mailslots, pipes, shared memory, file mapping, DDE, global mutexes, sockets, RPC, ...

[Arjay]

You've got a couple of issues to solve:

1) Synchronizing access to the log file.
2) Passing data between processes.


If you use an inproc server, then you rely on each process that loads your dll to play nice (and appropriately release any file locks). Instead of approaching the problem from this direction, consider having a file 'manager' process that is responsible for logging the requests to the file - only the file manager has access to open, close and write to the log file. This makes synchronization to the file simpler because only one (file manager) process ever has access to the file.

With that in mind, the problem is how to pass data from the client processes to the file manager process. As previously mentioned, there are various interprocess communication techniques to choose from (named pipes, memory mapped files, etc.).

But....

As OReubens mentioned, the out of proc COM server is a great choice because it can serve as the file manager process (as an out of proc COM server is an exe) and it can use the COM sub system to solve the how to pass data between processes problem.

I'd probably make the out-of-proc COM logging server a windows service too. That way, it could be self starting (on Demand) and accessed from different desktops or even remotely. Inside the logging service, I would abstract the file handling functionality into a singleton class which contains a queue for incoming requests. I'd expose a couple of public for starting and stopping to handle the typical service operations and one or more Log methods. The log methods simply push the log request onto a queue and return. A secondary thread would service the log queue and put the requests into the file. The COM interface portion of the service would just take the incoming requests, and call the Log method on the singleton class. Of course the singleton class would have to be made thread safe.

If you need some sample code, check out the Win32 synchronization articles listed in my signature line. There's a LogSnd and LogRcv pair of applications that send logging information between two processes using memory mapped files. Now the out of proc COM server approach negates the need for using memory mapped files to send the data, but there are other ideas in the code that you might find useful like a thread safe queue and a secondary thread that services the queue requests.

Btw, I prefer to use a queue in logging cases because, I don't want the client applications to block while waiting for a file operation to occur. With the queue, the queue is locked, the log request is pushed onto the queue and the queue is unlocked. It's fast and the Log method returns practically immediately.

[Deepak_Sharma]

hello OReubens,

Initially I thought of the out of pro server. But I was not sure if that would create bottleneck situation when we are at verbose mode. So i thought that why not make it in proc server. But then I encountered this problem.

Arjay:
I have seen your articles and using them in my project. Especially the lock class, it is very usefully. Thanks for sharing it

As stated above, I was not clear that would it create any problem when n exes are writing to same COM Server (out of process).

First if say I want to have in-proc Logger. Then what should be the solution to this problem.

Here is what I think.
Each Process would load a Proxy COM DLL which has instance of another COM Logger. This instance would be singleton. I mean when proxy DLL would load it i would return the same instance. And since actual COM server would have only one instance. i need not worry about the access to log file.

What do you guys think? Does it make sense or am I missing something here?

Second if say i want to have COM server (out of proc ) assuming it would not create bottleneck. Then the solution you proposed making COM logger as a service looks perfectly fine to me. Now to pass logging data to this COM server, i need not worry inter-process communication, Since they would be just a COM calls.

I have three more methods based on you feedback.

How about creating a Named pipe inside the COM server (out of proc, service), And all the client application would log to this named pipe. Will it be a feasible solution? Or what could be its implications.

Second is using Socket programming. All exe would write to a socket at which COM server is listening. The good part with this approach is that I could host the COM server to any where be it intranet or internet. Of-course with COM as a service I could have such facility.

What do you guys think?

Third is one which I have not explored which is mail-slots and shared memory. I have to read it before coming to a conclusion.

Could you guys please guide me how could I benchmark these methods for things like scalability and speed as well.

regards
Deepak

[OReubens]

If you potentially have large volumes of data that need to be logged and this needs to be done FAST. Then you really need to consider separate logs for each process. You can add a timestamp in the log, and when needed, merge data sorted by timestamp in a log viewer or log merging tool. This makes actual logging faster at the expense of extra processing time when viewing/merging the logs.

An in process Com will need interprocess communication to synchronize across processes.
An out of process Com uses interprocess communication to pass function parameters.

Inter process communication and synchronising is typically "slow" by nature. 'Slow' is relative of course, it all depends on logging volume.

[Arjay]

First if say I want to have in-proc Logger. Then what should be the solution to this problem.

Here is what I think.
Each Process would load a Proxy COM DLL which has instance of another COM Logger. This instance would be singleton. I mean when proxy DLL would load it i would return the same instance. And since actual COM server would have only one instance. i need not worry about the access to log file.

If your in-proc COM dll calls another out-of-proc COM dll (that does the file writing), it will work fine. However, if your in-proc COM dll calls another in-proc dll, you'll have the same file synchronization issues (if all the client processes write to the same log file). When you strip off the COM layers, you are essentially left with multiple processes trying to write to the same file (and as a result you have a bigger chore of synchronizating access to the file from multiple processes over what you would have if you only accessed the file from a single 'log manager' process).

Second if say i want to have COM server (out of proc ) assuming it would not create bottleneck. Then the solution you proposed making COM logger as a service looks perfectly fine to me. Now to pass logging data to this COM server, i need not worry inter-process communication, Since they would be just a COM calls.

Yes. For all but very high logging activity from multiple processes, this should work fine, but you need to determine your expected logging throughput and test to make your you are achieving it. Remember to use a queue in the log manager service so that each log call is as fast as possible.

In terms of Log manager (COM exe server or COM within a Windows service), what I was proposing is to use a singleton instance. In general I always abstract the implementation from the COM service code (and delegate the COM method calls to a method in an 'implementation' class that performs the real work). Since you need a singleton class, this delegation is made to a thread safe singleton class. Something like.

STDMETHODIMP CLogger::Log(BSTR msg)
{
  // Route Log call to the singleton instance
  return LogManager.GetInstance( ).Log( msg );
}

Of course, I would pass some sort of logging structure with additional data rather than just the string.

Inside the LogManager.Log( ) call, you would just push the log request onto a thread safe queue. As mentioned before, the LogManager would have a secondary thread pop log requests off the queue and put them into the file.

I have three more methods based on you feedback.

How about creating a Named pipe inside the COM server (out of proc, service), And all the client application would log to this named pipe. Will it be a feasible solution? Or what could be its implications.

Second is using Socket programming. All exe would write to a socket at which COM server is listening. The good part with this approach is that I could host the COM server to any where be it intranet or internet. Of-course with COM as a service I could have such facility.

What do you guys think?

Third is one which I have not explored which is mail-slots and shared memory. I have to read it before coming to a conclusion.

If you go the COM route, you don't need any of these methods because COM becomes your interprocess communication.

Could you guys please guide me how could I benchmark these methods for things like scalability and speed as well.

Just create a test harness app, that launches x number of processes each performing y number of logs. Record the amount of time it takes to log and any other metrics you are interested in. Monitor the size of the Log Manager queue (run the Log service in debug mode and print out a dbg statement, and use DbgView to see the output). You'll want to make sure the queue doesn't keep growing.

What sort of log frequency were you expecting from the clients in total?

[Deepak_Sharma]

hello guys,

sorry for being late. i was assigned to some different critical project for some duration.
anyways i am back.
here is the update and what i have come so far.

i made a out of proc COM logger. thanks arjay!!
i have yet to test it for high volume logging. it works in the same way as before.
all clients would create instance of CLoggerFactory() COM object which would return a global COM Interface(CLogger) used by the clients. when clients log the Com API would just put the request into Q. which would be processed by a separate logging thread in the COM logger itself.

for a single client, having multiple threads, i need to pass this COM interface across threads
which ask for COM marshaling for each thread.

what would be the cleaner way of doing this. i mean utilizing single COM interface across the multiple threads. one way i think is to have a separate logging thread in each clients where all the threads would put request to a Q and COM object would be initialized on this separate thread so there would not need for Marshalling.

or is there any clearer approach?

what do you guys think?

regards

[Deepak_Sharma]

i have one more question that how you guys handle a situation like RPC_SERVER_UNAVAILABLE
i think we try to reinitialize the COM pointers vai COM calls with some count maintained if could not then just display the msg and exit.


regards