How to make logging thread-safe

Posted by smahanta on 07-Dec-2009 22:58

We have a sonic distributed deployment where we are using log4j for application logging.Now say we have 3 containers containerA, containerB & containerC residing in same linux box logging to the same log file /home/subhendu/application.log.Now log from 3 containers are getting mixed up.We want all 3 containers to the same file. We can send the log messages to a general queue & another sonic service can read the log messages & log to application log.But my doubt is there will be too much performance penalty. Please suggest a remedy.

--Subhendu  

All Replies

Posted by Bill Wood on 08-Dec-2009 18:34

One option is to write your own log4j appender.  Another would be to write separate logs, and then combine them after the fact. (or also to run a second app that reads the files and then writes to the 'common log'.   You may want to check the forums on log4j to see other options -- the case you have is not too unusual, but the log4j's normal file appender sort of assumes you aren't sharing one file for different containers.

Posted by Bill Wood on 11-Dec-2009 19:17

One more comment --- Logging is 'thread-safe'.  What you are seeing is the specified behavior for logging in log4j.  Specifically, you are not seeing one application log block other writers -- each log is being added atomically.   What you are looking for seems to be more "transactional" logging.  I am guessing you do not want blocks of message from one writer to be interleaved in blocks from another.

Posted by smahanta on 11-Dec-2009 21:04

Yes, you are correct. I do not want blocks of message from one writer to be interleaved in blocks from another.At times half a sentence of log message is mixed with that with another.So thinking of using JMSAppender of log4j.If that also does not work we have to make different containers  to different files.Moreover, suppose we have a service - ServiceA. We make single instance of the service ServiceInstanceA. For improving performance we are deploying the same instance in Container1, Container2 & Container3. Now if 3 containers log to file1, file2, file3 it is difficult to get a end-to-end view of a message flow.

--Subhendu     

Posted by Bill Wood on 12-Dec-2009 08:32

Welcome to distributed computing.  This will be an issue if you combine logs as you are doing without adding some context.  If you want end-to-end tracking across multiple services and containers, it is a complex problem.  I'd look into Actional which has a huge engineering effort in working on this problem.

As you note, logs will only get you so far, and you have to manually figure out how to correlate events where itineraries are deployed across multiple contaienrs.  (NOTE that the 'common log' solution is actually one that is a little short sighted as it only handles the case where containers have access to the same log file, which would minimize the usefulness.

I have not used it, but I think one of the Sonic 7.6.x versions did allow the MF containers to centralize the log collections over JMS to a common collection point.  Check the doc on logging.

This thread is closed