Monitoring message flow

Posted by habin123 on 12-Jan-2011 07:52

Hi All,

            I have requirement to develop something by which I should be able to see the complete end to end message flow of my transaction.

            I could be able to know, message is flowing from which all services and processes.something which we can observe by enabling tracking level 2.

         1.  There are multiple process be situated in different containers.

         2. I can't enable tracking due to performance issue.

I had an idea to place logging after every service in the process itinerary, but the application teams will not allow me to change their process.I beleive its also expensive.Therefore I dropped.

Does there exists any APIs provided by sonic, any thrid party tool (not actional) or something or any approch to accomplish this ?

Thanks for reading this problem

- Habin    

All Replies

Posted by sedge on 13-Jan-2011 14:18

We have a similar requirement on a new development. The approach we are planning it to use Tracking and a process to monitor the Tracking End Point to capture the results to a database. I would be happy for:

- Feedback on the viability of this approach

- Suggestions for alternate approaches

- Any code that people may be willing to share

Habin:

- whatever you do is going to have a performance impact. It is just impossible to get something for nothing. I'd be interested in what metrrics you have to suggest that the impact of the tracking approach is unacceptable.

- You could try turninng Tracking on and write the Queue/Topic monitor in Java or .NET running on another computer.

Regards

Steve  

Posted by rrudis on 13-Jan-2011 14:32

Can you provide more insight into why a 3rd party tool would be an acceptable solution but Actional would not be acceptable?

Posted by habin123 on 14-Jan-2011 01:39

Stephen,

                  I also thought about the approach of storing the tracking message in the DB and generating reports based on that data.

                  The one concern here is, there will be a large increase in the traffic to broker.How this will impact the broker performance ?

                  whether such increased traffic will make the broker slow or down ?

Ours is an environment with 190 processes.140 process are having multiple services (say 4) in it.With tracking level 2,

2 * 190 (process entry/exit) + 4*2*140 (service entry/exit)  = 1500 extra messages

I'm concerned how this extra traffic will impact my broker.

Please share your suggestions.

Ronald,

                I understand the Actional will be a good solution for all the monitoring problem.I may not be getting a green signal for a

               lisence due to to the cost involved . Therefore, I'm looking for some freeware which can be utilized here and develop something in house.

Thanks,

Habin

Posted by sedge on 16-Jan-2011 23:30

Hi Habin

As I understand the logging process you only have to set the logging level for each entry point Process, effectively each on ramp. It looks to me like lower level Processes will adopt the logging level from above. What we haven't looked at yet is ways to downgrade the priority of the process that reads the tracking data.

One of our early objectives will be to collect some throughput metrics:

- With no tracking

- With Tracking turned on

- With Tracking on and the monitor process to write the database active.

I was hoping you might have already done that, rather than worrying (without evidence?) that it will slow the server down.

Even to use Actional you have to turn the Actional tooling on. I have no measurements for that either. Any monitoring tool has to get it's data from somewhere. Our deployment model and the prevents us from relying on Actinal at all sites. We may use it in some places as a "monitoing console" but not for message tracking.

For the time being we are progressing down the path of dumping the Tracking data into a database. Unless we find the cost of that exorbitant we'll just factor it into the server sizing. We have the luxury that it is a new development and not being retro-fitted. Did your designers forget to include message tracking and assume everything would run perfectly?

For you it may come down to the cost/benefit of Actional as opposed to the cost/benefit of developing something home grown.

Concerning the impact on your Broker....

- Depends more on the message throughput, rather than the number of processes.

- As mentioned im my earlier post you could offload the actual Queue/Topic extract to DB  to another server. You would probably at least put the database itself on another server.

- Look for ways to downgrade the priority of the Process/Service that is doing the database work so that it is only happening in "spare" time. 

Regards

Steve

Posted by mnair on 17-Jan-2011 22:01

Hi Habin,

Another option for you to consider is to make use of Sonic interceptors that we introduced in ESB 8.0. Chapter 12 in the Sonic ESB Developers guide has details on using this feature. This feature was primarily introduced for augmenting the fault handling behavior provided by service types so I am not sure if its well suited for generating tracking information, esp. since you need this for each step in the process. But if you are considering options this might be worth exploring.

You did mention non-technical reasons for not using Actional. But from a technical standpoint I would like to reiterate that enabling Actional instrumentation in Sonic seems like the best solution for your tracking/monitoring needs.

Thanks

-Mahesh

Posted by habin123 on 02-Feb-2011 02:09

Thanks a lot everybody for reading and posting your suggestions.

Finally we have decided to accomplish this by using AOP.The solution is that we will create an aspect and deploy in the container.This aspect will crosscut the service() method in each service.So, I'll be able to trace complete message flow with negligible performace hit.

Thanks,

Habin

Posted by sedge on 06-Feb-2011 03:31

Hi Habin

The terminology you used didn't have any meaning to me. Can you elaborate?

What is AOP? What is an aspect? What does "...crosscut the service() method..." mean?

How do you know that the performance hit that the performance hit will be negligable? How do you know that it will perform better than traching which, at first sight, appears a lot simpler to implement?

Since we're going down  a similar path for a greenfield development it is woth me investigating all options.

Thanks

Steve

Posted by habin123 on 07-Feb-2011 01:09

Steve,
         AOP stands for Aspect-oriented programming.You can have a look at wiki @ http://en.wikipedia.org/wiki/Aspect-oriented_programming.

The basic idea here is that, every sonic service contains a service() method.Using AOP we can trace when the service method is invoked and thus,
we can trace the message flow.We can define what actions(log or db update) we need to perform when the service() method is invoked for a particular service.

An aspect is just like a java class.In an aspect you have to define what all action you need to perform when a particular point is reached in
a main program flow.We are using AspectJ (http://www.eclipse.org/aspectj/doc/released/progguide/starting.html) to implement AOP.

what you need to do :
1. Create an aspect using aspectj such that it can crosscut the service() method.(Make you aware when service method is executed.)
2. Define the action you need to perform(say db update of message) when service() is called.
3. Put the aspect jar in the container class path.

Now you will be able to trace the complete flow of messages without touching any existing code or hitting the broker.

Hope this is useful.In case of any doubt, please reply back.

Habin

This thread is closed