Finding Web Service Intermittent Failures

Posted by acampbelle1 on 30-Aug-2012 11:04

We are load testing some Sonic Connect web services, and on very rare occasion we have failures (out of 5,000 at 100 milliseconds apart we have less than 50 failures). I don't see anything in the container logs for the container hosting the Sonic Connect service (even though it is difficult to see much because of all the traffic). Obviously the service is working since it has such a high success rate, it's just that on these rare occasions the client calling the web service receives a message that the connection was closed by the remote host (i.e. Sonic). Is it possible the web service endpoint is being overloaded to the point it temporarily denies requests? What possible settings can I change to possibly decrease the amount of these failures? Any ideas would be appreciated, thanks.

All Replies

Posted by wtam on 30-Aug-2012 11:50

Is your test running on Windows?  It sounds like some issue occured on the network stack (HTTP/TCP) that error was not propagated to Sonic.  We heard about this type of failure in Windows only but the exact cause is still unknown.  It could be a wide range of things such as network, machine resource, OS (which provides the TCP stack), Jetty which provides the HTTP stack.  If Sonic (Connect) is overloaded, I would expect errors in container log.  You will see a bunch of process timeout messages.  To diagnose it, it may mean to capture the TCP dump when the problem occurs in order to tell why the connection was closed prematurely.

Thanks.

Posted by acampbelle1 on 30-Aug-2012 12:03

Thank you William (and yes Sonic is hosted on a Windows server). Do you have any recommendations of a good tool to capture this traffic in a way that it can easily be analyzed? I have used Wireshark before, but with 5,000 transmissions, finding the failed transmissions seems near impossible unless I'm missing something. Thanks.

Posted by wtam on 30-Aug-2012 20:12

I know it is hard to capture meaningful tcp dump if we don't when it will fail.  You can submit a support ticket.

Thanks.

This thread is closed