Lately my local AppServers have been failing to close - intermittently. Progress 11.2.1 on Win7.
The following is in the Admin Server logs:
[07/08/15 16:11:11] [3] [AdminServer] * AdminServer GUI Plugin internal error 2: SvcControlCmd.pingService(Live) failure: java.rmi.ConnectException: Connection refused to host: 192.168.125.171; nested exception is: [07/08/15 16:11:11] [3] [AdminServer] * java.net.ConnectException: Connection timed out: connect (8172)(java.rmi.ConnectException: Connection refused to host: 192.168.125.171; nested exception is: [07/08/15 16:11:11] [3] [AdminServer] * java.net.ConnectException: Connection timed out: connect) [07/08/15 16:11:11] [3] [AdminServer] Cannot stop a service process(UBroker.AS.Live) that is not running! (8175) [07/08/15 16:11:12] [3] [Security] jpalmer:N:No Group Checking: User is authenticated and authorized (9898) [07/08/15 16:11:12] [2] [AdminServer] Tool added: com.progress.ubroker.tools.UBRemoteCommand@168d2e7 (7415) [07/08/15 16:11:33] [3] [AdminServer] * AdminServer GUI Plugin internal error 2: SvcControlCmd.pingService(LiveBatch) failure: java.rmi.ConnectException: Connection refused to host: 192.168.125.171; nested exception is: [07/08/15 16:11:33] [3] [AdminServer] * java.net.ConnectException: Connection timed out: connect (8172)(java.rmi.ConnectException: Connection refused to host: 192.168.125.171; nested exception is: [07/08/15 16:11:33] [3] [AdminServer] * java.net.ConnectException: Connection timed out: connect) [07/08/15 16:11:33] [3] [AdminServer] Cannot stop a service process(UBroker.AS.LiveBatch) that is not running! (8175)
When shutting the database down it says it's still in use, and then I get a crash of the process with a protrace.
===================================================== PROGRESS stack trace as of Fri Aug 07 16:13:32 2015 ===================================================== Progress OpenEdge Release 11.2 build 1236 SP01 on WINNT Startup parameters: -pf C:\Progress\OpenEdge\startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,-debugalert,-Mm 4096,-h 10,(end .pf),-classpath C:\Progress\OpenEdge\jdk\lib\tools.jar;C:\Progress\OpenEdge\java\progress.jar;C:\Progress\OpenEdge\java\messages.jar;C:\Progress\OpenEdge\java\prosp.jar,-pf C:\Users\jpalmer\Copy\DBStart\icmasliv.pf,-db E:\Database\Live\icmasliv.db,-prefetchFactor 100,-B 50000,-minport 3000,-maxport 5000,-lruskips 50,-B2 600,-Bpmax 64,-Mxs 25,-L 8192,-hash 13063,-spin 6214,-shmsegsize 512,-bibufs 20,-aibufs 20,-Ma 5,-Mn 6,-Mi 1,-n 50,-Mm 8192,-Mpb 5,-tablerangesize 620,-indexrangesize 2020,-omsize 3000,-S 8000,(end .pf) Exception code: C0000005 ACCESS_VIOLATION Fault address: 00E5D981 01:0012C981 C:\PROGRESS\OpenEdge\bin\_mprosrv.exe Registers: EAX:FFFFFFFF EBX:00000000 ECX:003D8B48 EDX:028F0420 ESI:00FB18E0 EDI:028F2D78 CS:EIP:0023:00E5D981 SS:ESP:002B:0023F74C EBP:003DA6D8 DS:002B ES:002B FS:0053 GS:002B Flags:00210286 Debugging dll: C:\PROGRESS\OpenEdge\bin\DBGHELP.DLL Symbol Path: C:\PROGRESS\OpenEdge\bin;C:\Progress\OpenEdge\pdbfiles Call Stack: Address Frame 00E5D981 003DA6D8 rcUpdateFieldMap+CDD1 028F2D78 003DA6DC 0000:00000000
Before shutting down, are you able to query the server using asbman and are the results what is expected?
Dave
Use mergeprop -validate to check out your ubroker.properties and possibly fix/restore from backup... there are some random bugs that can mess up the ubroker.properties file from time to time.
Also make sure all of your environment variables are correct (proenv) and that all of the paths in ubroker.properties are valid.
Thanks David. Unfortunately as it's intermittent I don't know whether or not it's going to fail or not until it does so, but I suppose I can build a query of the server into my shutdown process. I'll report back my findings.
Thanks MadDBA - that's a good call. I'll check it next time I'm on my home PC and report back.
[quote user="David Cleary"]
Before shutting down, are you able to query the server using asbman and are the results what is expected?
Dave
[/quote]
Hi [mention:fa9b41ed83d848b09612bd8983e98013:e9ed411860ed4f2ba0265705b8793d05] I've just tried a query of my appserver:
C:\OpenEdge\WRK>asbman -name Live -query
OpenEdge Release 11.2.1 as of Mon Apr 29 19:26:32 EDT 2013
Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280)
Searching for Live (8288)
Connecting to Live (8276)
Live is not responding (12076)
From the admin server logs:
[14/08/15 16:22:01] [3] [STDERR] java.rmi.ConnectException: Connection refused to host: 192.168.125.161; nested exception is: [14/08/15 16:22:01] [3] [STDERR] java.net.ConnectException: Connection timed out: connect [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110) [14/08/15 16:22:01] [3] [STDERR] at com.progress.ubroker.broker.ubListenerThread_Stub.getStatusFormatted(Unknown Source) [14/08/15 16:22:01] [3] [STDERR] at com.progress.ubroker.tools.UBRemoteCommand.runIt(UBRemoteCommand.java:682) [14/08/15 16:22:01] [3] [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [14/08/15 16:22:01] [3] [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [14/08/15 16:22:01] [3] [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [14/08/15 16:22:01] [3] [STDERR] at java.lang.reflect.Method.invoke(Method.java:597) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.Transport$1.run(Transport.java:159) [14/08/15 16:22:01] [3] [STDERR] at java.security.AccessController.doPrivileged(Native Method) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.Transport.serviceCall(Transport.java:155) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) [14/08/15 16:22:01] [3] [STDERR] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [14/08/15 16:22:01] [3] [STDERR] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [14/08/15 16:22:01] [3] [STDERR] at java.lang.Thread.run(Thread.java:662) [14/08/15 16:22:01] [3] [STDERR] Caused by: java.net.ConnectException: Connection timed out: connect [14/08/15 16:22:01] [3] [STDERR] at java.net.PlainSocketImpl.socketConnect(Native Method) [14/08/15 16:22:01] [3] [STDERR] at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) [14/08/15 16:22:01] [3] [STDERR] at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) [14/08/15 16:22:01] [3] [STDERR] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) [14/08/15 16:22:01] [3] [STDERR] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [14/08/15 16:22:01] [3] [STDERR] at java.net.Socket.connect(Socket.java:529) [14/08/15 16:22:01] [3] [STDERR] at java.net.Socket.connect(Socket.java:478) [14/08/15 16:22:01] [3] [STDERR] at java.net.Socket.<init>(Socket.java:375) [14/08/15 16:22:01] [3] [STDERR] at java.net.Socket.<init>(Socket.java:189) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) [14/08/15 16:22:01] [3] [STDERR] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) [14/08/15 16:22:01] [3] [STDERR] ... 19 more
Nothing in the broker and server logs.
[quote user="TheMadDBA"]
Use mergeprop -validate to check out your ubroker.properties and possibly fix/restore from backup... there are some random bugs that can mess up the ubroker.properties file from time to time.
Also make sure all of your environment variables are correct (proenv) and that all of the paths in ubroker.properties are valid.
[/quote]
Struggling to come up with a command line for mergeprop that will validate. It's not the most user friendly of commands :(
Syntax for mergeprop... knowledgebase.progress.com/.../P116259
mergeprop -delta ubroker.properties -validate -type ubroker -nobackup
I assume the users are still connecting and disconnecting from the appserver even though you can't query it? What does the output from nsman show?
Syntax for mergeprop... knowledgebase.progress.com/.../P116259
mergeprop -delta ubroker.properties -validate -type ubroker -nobackup
I assume the users are still connecting and disconnecting from the appserver even though you can't query it? What does the output from nsman show?
Flag this post as spam/abuse.
Hi James,
it looks like your AppServer broker is becoming unresponsive. This could happen for several reasons but the most common is running out resources, memory and/or handles.
First are there any errrors in the Live.broker.log file? Live.server.log file?
Next can you verify the AppServer broker process is still running when you try to shut it down and you get the ConnectionRefused error?
Then can you try getting a java dump of the AppServer Broker? You will need to use jstack.
for usage of jstack, type this in a PROENV window
%DLC%\jdk\bin\jstack
most commonly its this command in a PROENV window
%DLC%\jdk\bin\jstack -l PID
Let us know what you find,
Roy
Thanks [mention:30c5058a319047d288fe606ee12c9bbe:e9ed411860ed4f2ba0265705b8793d05] - no errors in the broker/server log files that I can see. I'll get the jstack next time it happens.
[mention:30c5058a319047d288fe606ee12c9bbe:e9ed411860ed4f2ba0265705b8793d05], which PID am I meant to be using? I'm using the _proapsv.exe one and getting
C:\OpenEdge\WRK>%DLC%\jdk\bin\jstack -l 4628
4628: Not enough storage is available to process this command
I'm running proenv as administrator.
Hi James,
you need to run that against the "java" process for the broker, since it is the broker that seems to be non-responsive.
LMK, Roy
Thanks [mention:30c5058a319047d288fe606ee12c9bbe:e9ed411860ed4f2ba0265705b8793d05] I'll try again next time.