Hi All,
I loaded Windows updates yesterday (and again today) and now my KUIB app doesn't work because some of my custom calls for data broke. The app contains data from data sources (configured via KUIB) as well as manual PUT calls via $http.put(). After checking a couple views it appears that the data source calls work correctly and the $http.put() calls no longer do (500 errors). Everything worked correctly yesterday morning before the Windows updates.
The REST service is a PAS DOH and the $http.put() calls that result in the 500 errors generate the following in my oepas1.agent.log:
[18/04/11@15:58:41.177-0500] P-010812 T-010476 1 AS-17 -- (Procedure: 'pas-session-startup.p' Line:28) Starting session () [18/04/11@15:58:41.368-0500] P-010812 T-010476 1 AS-17 -- pas-session-startup.p startup procedure ended with an ERROR condition. (8025) [18/04/11@15:58:41.368-0500] P-010812 T-010476 1 AS-17 MSAS Error (-6) running session startup procedure. [18/04/11@15:58:41.368-0500] P-010812 T-010476 1 AS-17 MSAS Unable to initialize session! [18/04/11@15:58:41.371-0500] P-010812 T-010476 1 AS-Aux-0 MSAS ERROR: Could not get and/or initialize 'State-free' session! Cannot process request. [18/04/11@15:58:41.371-0500] P-010812 T-010476 1 AS-Aux-0 MSAS Error handling request! Status=-1003 [18/04/11@15:58:41.371-0500] P-010812 T-010476 1 AS-Aux-0 MSAS Worker Thread exiting. Number: 4, Status: -14
The .2018-04-11.log contains the following:
16:24:16.219/3332159 [catalina-exec-5] ERROR com.progress.O4gl.Trace - READPACKET IOException : com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent (NetworkError[java.io.EOFException : Error reading message for (HODs4xnjT8ulxbMzxWK1gw) = null]:Agent) com.progress.appserv.broker.exception.BrokerException$NetworkException: NetworkError[java.io.EOFException : Error reading message for (HODs4xnjT8ulxbMzxWK1gw) = null]:Agent at com.progress.appserv.broker.session.LocalSession.processAgentRsp(LocalSession.java:1228) ~[oebroker.11.7.2.jar:na] at com.progress.appserv.broker.session.LocalSession.readMsg(LocalSession.java:958) ~[oebroker.11.7.2.jar:na] at com.progress.appserv.clientrt.broker.BrokerSystemOEWeb.readMsg(BrokerSystemOEWeb.java:84) ~[oeclientrt.11.7.2.jar:na] at com.progress.appserv.clientrt.broker.AbstractBrokerSystem.readPacket(AbstractBrokerSystem.java:1952) [oeclientrt.11.7.2.jar:na] ... at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] 16:24:16.221/3332161 [catalina-exec-8] ERROR c.p.appserv.adapters.web.Request - Open4GLException executing PUT request = com.progress.open4gl.Open4GLException: Communication layer message: General Error: READPACKET IOException : com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent (NetworkError[java.io.EOFException : Error reading message for (p5fDLXGUSQGBo_hLrdf9ww) = null]:Agent). (7175) : reason= null 16:24:16.222/3332162 [catalina-exec-5] ERROR c.p.appserv.adapters.web.Request - Open4GLException executing PUT request = com.progress.open4gl.Open4GLException: Communication layer message: General Error: READPACKET IOException : com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent (NetworkError[java.io.EOFException : Error reading message for (HODs4xnjT8ulxbMzxWK1gw) = null]:Agent). (7175) : reason= null
Also, if I make the same exact calls that are failing via Postman, they work correctly every time.
Could a Windows update change the way the $http.put() works? I've been pulling my hair out trying to figure this one out.
Thoughts?
Louis Winter
Found it! Well, it wasn't Microsoft after all.
After one of the reboots the other day I recall Developer Studio pushing old settings to PAS. I knew this because it changed my startup procedures and I had to change them back to what I had set via OpenEdge Explorer. What I didn't catch was that it changed the PROPATH as well. I found that the "Run configurations" is where the bad information was at that was killing me. Obviously I'm not a Dev Studio expert as we do all of our coding except for the new PAS code using good ol' vi.
So, the change that caused the issue in the PROPATH was my "AppServer" source folder vs the "webapps\JTS\WEB-INF\openedge" folder (with .r files). My guess is that when the multiple simultaneous requests hit PAS from the KUIB app, PAS stomps on itself compiling the source which is why one of the requests usually worked and the others failed. This also explains why the requests work from Postman, they are all single requests.
Now to make up for lost time...
Louis
Did you restart the system?
Yes, I even rebooted this afternoon just to see if that would fix it but it didn't.
Did you restart your servers and databases?
Yes. This is my dev environment and I have everything running on my laptop. Everything was restarted (DB, AdminServer, PAS, Kendo, etc.).
I find that usually the things most affected by updates are things that are supposed to autostart.
(Seeing as I am now free of being part of the Progress Support team, I feel it necessary for me to ask the question I always first thought of when getting this sort of issue)
What did Microsoft say when you asked them about this issue? It seems the point of change is the Windows Update. Shouldn't we be asking what they broke?
(Okay, now I got that off my chest, yes I understand that after they finished laughing behind their hands, they would say talk to Progress. There was a reason I use to claim the cheapest way to get support for any computer/application was to buy Client Networking with Maintenance and then when anything at all went wrong, blame Progress and log a support call! In order to prove it wasn't a Progress issue you normally have to solve/diagnose the underlying issue anyway)
(Okay, I know that wasn't helpful for you but I feel like I have saved some money on going to a therapist! ;-) )
Okay to try to be more helpful, what was in the Windows Update? Any changes to security? Any java changes? Any more clues in the Event Log?
From the log, it looks like you are failed in your startup procedure. Were you out of database connections ?
Hello,
Here is a suggestion for troubleshooting.
> Also, if I make the same exact calls that are failing via Postman, they work correctly every time.
If you are seeing different results from seemingly the same call ($http.put() and Postman), perhaps, you could look at the input to see if there are differences.
You could use a proxy to see the network traffic. You can use tools such as WSAViewer (run wsaviewer from proenv) or Fiddler.
Perhaps, you would see a difference in the HTTP payload, a cookie or an HTTP header that is different.
I hope this helps.
Edsel,
Good idea, I should have thought of doing that initially. Unfortunately nothing is jumping out at me that is different that seems like it could cause issues. Also, I have 1 call that is done twice (from different places in the KUIB code), when I tested it while capturing with Fiddler, I found that one of the calls worked, and the other returned a 500 error. In the trace, both requests are identical. And again, the Postman requests worked correctly.
I also tried 4 different browsers today and none of them worked correctly. I didn't get the same results across them but of the 4 PUT requests, either 3 or 4 failed in each browser.
After going through the Microsoft updates, which gave me a headache, I found one (KB4099950) that referenced NIC issues. I uninstalled this KB and it still doesn't work.
I'm also not seeing anything in the event logs.
Could there be some package that was automatically updated in my KUIB project?
Louis
I've uninstalled 5 Windows updates, only 2 remain that are for powershell so I'm leaving them. The Installed Updates control panel shows 105 Office KBs installed yesterday. I'm not going to attempt to start uninstalling any of them.
After uninstalling the Windows updates that were installed on 4/10 and 4/11, my issue still exists with the same characteristics.
I'm at a loss as to what to look at next. Thoughts?
Louis
Run the debugger on the startup procedure?
Found it! Well, it wasn't Microsoft after all.
After one of the reboots the other day I recall Developer Studio pushing old settings to PAS. I knew this because it changed my startup procedures and I had to change them back to what I had set via OpenEdge Explorer. What I didn't catch was that it changed the PROPATH as well. I found that the "Run configurations" is where the bad information was at that was killing me. Obviously I'm not a Dev Studio expert as we do all of our coding except for the new PAS code using good ol' vi.
So, the change that caused the issue in the PROPATH was my "AppServer" source folder vs the "webapps\JTS\WEB-INF\openedge" folder (with .r files). My guess is that when the multiple simultaneous requests hit PAS from the KUIB app, PAS stomps on itself compiling the source which is why one of the requests usually worked and the others failed. This also explains why the requests work from Postman, they are all single requests.
Now to make up for lost time...
Louis
I've been bitten by PDS a few times myself. My strategy currently consists of the following guidelines:
1) Turn on the option at Windows > Preferences > Progress OpenEdge > Server > Update properties from server before starting/launching
2) Don't use the built-in property editor in PDSOE for your PAS server. If at all possible set options via the tlr/merge.template or by directly editing the openedge.properties file for your instance.
3) Always keep a valid backup of the openedge.properties file and compare it to the existing file when things go wrong. Chances are something modified/reverted options in the file which you didn't anticipate.