Deploy Services Failed on Server Startup

Posted by Ricardo Jorge on 03-Jul-2017 04:52

Hi,

We've Corticon Server instaled in Azure (programmed to restart every day) and we're facing the followin problem.


Context:
Some Services are using EDC to connect Oracle Database.

At some days on Server start up something fails on Oracle Listener and causes Corticon Services that use that EDC were undeploy. Services that don't use EDC were correctly deployed.

Is there a way to force Service Deploy without validating the EDC?

Thanks,
Ricardo Jorge

Corticon - Forum

All Replies

Posted by Gertjan Hendriks on 03-Jul-2017 08:56

Hi Ricardo,

The same behaviour can be seen when the datamodel of a database is altered, without importing the metadata in Studio after that database-change and re-publishing the Decision Service.

As far as I know there is no fix yet, but I'll leave it to the dev-team to answer that.

-Gertjan

Posted by Ricardo Jorge on 04-Jul-2017 04:07

Hi, Gerjan,

Thanks for the replying.

Is there a way of using a command line to check if services are deployed and if not deploy them. We was thinking on a Schedule task that time to time checks it and deploy if necessary.

This enviromnent have to be available all days and we have to create a work around for this behaviour.

Thanks,

Ricardo Jorge

Posted by Gertjan Hendriks on 04-Jul-2017 07:38

Hi Ricardo,

The quick and dirty option that I see to work around this, would be to create a copy of a valid ServerState.xml with all Decision Services succesfully deployed. In case the Oracle Listener was unsuccesful, Corticon Server will clear the ServerState.xml, with no deployed Decision Services as a result. The ServerState.xml can be found in ~\CorticonWork\SER\CcServerSandbox\DoNotDelete\.

Perhaps your scheduled task can check the ServerState.xml to see if it is invalid (no decision service present), stop Corticon Server, replace the faulty ServerState.xml with the backed-up valid ServerState.xml, and restart Corticon Server again when the Listeners are up and running.

I hope this makes sense.

-Gertjan

Posted by James Arsenault on 04-Jul-2017 07:48

Ricardo,
 
The testserveraxis.bat utility can be used to inspect a Corticon server but it’s an interactive tool and not good in a script. However the Corticon server REST and SOAP APIs can be used to do what you want. For the REST APIs have a look at the swagger documentation (change the host and port to your Corticon server)
 
 
For example the API
 
 
Lists deployed decision services. Using curl you can get a list such as
 
{
  "decisionServices": [
    {
      "loadedFromCdd": true,
      "name": "AllocateTrade",
      "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C2_1499171248430.619502/AllocateTrade_v1_14.eds",
      "majorVersion": 1,
      "minorVersion": 14
   },
    {
      "loadedFromCdd": true,
      "name": "Candidates",
      "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C3_1499171254573.549221/Candidates_v1_14.eds",
      "majorVersion": 1,
      "minorVersion": 14
    },
    {
      "loadedFromCdd": true,
      "name": "Cargo",
      "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C0_1499171236109.374753/Cargo_v0_16.eds",
      "majorVersion": 0,
      "minorVersion": 16
    },
    {
      "loadedFromCdd": true,
      "name": "ProcessOrder",
      "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C1_1499171240858.284274/ProcessOrder_v1_10.eds",
      "majorVersion": 1,
      "minorVersion": 10
    }
  ]
}
 
That’s formatted, curl will give unformatted output:
 
C:\tools\curl-7.48.0-win64-mingw\bin>curl localhost:8850/.../list
{"decisionServices":[{"loadedFromCdd":true,"name":"AllocateTrade","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C2_1499171248430.619502/AllocateTrade_v1_14.eds","majorVersion":1,"minorVersion":14},{"loadedFromCdd":true,"name":"Candidates","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C3_1499171254573.549221/Candidates_v1_14.eds","majorVersion":1,"minorVersion":14},{"loadedFromCdd":true,"name":"Cargo","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C0_1499171236109.374753/Cargo_v0_16.eds","majorVersion":0,"minorVersion":16},{"loadedFromCdd":true,"name":"ProcessOrder","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C1_1499171240858.284274/ProcessOrder_v1_10.eds","majorVersion":1,"minorVersion":10}]}
 
But there are json utilities available to help you parse it.
 
Jim
 

Posted by Ricardo Jorge on 17-Jul-2017 08:18

Hi Gertjan,

We have chosen your "dirty" way.

I created a powershell script that checks if service is on and if it is not restarts all necessary services replacing the faulty ServerState.xml with the backed-up valid one.

Thanks.

Ricardo Jorge

Posted by Gertjan Hendriks on 18-Jul-2017 06:52

Hi Ricardo,

Good to hear I could help you on this. You're welcome. And yes, keep in mind that this fix is very dirty ;-)

-Gertjan

Posted by Chris S. Hogan on 18-Jul-2017 10:10

Sorry for the late reply, but there is a property to disable the validation of EDC enabled decision services at startup

###############################################################################################################

# Determines whether the server will invoke the JPOX metadata validation service at Ruleset deployment

# time. If this property is set to "true" the service will compare the external JDO document with the

# database metadata and will prevent deployment if any discrepancies are detected.

#

# Default is true

###############################################################################################################

com.corticon.ccserver.validateSchema=true

If you set this property to false, then the validation step should be skipped at startup.

This might be a workable solution until a more robust error handling is put in pace.

Posted by Gertjan Hendriks on 20-Jul-2017 07:53

Thanks for this suggestion, Chris.

As changing the datamodel, without re-importing the metadata and re-deploying *all* decision services after the change, can also cause an undeployment of decision services (even a datatype-change in a database-table can trigger this!), this is an acceptable work-around to avoid this problem.

Yet good error handling on this is much preferred.

Posted by Ricardo Jorge on 02-Aug-2017 04:57

Hi Chris,

First of all thanks for your suggestion, that seems to be a more correct way of doing it.

As far as I know that configuration you mention is part of CcServer.properties file right?

Well, we installed the corticon server in a websphere environment and I can't find the file there... is there a way to configure it somewhere else?

Hi Gertjan,

The dirty way fails when someone changed rules and din't backed up the ServerState.xml file with the new changes. That way when server restarts we loose lastest rule changes.

I'll try to use Chris suggestion but if it won't work (i'm not finding where to change property) I'll consider the following workaround:

1) Iterate ServerState.xml file and for each DecisionService (name and version);

2) Search DecisionService most recent file (ex. servicev1.0.eds) in the folder "DoNotDelete\DecisionServices"

3) then I'll update EDS path with the one of most recente file

4) I'll do the same with database properties.

I think that way I'll mitigate the problem.

Ricardo Jorge

Posted by Chris S. Hogan on 02-Aug-2017 12:32

The properties files can either be modified inside CcConfig.jar file or you can create a brms.properties file and place it in the Server Work directory. Any setting specified in there will over write existing settings.
 
For more details on setting properties, look at “Configuring Corticon properties and settings” from the Corticon Server:Integration & Deployment Guide
 
Christopher S. Hogan
Principal Systems Engineer
 
Progress
 
Office:  +1-346-352-0699
Mobile:  +1-646-243-4282
 
 
TwitterFacebook LinkedIn Google+
 

Posted by Ricardo Jorge on 07-Aug-2017 05:47

Hi Chris,

I have read the section you refered from https://documentation.progress.com/output/Corticon/5.6.1/corticon_integration_deployment.pdf but didn't found anything related to the property "com.corticon.ccserver.validateSchema".

Even so i followed your instructions: created a brms.properties file and placed it at corticon work directory (C:/IBM/WebSphere/AppServer)

My brms file only have one uncommented line: "com.corticon.ccserver.validateSchema=false"

Today on server startup something have failed with oracle listener.
Corticon Decision Services didn't deploy as cause of listener failure.CcServerDatabaseValidationException

What did i do wrong?

Error Log from "CcServer.log"

2017-08-07 07:50:29.382 INFO  [server.startup : 2] Cc  - Corticon Work: C:/IBM/WebSphere/AppServer

2017-08-07 07:51:07.029 ERROR  [server.startup : 2] Cc  -  com.corticon.service.ccserver.exception.CcServerDatabaseValidationException: [progress][Oracle JDBC Driver]Error establishing socket to host and port: 0:0:0:0:0:0:0:1:1521. Reason: Connection refused: connect

2017-08-07 07:51:07.029 ERROR  [server.startup : 2] Cc com.corticon.eclipse.server.core.impl.CcServerInitialization - CcServerInitialization().loadAndDeployCcServerInfoNonCDDs() = Failed to deploy Decision Service. com.corticon.service.ccserver.exception.CcServerDatabaseValidationException: [progress][Oracle JDBC Driver]Error establishing socket to host and port: 0:0:0:0:0:0:0:1:1521. Reason: Connection refused: connect

at com.corticon.eclipse.server.core.impl.CcServerImpl.configureAndValidateForDecisionsServiceDatabaseAccess(CcServerImpl.java:4095)

at com.corticon.eclipse.server.core.impl.CcServerImpl.addDecisionService(CcServerImpl.java:3611)

at com.corticon.eclipse.server.core.impl.CcServerInitialization.loadAndDeployCcServerInfoNonCDDs(CcServerInitialization.java:494)

at com.corticon.eclipse.server.core.impl.CcServerInitialization.loadAndDeployDecisionServices(CcServerInitialization.java:72)

at com.corticon.eclipse.server.core.impl.CcServerImpl.initialize(CcServerImpl.java:805)

at com.corticon.eclipse.server.core.impl.CcServerImpl.<init>(CcServerImpl.java:685)

at com.corticon.eclipse.server.core.CcServerFactory.getCcServer(CcServerFactory.java:30)

at com.corticon.servlets.CcServerServletInitializer.<clinit>(CcServerServletInitializer.java:21)

at java.lang.J9VMInternals.initializeImpl(Native Method)

at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)

at java.lang.Class.forNameImpl(Native Method)

at java.lang.Class.forName(Class.java:213)

at java.beans.Beans.instantiate(Beans.java:189)

at java.beans.Beans.instantiate(Beans.java:80)

at com.ibm.ws.webcontainer.servlet.ServletWrapper$1.run(ServletWrapper.java:1483)

at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadServlet(ServletWrapper.java:1472)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1351)

at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:606)

at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:576)

at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)

at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)

at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)

at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)

at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)

at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)

at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)

at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1175)

at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)

at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)

at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)

at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:774)

at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2182)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)

at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)

at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

 

 

Thanks for all help,

Ricardo Jorge

Posted by Chris S. Hogan on 07-Aug-2017 12:16

Ricardo,
 
Are you sure you have the correct work dir location? This is the Corticon Server work dir, not the app server work dir.
 
You can use this SOAP API to retrieve the appropriate setting for the Corticon Server:
 
Christopher S. Hogan
Principal Systems Engineer
 
Progress
 
Office:  +1-346-352-0699
Mobile:  +1-646-243-4282
 
 
TwitterFacebook LinkedIn Google+
 

Posted by Ricardo Jorge on 07-Aug-2017 13:33

I think it's the correct one.

I called the webmethod getCcPropertyValue for the property CORTICON_WORK_DIR and it returned:

"C:/IBM/WebSphere/AppServer"

Thanks,

Ricardo Jorge

Posted by Chris S. Hogan on 08-Aug-2017 15:45

Ricardo,

It appears that support for this feature was removed in a previous version. I am checking to see if it can be added back in a future release. However, for the time being it looks like you will have to use the previous solution.

Chris

This thread is closed