dsmLobUpdate causes inactivatable End step

Posted by Tom Oosterwijk on 28-Aug-2018 04:07

I have a process that is working perfectly fine for quite some time now (OEBPM 11.4), but last week the following issue happened twice for some process instances:

A subprocess gets suspended in the End step. I cannot reactivate it, because the process is completed according to the BIZLOGIC_PROCESSINSTANCE table (STATUS = 11), but it is not according to the PROCESSINSTANCE table (STATUS = PI_ACTIVATED). Is there any place where I can see what the status in BIZLOGIC_PROCESSINSTANCE means anyway?

I solved my problem by executing the following query:

UPDATE BIZLOGIC_PROCESSINSTANCE
SET
    STATUS = 7
WHERE
    BIZLOGIC_PROCESSINSTANCE.STATUS IN (6, 11)
    AND BIZLOGIC_PROCESSINSTANCE.PROCESS_TEMPLATE_ID = 4
    AND BIZLOGIC_PROCESSINSTANCE.PROCESS_INSTANCE_ID IN (
        SELECT bpi.PROCESS_INSTANCE_ID
        FROM BIZLOGIC_PROCESSINSTANCE bpi
        LEFT JOIN PROCESSINSTANCE pi ON bpi.PROCESS_INSTANCE_ID = pi.PROCESS_INSTANCE_ID
        WHERE
            bpi.STATUS IN (6, 11)
            AND bpi.PROCESS_TEMPLATE_ID = 4
            AND pi.STATUS = 'PI_ACTIVATED'
        )
;

After this, I can activate my End steps.

My customer wants to know what caused this (frankly, so do I), and I found the following in the bpserver.log:

[#| 17 Aug 2018 14:44:57,270 | BPServer | ERROR | ejbServer | BPServer | (1621) SQL Exception occurs in database transaction.; context(BLDAOService.handle(SQLEx))[DataDirect][OpenEdge JDBC Driver][OpenEdge] Internal error -100007 (see dsmret.h) in SQL from subsystem DSM SERVICES function dsmLobUpdate called from proLob::updateData on  for . Save log for Progress technical support. | WorkerThread#3[127.0.0.1:50922] |#]
java.sql.SQLException: [DataDirect][OpenEdge JDBC Driver][OpenEdge] Internal error -100007 (see dsmret.h) in SQL from subsystem DSM SERVICES function dsmLobUpdate called from proLob::updateData on  for . Save log for Progress technical support.
 at com.ddtek.jdbc.openedge.client.dde.ap(Unknown Source)
 at com.ddtek.jdbc.openedge.client.dde.h(Unknown Source)
 at com.ddtek.jdbc.openedge.client.dde.g(Unknown Source)
 at com.ddtek.jdbc.openedge.ddf.b(Unknown Source)
 at com.ddtek.jdbc.openedgebase.ddej.u(Unknown Source)
 at com.ddtek.jdbc.openedgebase.ddej.w(Unknown Source)
 at com.ddtek.jdbc.openedgebase.ddde.executeUpdate(Unknown Source)
 at com.ddtek.jdbcx.openedgebase.ddn.executeUpdate(Unknown Source)
 at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeUpdate(CachedPreparedStatement.java:96)
 at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
 at com.savvion.sbm.bizlogic.server.dao.DataSlotDAO.update(DataSlotDAO.java:1252)
 at com.savvion.sbm.bizlogic.server.dao.DataSlotDAO.store(DataSlotDAO.java:165)
 at com.savvion.sbm.bizlogic.server.WFDataSlotInstance.updateSlotValue(WFDataSlotInstance.java:643)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.updateSlotValue(WFProcessContext.java:710)
 at com.savvion.sbm.bizlogic.server.WFProcessInstance.completeCaller(WFProcessInstance.java:1575)
 at com.savvion.sbm.bizlogic.server.WFProcessInstance.complete(WFProcessInstance.java:1424)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.completeProcessInstance(WFProcessContext.java:953)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.complete(WFAtomicInstance.java:289)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.complete(WFAtomicInstance.java:195)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.execute(WFAtomicInstance.java:183)
 at com.savvion.sbm.bizlogic.server.WFWorkstepInstance.runIt(WFWorkstepInstance.java:2109)
 at com.savvion.sbm.bizlogic.server.WFWorkstepInstance.activate(WFWorkstepInstance.java:1191)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.activateWorkstep(WFProcessContext.java:991)
 at com.savvion.sbm.bizlogic.server.WFWorkstepInstance.activateNextWorksteps(WFWorkstepInstance.java:982)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.complete(WFAtomicInstance.java:291)
 at com.savvion.sbm.bizlogic.server.WFWorkItem.checkWorkStepCompletion(WFWorkItem.java:440)
 at com.savvion.sbm.bizlogic.server.WFWorkItem.complete(WFWorkItem.java:336)
 at com.savvion.sbm.bizlogic.server.WFWorkItem.complete(WFWorkItem.java:242)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.completeWorkItem(WFProcessContext.java:1160)
 at com.savvion.sbm.bizlogic.server.ejb.WorkItemSBBean.complete(WorkItemSBBean.java:320)
 at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:386)
 at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
 at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)
 at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)
 at org.jboss.ejb.plugins.security.PreSecurityInterceptor.process(PreSecurityInterceptor.java:97)
 at org.jboss.ejb.plugins.security.PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:81)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)
 at org.jboss.ejb.Container.invoke(Container.java:1029)
 at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
 at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:232)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
 at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
[#| 17 Aug 2018 14:44:57,270 | BPServer | ERROR | ejbServer | BPServer | (931) Exception occurs during workstep execution for PI <InvoiceApprovalSubprocess#265252>, Workstep <End 1> at <WFWorkstepInstance.runIt> | WorkerThread#3[127.0.0.1:50922] |#]
com.savvion.sbm.bizlogic.util.BizLogicException: (1621) SQL Exception occurs in database transaction.; context(BLDAOService.handle(SQLEx))[DataDirect][OpenEdge JDBC Driver][OpenEdge] Internal error -100007 (see dsmret.h) in SQL from subsystem DSM SERVICES function dsmLobUpdate called from proLob::updateData on  for . Save log for Progress technical support.
 at com.savvion.sbm.bizlogic.util.BLDAOService.handle(BLDAOService.java:143)
 at com.savvion.sbm.bizlogic.server.dao.DataSlotDAO.update(DataSlotDAO.java:1254)
 at com.savvion.sbm.bizlogic.server.dao.DataSlotDAO.store(DataSlotDAO.java:165)
 at com.savvion.sbm.bizlogic.server.WFDataSlotInstance.updateSlotValue(WFDataSlotInstance.java:643)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.updateSlotValue(WFProcessContext.java:710)
 at com.savvion.sbm.bizlogic.server.WFProcessInstance.completeCaller(WFProcessInstance.java:1575)
 at com.savvion.sbm.bizlogic.server.WFProcessInstance.complete(WFProcessInstance.java:1424)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.completeProcessInstance(WFProcessContext.java:953)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.complete(WFAtomicInstance.java:289)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.complete(WFAtomicInstance.java:195)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.execute(WFAtomicInstance.java:183)
 at com.savvion.sbm.bizlogic.server.WFWorkstepInstance.runIt(WFWorkstepInstance.java:2109)
 at com.savvion.sbm.bizlogic.server.WFWorkstepInstance.activate(WFWorkstepInstance.java:1191)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.activateWorkstep(WFProcessContext.java:991)
 at com.savvion.sbm.bizlogic.server.WFWorkstepInstance.activateNextWorksteps(WFWorkstepInstance.java:982)
 at com.savvion.sbm.bizlogic.server.WFAtomicInstance.complete(WFAtomicInstance.java:291)
 at com.savvion.sbm.bizlogic.server.WFWorkItem.checkWorkStepCompletion(WFWorkItem.java:440)
 at com.savvion.sbm.bizlogic.server.WFWorkItem.complete(WFWorkItem.java:336)
 at com.savvion.sbm.bizlogic.server.WFWorkItem.complete(WFWorkItem.java:242)
 at com.savvion.sbm.bizlogic.server.WFProcessContext.completeWorkItem(WFProcessContext.java:1160)
 at com.savvion.sbm.bizlogic.server.ejb.WorkItemSBBean.complete(WorkItemSBBean.java:320)
 at sun.reflect.GeneratedMethodAccessor759.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:386)
 at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
 at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)
 at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)
 at org.jboss.ejb.plugins.security.PreSecurityInterceptor.process(PreSecurityInterceptor.java:97)
 at org.jboss.ejb.plugins.security.PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:81)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)
 at org.jboss.ejb.Container.invoke(Container.java:1029)
 at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
 at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:232)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
 at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
Caused by: java.sql.SQLException: [DataDirect][OpenEdge JDBC Driver][OpenEdge] Internal error -100007 (see dsmret.h) in SQL from subsystem DSM SERVICES function dsmLobUpdate called from proLob::updateData on  for . Save log for Progress technical support.
 at com.ddtek.jdbc.openedge.client.dde.ap(Unknown Source)
 at com.ddtek.jdbc.openedge.client.dde.h(Unknown Source)
 at com.ddtek.jdbc.openedge.client.dde.g(Unknown Source)
 at com.ddtek.jdbc.openedge.ddf.b(Unknown Source)
 at com.ddtek.jdbc.openedgebase.ddej.u(Unknown Source)
 at com.ddtek.jdbc.openedgebase.ddej.w(Unknown Source)
 at com.ddtek.jdbc.openedgebase.ddde.executeUpdate(Unknown Source)
 at com.ddtek.jdbcx.openedgebase.ddn.executeUpdate(Unknown Source)
 at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeUpdate(CachedPreparedStatement.java:96)
 at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
 at com.savvion.sbm.bizlogic.server.dao.DataSlotDAO.update(DataSlotDAO.java:1252)
 ... 52 more

Can I prevent this issue? How can this happen?

All Replies

Posted by Tushar Dhaka on 28-Aug-2018 05:04

This error is thrown at SQL level, and based on the following looks like its due to some corrupted BLOB data.

knowledgebase.progress.com/.../000056311

Posted by Tom Oosterwijk on 28-Aug-2018 05:17

Thanks. This might be useful, but when I reactivate the End task, everything runs smoothly again. I would expect corrupted BLOB data to be a consequent issue.

Unless it can be caused by the db suddenly crashing or something? The instances that had the problem all had them at approximately the same time.

Posted by Tushar Dhaka on 28-Aug-2018 05:30

Since the origin of the error is while executing the update on database, it has to do something related to database state or data itself. According to the article I shared error code 100007 indicates BLOB is corrupt, so it should be due to corrupt data.

Reactivating the task makes everything work fine? I see you mentioned executing a query to manually update the status for this.

Posted by Tom Oosterwijk on 28-Aug-2018 06:52

Yes, I reset the state of BIZLOGIC_PROCESSINSTANCE back to active, and then (the second try) the process decently close itself. No changes to the blob were made.

Posted by Tushar Dhaka on 28-Aug-2018 07:03

Interesting. Please share all log files and a reproducible case if possible, will check out possible cause.

Posted by Tom Oosterwijk on 28-Aug-2018 09:49

Unfortunately this logging was the only useful logging I could still find, and I don't know how to reproduce it (yet).

There are also no errors or restarts visible in the db.

Perhaps I just need to cross my fingers and hope it was an incident...

Posted by Tushar Dhaka on 28-Aug-2018 23:31

I'll check this scenario with our SQL team once to get some understanding of possible causes and get back to you. Expect some delay in my response though.

Posted by Tom Oosterwijk on 13-Sep-2018 01:21

I just found a knowledgebase article about the error message:

knowledgebase.progress.com/.../000056311

It does not explain how it is possible that a process is marked complete in one table in OEBPM but active in another (why are there multiple tables keeping the status anyway, and why is the status kept in different types? I still don't know what all the status numbers in the BIZLOGIC_PROCESSINSTANCE table mean).

So I'm going to see if the following helps:

ALTER TABLE BIZLOGIC_DS_357 ALTER COLUMN LARGE_DATASLOTS SET PRO_SQL_WIDTH 2147483600;

(EDIT: The above query is invalid, though I do not know why yet.)

Posted by Tom Oosterwijk on 18-Sep-2018 05:14

At the moment I'm afraid the query does not work because the column LARGE_DATASLOTS is an lvarbinary, since is does work for a varchar column.

Posted by Tom Oosterwijk on 11-Apr-2019 08:57

Hi Tushar, got any luck finding the problem? *fingers crossed*

Posted by Jawahar Surapaneni on 15-Apr-2019 12:08

Hi Tom,

I have gone through the thread. As you have mentioned, that you were having issues with data that holds blob type data.

As per my understanding, you are seeing the issue on 11.4.0  version.

In 11.4.0 version, we have introduced a new dataslot called Longchar, which can hold huge data ranging from 1MB to 5MB. Is it possible for you to use Longchar dataslot to process the blob type data. I cannot guarantee the proposed workaround will work 100% , but  based on the discussion happened  in this thread, i feel, it could solve your issue.

Regards,

Jawahar.

This thread is closed