We have a DB + appserver application with a particular screen that stays open and continually processes jobs. The screen closes itself after 100 jobs processed to delete the persistent procedure that governs the screen as a pre-emptive measure against memory leaks. We use a Progress backend and a VB rendering engine/client. The processor runs on the same machine where the application server is and connects to it directly, and the appserver connects to a DB on another machine.
About once a week, we start getting errors in one of our processes that depends on the movement of files using OS-COPY/DELETE/MOVE/COMMAND that necessitates restarting the machine. Several hours before our team sees anything that suggests something is wrong, the windows event viewer Application logs become littered with the following:
A single instance of: Faulting application CONSPAWN.EXE, version 11.1.0.1143, time stamp 0x4fdbcf95, faulting module kernel32.dll, version 6.0.6002.19346, time stamp 0x55024102, exception code 0xc0000142, fault offset 0x0006f49f, process id 0x26f0, application start time 0x01d1bcd825479176.
CONSPAWN.EXE
11.1.0.1143
4fdbcf95
kernel32.dll
6.0.6002.19346
55024102
c0000142
0006f49f
26f0
01d1bcd825479176
Repeated instances of: Faulting application _proapsv.exe, version 11.1.0.1143, time stamp 0x544f8f95, faulting module kernel32.dll, version 6.0.6002.19346, time stamp 0x55024102, exception code 0xc0000142, fault offset 0x0006f49f, process id 0xdcc, application start time 0x01d1bcdc7cb0d0ba.
_proapsv.exe
11.1.0.1143
544f8f95
kernel32.dll
6.0.6002.19346
55024102
c0000142
0006f49f
dcc
01d1bcdc7cb0d0ba
Repeated instances of: Faulting application cmd.exe, version 6.0.6001.18000, time stamp 0x47919317, faulting module kernel32.dll, version 6.0.6002.19346, time stamp 0x55023e3e, exception code 0xc0000142, fault offset 0x00000000000b6f68, process id 0x115c, application start time 0x01d1bcdd56c68663.
I've seen few suggestions as to why this could be happening, mostly from this thread: https://community.progress.com/community_groups/openedge_general/f/26/t/16566. It seems like a vague enough error, but since the errors are coming from calls to not only two Progress executables but calls to cmd.exe sourced in our application, the customer's IT seems unwilling to research system resource issues in our place. They're running Server 2007 - I'm not sure if anyone has experience installing dheapmon on this OS, but I'm unable to install it on my Windows 8.1 laptop due to several problems, rendering testing it pretty difficult. I'm getting pretty desperate at this point.
The processes running on this processor moving files run quite fine for the majority of the time - it usually takes about a week for the errors to start, leading me to believe there's some sort of resource leak involved, but we're not sure at all where or what to monitor. Even if we were able to get dheapmon installed and were lucky enough to find that it's a desktop heap problem, increasing the size of the heap would only serve to delay the issue, I would think.
Forgot to mention that the customer is running OE 11.1.