v11.3.2 on Redhat Linux 6.7
Guys,
Any idea on this?
top - 15:03:17 up 28 days, 6:21, 206 users, load average: 39.28, 37.94, 37.04
Tasks: 2458 total, 41 running, 2415 sleeping, 0 stopped, 2 zombie
Cpu(s): 59.0%us, 2.8%sy, 0.0%ni, 38.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 504452988k total, 502160460k used, 2292528k free, 3889252k buffers
Swap: 67108860k total, 11272316k used, 55836544k free, 456598796k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19629 coins 20 0 337g 12g 12g R 100.0 2.7 11587:23 _progres
25539 coins 20 0 337g 11g 11g R 100.0 2.4 34845:15 _progres
25555 coins 20 0 337g 11g 11g R 100.0 2.4 34841:42 _progres
41335 coins 20 0 337g 12g 11g R 100.0 2.5 24615:18 _progres
41355 coins 20 0 337g 3.9g 3.8g R 100.0 0.8 24619:19 _progres
42444 coins 20 0 337g 2.9g 2.9g R 100.0 0.6 16078:59 _progres
42470 coins 20 0 337g 16g 16g R 100.0 3.4 15748:06 _progres
43664 coins 20 0 337g 13g 13g R 100.0 2.8 1442:10 _progres
etc
There are 25 active agents, 1 busy.
I've trimmed most agents - the OS processes did not go away
The db is up 24x7.
CPU utilisation history for the VM shows steady increase from application startup 10% to 60% plus now.
In the db log I'm seeing these:
[2016/11/10@14:09:46.000+0000] P-43690 T-140536583591712 I WSAGEN149: (49) SYSTEM ERROR: Memory violation.
[2016/11/10@15:35:48.000+0000] P-43679 T-139721194215200 I WSAGEN141: (49) SYSTEM ERROR: Memory violation.
[2016/11/11@08:34:05.000+0000] P-15676 T-140475284068128 I WSAGENT99: (49) SYSTEM ERROR: Memory violation.
[2016/11/11@13:32:14.000+0000] P-30294 T-140059337537312 I WSAGEN113: (49) SYSTEM ERROR: Memory violation.
[2016/11/11@13:36:19.000+0000] P-30337 T-140699800528672 I WSAGEN144: (49) SYSTEM ERROR: Memory violation.
Two suggestions:
1. Download ProTop Free and you'll see exactly what each one of these processes is doing.
2. Run $DLC/bin/proGetStack <pid> to get a stack trace of what program they're in.
Paul
Hi Peter
I have protop chui running, but don't know how to use it other than look at default and c screens...
I ran a proGetStack. Where does the output go?
Peter and Mary are busy. This is Paul. :-)
proGetStack: the protrace.<pid> file is written to the processes working directory. If you run lsof -p <pid> | grep cwd you'll see the current working directory.
In ProTop, the help screen is "h" (ProTop is case sensitive). Hit P and enter the PID of one of the agents. Then hit U to display the User Information Viewer.
proGetStack will tell you the program stack trace of dot-ps.
ProMon will tell you what tables and indexes are being used, assuming you have -tablerangesize and -indexrangesize set correctly as database startup parameters. If not, at the very least you'll see userIO data at the top - but take note to add -tablerangesize and -indexrangesize to your DB startup parameters.
The "*" command in ProTop will enable the client statement cache. This will show you, in real time, what line# of what program was running the last time each session communicated with the DB (look on the right-hand side of the UIO panel).
A second "*" will shut off that monitoring. You can also turn the statement cache on and off for individual sessions via PROMON.
Caution: Do not just turn it on and leave it on. There have been "issues" with the client statement cache on some releases of OpenEdge. It is a very powerful feature but, possibly because it is not well known, it still seems to have some rough edges.
Sorry Paul - I got your monika wrong!
I got the proGetStack output which I'll forward on to the vendor.
I have index and table ranges set OK
Can't seem to get anything to change when using protop P function. I enter a PID, such as one already listed on the default screen, and then can't really see any change of screens. I'm on 3.3n
Thanks for the tip Tom
After you set the pid with "P", you need to hit "U" to enter the User Information Viewer..
Oops! I've just borked my protop session by pressing CTRL-C at the wrong time. I've got Message Invalid buffer handle...
Hehe...oops.
You should be able to quit out with q after the messages go away. If not you'll have to kill it.
Had to kill it. I can see how the P / U works now. I get an error when I try to see Statement Cache data though
OK. Statement Cache data can now be seen. Must be me