We're running OE 188.8.131.52.1497 64-bit on Redhat 6.10 with classic WebSpeed web agents.
The problem we have:
Does anyone have some other ideas we can try to see what's causing the memory buildup and performance degradation?
We have the same issue with our Webspeed and larger reports. The Devs love TEMP-TABLES but our agents always end up in a mess memory wise. On the surface the code looks good.
Our work around is just to use:
wtbman -name MyWebBorker -refresh
"refresh" is easier then trim/add can be done online (we use stateless); it just waits till the agent becomes AVAILABLE then kills/readds it back into the pool. Give us back lots of memory, once a day is enough for our Ops system. But we could do it more often. We also do it with our AppServer (classic).
Did you try to run proGetStack <agent pid> periodically through the day? Perhaps you are loading more and more persistent procedures?
Also, have you looked at pmap <pid> to see what kind of memory is growing? You'll see open files, "shmid" which is DB shared memory (assuming the agents are connected to a DB in shared memory mode) then a bunch of "anon". Track the output to see what changes throughout the day. Also look at the difference between pmap and pmap -x output. I'm going by memory here, but if I do remember correctly, it is the "Dirty" column in pmap -x which is actual memory used for anon lines.
$ pmap -x 11763 | grep -i -e Kbytes -e anon -e stack
Address Kbytes RSS Dirty Mode Mapping
00000000011f7000 2944 148 148 rwx-- [ anon ]
00000000011f7000 2944 0 0 rwx-- [ anon ]
0000000003486000 888 620 620 rwx-- [ anon ]
0000000003486000 888 0 0 rwx-- [ anon ]
This output is truncated, but looking at the above 4 lines only it looks like this PID is using 2944+888 Kb of mem (watch out for duplicate addresses), but really it's only using 148+620.
If I do the real sums of anon lines, this is what I get:
sum(Kbytes) = 4292, sum(RSS)=848, sum(Dirty)=844
For fun, if I remove the greps to include shared objects like files and DB shared memory segments:
sum(Kbytes) = 90828, sum(RSS)=13716, sum(Dirty)=2292
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
11763 test 20 0 90824 13m 11m R 33.2 0.1 609:20.71 _progres
you would count 2 MB, but I'm skeptical. pmap is a much more detailed view.
If we were loading more and more persistent procedures that would've shown up in the DynObject messages. That's not the case.
Thanks about the details of pmap. Will try to get something out of it.