Hi All,
In our database went down due to memory violation error.I checked in promon > 1> Status: Shared Memory Segments
It shows all memory utilized.
Now database and application up and running but shared memory segments is full for all database.
what action i need to take to resolve this issue.
Status: Shared Memory Segments
Seg Id Size Used Free
1 393265 134217728 134217724 4
2 393248 134217728 134217724 4
3 393264 134217728 134217716 12
4 393247 134217728 134217716 12
5 393228 134217728 120246044 13971684
6 393263 25251840 396 25251444
Enter <return>, R, P, T, or X (? for help):
19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
19:08:47 SQLSRV2 3: SYSTEM ERROR: Memory violation. (49)
19:08:50 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)
19:08:57 SQLSRV2 4: SYSTEM ERROR: Memory violation. (49)
19:08:59 SRV 35: Corrupt block detected when attempting to release a buffer. (4232)
19:08:59 SRV 35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
19:08:59 SRV 35: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block
PROGRESS Version 9.1E as of Tue Oct 12 17:19:22 EDT 2004
PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
131072 - - - (not PROGRESS)
131073 - - - (not PROGRESS)
We tried rebooting server option also but few seconds it is worked fine, again it showing shared memory segments full.
OS: Aix 5.3
Progress version : 9.1e
Application : Mfg/pro 8.3
> On Apr 11, 2018, at 9:06 AM, durgaprasad m wrote:
>
> 19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
Seen these many times before. These crashes are caused by very old bugs in the OpenEdge SQL Server code. You need to upgrade. Version 9.1E04 was released in 2006 and is now 12 years old.
Faced with this situation I'd run as diagnostics for hardware failures, particularly a faulty disc.
If you find this it, will need to be resolved asap.
Not sure what diagnostics to run on AIX but I'm sure others / the internet will know this.
Then I think your options for the database are either to go back to the latest backup from before the issues started or if this is not an option then do a dump and load to hopefully repair the corruption that has occured in area 6 as per the error.
If there are hardware issues there's not much point doing the above until after these issues are fixed as the same could just happen again.
Finally post
Due to memory violation and connection time out errors user are not able to access mfg/pro applications
> but shared memory segments is full for all database.
Last segments are NOT full
> Corrupt block detected when attempting to release a buffer. (4232)
Did you check the system memory during server's reboot? I guess it's a failure of RAM.
Present situation of the database
04/11/18 Status: Shared Memory Segments
09:25:07
Seg Id Size Used Free
1 22 134217728 134217724 4
2 23 134217728 134217724 4
3 24 134217728 134217716 12
4 25 134217728 134217716 12
5 26 134217728 120258516 13959212
6 27 25251840 396 25251444
Total Paging Space Percent Used
4608MB 1%
IPCS comand output :
p-10004*:/progdata/prd/slcdb1: ipcs -am
IPC status from /dev/mem as of Wed Apr 11 09:29:21 EDT 2018
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 131072 0x78000033 --rw-rw-rw- root system root system 1 33554432 41288 434112 6:55:29 9:21:16 2:05:50
m 131073 0x7800002e --rw-rw-rw- root system root system 1 33554432 41288 434112 6:55:29 9:21:16 2:05:50
m 131074 0x0d05332f --rw-rw---- root system root system 173 1440 16552 432508 9:29:16 9:29:21 2:05:02
m 3 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 1 2830336 40370 40370 2:07:16 no-entry 2:07:16
m 4 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 1 2830336 53958 294206 7:00:03 7:00:03 2:07:16
m 5 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 1 2830336 40006 198430 7:00:02 7:00:02 2:07:16
m 6 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 1 3678208 54462 54462 2:07:16 no-entry 2:07:16
m 7 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 52 6307840 40778 365608 8:26:08 8:26:08 2:07:16
m 8 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 1 2830336 44140 44140 2:07:16 no-entry 2:07:16
m 9 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 52 6426624 41602 357358 8:25:27 8:25:27 2:07:16
m 10 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 4 128491520 56762 130058 6:43:55 6:44:09 2:07:16
m 11 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 52 6246400 57020 365092 8:26:07 8:26:07 2:07:16
m 12 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 14 134217728 57794 148452 7:00:01 7:00:01 2:07:16
m 13 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 14 134217728 57794 148452 7:00:01 7:00:01 2:07:16
m 14 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 14 134217728 57794 148452 7:00:01 7:00:01 2:07:16
m 15 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 14 134217728 57794 148452 7:00:01 7:00:01 2:07:16
m 16 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 14 126713856 57794 148452 7:00:01 7:00:01 2:07:16
m 17 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 9 134217728 58310 155544 7:00:02 7:00:02 2:07:16
m 18 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 9 134217728 58310 155544 7:00:02 7:00:02 2:07:16
m 19 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 9 134217728 58310 155544 7:00:02 7:00:02 2:07:16
m 20 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 9 134217728 58310 155544 7:00:02 7:00:02 2:07:16
m 21 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 9 37187584 58310 155544 7:00:02 7:00:02 2:07:16
m 22 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 36 134217728 45974 485942 9:25:00 9:25:37 2:07:16
m 23 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 36 134217728 45974 485942 9:25:00 9:25:37 2:07:16
m 24 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 36 134217728 45974 485942 9:25:00 9:25:37 2:07:16
m 25 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 36 134217728 45974 485942 9:25:00 9:25:37 2:07:16
m 26 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 36 134217728 45974 485942 9:25:00 9:25:37 2:07:16
m 27 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 36 25251840 45974 485942 9:25:00 9:25:37 2:07:16
m 28 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 27 134217728 56506 310142 9:29:15 9:29:15 2:07:16
m 29 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 27 134217728 56506 310142 9:29:15 9:29:15 2:07:16
m 30 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 27 134217728 56506 310142 9:29:15 9:29:15 2:07:16
m 31 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 27 134217728 56506 310142 9:29:15 9:29:15 2:07:16
m 32 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 27 37175296 56506 310142 9:29:15 9:29:15 2:07:16
m 33 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 47 134217728 23630 472222 9:15:00 9:15:05 2:07:16
m 34 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 47 134217728 23630 472222 9:15:00 9:15:05 2:07:16
m 35 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 47 134217728 23630 472222 9:15:00 9:15:05 2:07:16
m 36 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 47 26300416 23630 472222 9:15:00 9:15:05 2:07:16
m 37 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 52 7151616 58570 476054 9:23:42 9:23:42 2:07:16
m 38 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 5 6672384 59860 160942 4:38:02 4:38:43 2:07:16
m 39 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 30 134217728 42856 450010 9:20:01 9:20:01 2:07:16
m 40 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 30 134217728 42856 450010 9:20:01 9:20:01 2:07:16
m 41 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 30 134217728 42856 450010 9:20:01 9:20:01 2:07:16
m 42 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 30 134217728 42856 450010 9:20:01 9:20:01 2:07:16
m 43 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 30 126652416 42856 450010 9:20:01 9:20:01 2:07:16
m 44 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 5 43663360 60890 263182 4:37:28 4:37:49 2:07:16
m 45 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 22 134217728 59090 478866 9:20:07 9:20:07 2:07:16
m 46 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 22 134217728 59090 478866 9:20:07 9:20:07 2:07:16
m 47 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 22 134217728 59090 478866 9:20:07 9:20:07 2:07:16
m 48 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 22 134217728 59090 478866 9:20:07 9:20:07 2:07:16
m 49 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 22 38039552 59090 478866 9:20:07 9:20:07 2:07:16
m 50 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 5 41148416 61408 465230 9:05:00 9:05:06 2:07:16
m 51 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress 4 2891776 61924 320412 7:00:02 7:00:02 2:07:16
System configuration: lcpu=6 mem=22528MB ent=2.90
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 0 1821474 70132 0 0 0 0 0 0 425 127514 9687 26 13 60 1 1.43 49.2
0 0 1821324 70097 0 0 0 0 0 0 611 19809 6391 6 7 84 3 0.47 16.2
1 0 1820366 71023 0 0 0 0 0 0 544 76921 4776 10 11 74 5 0.72 24.8
0 0 1819724 71637 0 0 0 0 0 0 305 15837 4518 3 5 91 1 0.30 10.3
1 0 1819998 70824 0 0 0 0 0 0 1070 227439 5869 12 11 77 1 0.81 27.8
PROGRESS Version 9.1E as of Tue Oct 12 17:19:22 EDT 2004
PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
131072 - - - (not PROGRESS)
131073 - - - (not PROGRESS)
*svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
196652 _sqlsrv2 2858997 34581 0 218297 N Y N
188694 _sqlsrv2 2858248 34577 0 217548 N Y N
105538 _progres 2858101 34569 0 216373 N N N
92372 _progres 2858101 34569 0 216373 N N N
125142 _progres 2858101 34569 0 216373 N N N
106838 _progres 2858101 34569 0 216373 N N N
189660 _sqlsrv2 2858034 34576 0 217334 N Y N
259304 _mprosrv 2856955 34569 0 215892 N N N
58062 _mprosrv 2856940 34569 0 215877 N N N
137794 _mprosrv 2856936 34569 0 215873 N N N
170554 _mprosrv 2856936 34569 0 215873 N N N
386766 _mprosrv 2856935 34569 0 215872 N N N
94514 _mprosrv 2856935 34569 0 215872 N N N
354772 _mprosrv 2856933 34569 0 215870 N N N
412070 _mprosrv 2856933 34569 0 215870 N N N
svmon -G
size inuse free pin virtual
memory 5767168 5693016 74152 275290 1818627
pg space 1179648 5645
work pers clnt other
pin 199722 0 0 75568
in use 1818627 3305255 569134
Your problem is not what you think it is.
Shared memory use is expected behavior and is completely normal. It has nothing to do with the memory violation error, the corrupt block or the db crash.
The fact that you are running version 9 is a much bigger problem than the fact that you can display the shared memory segments that are in use.
Version 9 was conceived and coded in the 1990s. The environments that is was designed to run in have changed considerably. Many, many things are dramatically different in todays world -- security and multi-core systems memory to name just a couple of the more obvious changes.
Is using v9 responsible for your crash? Probably not. It looks more like a problem with faulty hardware. If the hardware is as old and unsupported as v9 that isn't very surprising.
Thanks Tom
yes we know vesion is out of support.
May be in latest version this issue got fixed.
can i know why we are falling into this issue everytime.
"everytime"? You are posting that shared memory is being used. That is normal. There is nothing to solve about that.
You have reported one db crash so far.
Have you had more than 1 db crash? If so - do the log messages always refer to block 717557?
Have you opened a support call about:
19:08:59 SRV 35: Corrupt block detected when attempting to release a buffer. (4232)
19:08:59 SRV 35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
19:08:59 SRV 35: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block
Hi Tom,
We don't have vendor support and already we contacted for the same with progress already said they are out of support also.
> 19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block
Post the beginning of block dump.
Please find below for our database log history events during the issue
18:52:17 SRV 2: HANGUP signal received. (562)
18:52:17 SRV 2: Previous message sent on behalf of user 905. (5512)
18:52:17 SRV 2: Logout usernum 905, userid y550sxn, on p5lp5 281. (739)
18:52:20 SRV 14: HANGUP signal received. (562)
18:52:20 SRV 14: Previous message sent on behalf of user 1026. (5512)
18:52:20 SRV 14: Logout usernum 1026, userid y550sxn, on p5lp5 210. (739)
18:52:34 SRV 2: Login usernum 1084, userid y550axp, on p5lp5 72. (742)
18:53:26 SRV 2: HANGUP signal received. (562)
18:53:26 SRV 2: Previous message sent on behalf of user 1084. (5512)
18:53:26 SRV 2: Logout usernum 1084, userid y550axp, on p5lp5 72. (739)
18:53:37 SRV 2: Login usernum 1084, userid y550axp, on p5lp5 72. (742)
18:53:38 SRV 2: HANGUP signal received. (562)
18:53:38 SRV 2: Previous message sent on behalf of user 1084. (5512)
18:53:38 SRV 2: Logout usernum 1084, userid y550axp, on p5lp5 72. (739)
18:55:01 SRV 2: Login usernum 1084, userid pgresdba, on p5lp5 batch. (742)
18:55:01 SRV 2: Logout usernum 1084, userid pgresdba, on p5lp5 batch. (739)
18:56:37 SRV 8: Logout usernum 1068, userid cquinn, on mhlctxqad07a. (739)
18:58:00 Usr 44: Login by pgresdba on batch. (452)
18:58:05 SRV 11: HANGUP signal received. (562)
18:58:05 SRV 11: Previous message sent on behalf of user 930. (5512)
18:58:05 SRV 10: HANGUP signal received. (562)
18:58:05 SRV 10: Previous message sent on behalf of user 931. (5512)
18:58:05 SRV 11: Logout usernum 930, userid y020syw, on p5lp5 247. (739)
18:58:05 SRV 10: Begin transaction backout. (2252)
18:58:05 SRV 10: Transaction backout completed. (2253)
18:58:05 SRV 10: Logout usernum 931, userid y020syw, on p5lp5 246. (739)
18:58:25 SRV 2: Login usernum 1084, userid y020syw, on p5lp5 18. (742)
18:58:35 SRV 33: Logout usernum 947, userid eylst, on mhlctxqad02a. (739)
18:58:51 SRV 7: Login usernum 1083, userid y020syw, on p5lp5 60. (742)
18:59:13 SRV 21: HANGUP signal received. (562)
18:59:13 SRV 21: Previous message sent on behalf of user 1007. (5512)
18:59:13 SRV 21: Begin transaction backout. (2252)
18:59:13 SRV 21: Transaction backout completed. (2253)
18:59:13 SRV 21: Logout usernum 1007, userid y550jyw, on p5lp5 179. (739)
18:59:49 SRV 16: Usernum 1034 terminated abnormally. (794)
18:59:49 SRV 16: Begin transaction backout. (2252)
18:59:49 SRV 16: Transaction backout completed. (2253)
18:59:49 SRV 16: Logout usernum 1034, userid aaustin, on mhlctxqad08a. (739)
18:59:51 SRV 21: HANGUP signal received. (562)
18:59:51 SRV 21: Previous message sent on behalf of user 1009. (5512)
18:59:51 SRV 21: Begin transaction backout. (2252)
18:59:51 SRV 21: Transaction backout completed. (2253)
18:59:51 SRV 21: Logout usernum 1009, userid y550jyw, on p5lp5 177. (739)
19:00:00 Usr 46: Login by pgresdba on batch. (452)
19:00:00 SRV 7: Login usernum 1080, userid pgresdba, on p5lp5 batch. (742)
19:00:01 RFUTIL 47: Login by pgresdba on batch. (452)
19:00:01 RFUTIL 47: Logout by pgresdba on batch. (453)
19:00:01 SRV 8: Login usernum 1075, userid pgresdba, on p5lp5 batch. (742)
19:00:01 SRV 8: Logout usernum 1075, userid pgresdba, on p5lp5 batch. (739)
19:00:05 SRV 7: Logout usernum 1080, userid pgresdba, on p5lp5 batch. (739)
19:00:05 Usr 46: Logout by pgresdba on batch. (453)
19:00:30 SRV 24: Logout usernum 1064, userid npassey, on mhlctxqad02a. (739)
19:00:32 Usr 44: Logout by pgresdba on batch. (453)
19:01:35 SRV 7: Login usernum 1080, userid jonewman, on mhlctxqad08a. (742)
19:02:37 SRV 9: Logout usernum 1070, userid mthorne, on mhlctxqad08a. (739)
19:03:26 SRV 17: Logout usernum 1029, userid aaldrich, on MHLCTXQAD03A. (739)
19:05:00 Usr 44: Login by pgresdba on batch. (452)
19:05:00 Usr 46: Login by pgresdba on batch. (452)
19:05:00 Usr 47: Login by pgresdba on batch. (452)
19:05:01 SRV 8: Login usernum 1075, userid pgresdba, on p5lp5 batch. (742)
19:05:01 SRV 8: Logout usernum 1075, userid pgresdba, on p5lp5 batch. (739)
19:05:13 Usr 44: Logout by pgresdba on batch. (453)
19:05:16 Usr 46: Logout by pgresdba on batch. (453)
19:06:07 Usr 47: Logout by pgresdba on batch. (453)
19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
19:08:47 SQLSRV2 3: SYSTEM ERROR: Memory violation. (49)
19:08:50 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)
19:08:57 SQLSRV2 4: SYSTEM ERROR: Memory violation. (49)
19:08:59 SRV 35: Corrupt block detected when attempting to release a buffer. (4232)
19:08:59 SRV 35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
19:08:59 SRV 35: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block
19:08:59 SRV 35: pbktbl = 0x3050bb3c
19:08:59 SRV 35: pbktbl->qself = 0x5850bb3c
19:08:59 SRV 35: XBKBUF(pbktbl->qself) = 0x3050bb3c
19:08:59 SRV 35: pbktbl->bt_qbuf = 0x642be444
19:08:59 SRV 35: XBKBUF(pbktbl->bt_qbuf) = 0x442be444
19:08:59 SRV 35: pbkbuf = 0x442be444
19:08:59 SRV 35: Block dbkey = 0x1b5b7980 bt_offset = 0x0
19:08:59 SRV 35: File: /progdata/prd/slcdb6/prdslc.d55
19:08:59 SRV 35: 0000: 5846 624c 0000 08e4 5850 bbec 642b e444
19:08:59 SRV 35: 0010: 0000 0001 0100 038a 0000 a26b 0019 0000
19:08:59 SRV 35: 0020: 1b5b 7980 0000 0006 67ff f3f8 0000 0000
19:08:59 SRV 35: 0030: 0000 0000 3dc3 8000 ffff ffff 0000 0000
19:08:59 SRV 35: 0040: 0000 0000 0000 0000 0000 0000 584d 49dc
19:08:59 SRV 35: 0050: 5878 ea4c 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0060: 0000 0000 0000 0000 0001 0100 0000 0000
19:08:59 SRV 35: 0070: 1e7f 0000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0080: 0000 0000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0090: 0000 0000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 00a0: 5850 bb3c 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: Block buffer dump follows, from address 0x442be444:
19:08:59 SRV 35: 0000: 00ff fa00 05fd fdfd fdff 0000 0000 4fe7
19:08:59 SRV 35: 0010: 0002 0042 0200 ea08 3135 3330 3031 3530
19:08:59 SRV 35: 0020: 0431 3030 3608 3135 3331 3031 3530 0004
19:08:59 SRV 35: 0030: 8223 329f 0e49 4330 3030 3532 3534 3030
19:08:59 SRV 35: 0040: 3330 3203 1d94 b306 4953 532d 5452 0000
19:08:59 SRV 35: 0050: 0102 0101 0000 0401 ac86 6500 0000 0114
19:08:59 SRV 35: 0060: e700 0e00 3800 6900 9700 ba00 ca00 f101
19:08:59 SRV 35: 0070: 0102 008c 0735 3539 3531 3530 0247 6efd
19:08:59 SRV 35: 0080: 0649 5353 2d54 5207 7265 7475 726e 7302
19:08:59 SRV 35: 0090: 806f 0380 153f 0000 0002 4541 0247 6e08
19:08:59 SRV 35: 00a0: 636c 696e 6970 6164 0000 0000 0000 0585
19:08:59 SRV 35: 00b0: 3851 008f 0486 0025 5304 8602 3625 0687
19:08:59 SRV 35: 00c0: 3888 0845 2f03 1d94 b304 8223 329f 0000
19:08:59 SRV 35: 00d0: 0739 3338 3631 312a 00fd 0200 6f07 7935
19:08:59 SRV 35: 00e0: 3530 7078 7208 3336 616b 3030 3333 0247
19:08:59 SRV 35: 00f0: 6e04 4136 3536 0000 0000 0000 0003 5553
19:08:59 SRV 35: 0100: 4402 801f 0002 6beb 0587 3445 872f 0433
19:08:59 SRV 35: 0110: 3630 3008 7175 6172 6e6d 6f76 0002 48c2
19:08:59 SRV 35: 0120: 0000 0000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0130: 00fd fdfd fdfd 0000 0000 0000 0000 0214
19:08:59 SRV 35: 0140: 700a 6963 6c6f 7472 3032 2e70 0004 3336
19:08:59 SRV 35: 0150: 3030 0000 00fd 00fa 0005 0000 0000 ff00
19:08:59 SRV 35: 0160: 0000 0000 0000 0000 0000 0000 fd00 0000
19:08:59 APW 39: Stopped. (2520)
19:08:59 SRV 35: 0170: 0000 fd00 00ea e700 0e00 3100 4d00 7400
19:08:59 SRV 35: 0180: 9000 a000 c700 d702 008c 0937 3030 3032
19:08:59 SRV 35: 0190: 3442 3031 0247 6efd 0652 4354 2d54 5205
19:08:59 SRV 35: 01a0: 5354 4147 4500 0380 107f 0003 8010 0f00
19:08:59 SRV 35: 01b0: 0245 4102 476e 0000 0000 0000 0003 8223
19:08:59 SRV 35: 01c0: 7200 0003 8223 7203 1d94 b203 8023 7200
19:08:59 SRV 35: 01d0: 0000 00fd 0380 100f 0779 3535 3062 766b
19:08:59 SRV 35: 01e0: 0002 476e 0441 3035 3400 0000 0000 0000
19:08:59 SRV 35: 01f0: 0355 5344 0280 1f00 026b dd00 0433 3031
19:08:59 SRV 35: 0200: 3208 494e 2053 5441 4745 00fd 0000 0000
19:08:59 SRV 35: 0210: 0000 0000 0000 0000 0000 0000 00fd fdfd
19:08:59 SRV 35: 0220: fdfd 0000 0000 0000 0000 0214 700a 6963
19:08:59 SRV 35: 0230: 6c6f 7472 3032 2e70 0004 3330 3132 0000
19:08:59 SRV 35: 0240: 00fd 00fa 0005 0000 0000 ff00 0000 0000
19:08:59 SRV 35: 0250: 0000 0000 0000 0000 fd00 0000 0000 fd00
19:08:59 SRV 35: 0260: 004e e700 0200 4102 00ea 0831 3533 3130
19:08:59 SRV 35: 0270: 3135 3000 0831 3531 3130 3135 3004 3130
19:08:59 SRV 35: 0280: 3030 0380 2372 0e49 4330 3030 3532 3534
19:08:59 SRV 35: 0290: 3031 3230 3103 1d94 b206 5243 542d 5452
19:08:59 SRV 35: 02a0: 0000 0107 0101 0000 0401 ac86 6200 0000
19:08:59 SRV 35: 02b0: 004e e700 0200 4102 00ea 0831 3531 3130
19:08:59 SRV 35: 02c0: 3135 3004 3130 3030 0831 3533 3130 3135
19:08:59 SRV 35: 02d0: 3000 0380 2372 0e49 4330 3030 3532 3534
19:08:59 SRV 35: 02e0: 3031 3230 3103 1d94 b106 4953 532d 5452
19:08:59 SRV 35: 02f0: 0000 0106 0105 0000 0401 ac86 5e00 0000
19:08:59 SRV 35: 0300: 00eb e700 0e00 3200 4e00 7500 9100 a100
19:08:59 SRV 35: 0310: c800 d802 008c 0937 3030 3032 3442 3031
19:08:59 SRV 35: 0320: 0247 6efd 0649 5353 2d54 5203 5152 4d03
19:08:59 SRV 35: 0330: 8010 0f03 8020 7f00 0300 100f 0002 4541
19:08:59 SRV 35: 0340: 0247 6e00 0000 0000 0000 0382 2372 0000
19:08:59 SRV 35: 0350: 0382 2372 031d 94b1 0380 2372 0000 0000
19:08:59 SRV 35: 0360: fd03 0010 0f07 7935 3530 6276 6b00 0247
19:08:59 SRV 35: 0370: 6e04 4130 3534 0000 0000 0000 0003 5553
19:08:59 SRV 35: 0380: 4402 801f 0002 6bdd 0004 3330 3132 0849
19:08:59 SRV 35: 0390: 4e20 5354 4147 4500 fd00 0000 0000 0000
19:08:59 SRV 35: 03a0: 0000 0000 0000 0000 0000 fdfd fdfd fd00
19:08:59 SRV 35: 03b0: 0000 0000 0000 0002 1470 0a69 636c 6f74
19:08:59 SRV 35: 03c0: 7230 322e 7000 0433 3031 3200 0000 fd00
19:08:59 SRV 35: 03d0: fa00 0500 0000 00ff 0000 0000 0000 0000
19:08:59 SRV 35: 03e0: 0000 0000 00fd 0000 0000 00fd 0000 4fe7
19:08:59 SRV 35: 03f0: 0002 0042 0200 ea08 3135 3331 3031 3530
19:08:59 SRV 35: 0400: 0008 3135 3330 3031 3530 0431 3030 3604
19:08:59 SRV 35: 0410: 8211 664f 0e49 4330 3030 3532 3534 3030
19:08:59 SRV 35: 0420: 3330 3203 1d94 b006 5243 542d 5452 0000
19:08:59 SRV 35: 0430: 0104 0103 0000 0401 ac86 5b00 0000 0062
19:08:59 SRV 35: 0440: e700 0400 3b00 5801 3b05 3738 3130 3804
19:08:59 SRV 35: 0450: 3738 3430 0257 4f02 4750 0247 4f04 8211
19:08:59 SRV 35: 0460: 164f 0e57 4f30 3030 3432 3436 3337 3730
19:08:59 SRV 35: 0470: 3100 0007 7935 3530 6c78 6200 0000 0355
19:08:59 SRV 35: 0480: 5344 0004 3130 3130 0101 0280 1f00 0000
19:08:59 SRV 35: 0490: 0000 0001 3103 3135 3000 0002 801f 0000
19:08:59 SRV 35: 04a0: 0000 0062 e700 0400 3b00 5801 3b05 3738
19:08:59 SRV 35: 04b0: 3939 3704 3738 3638 0257 4f02 4750 0247
19:08:59 SRV 35: 04c0: 4b04 0212 055f 0e57 4f30 3030 3432 3036
19:08:59 SRV 35: 04d0: 3337 3730 3100 0007 7935 3530 6c78 6200
19:08:59 SRV 35: 04e0: 0000 0355 5344 0004 3130 3130 0108 0280
19:08:59 SRV 35: 04f0: 1f00 0000 0000 0001 3103 3135 3000 0002
19:08:59 SRV 35: 0500: 801f 0000 0000 00f5 e700 0e00 3a00 5c00
19:08:59 SRV 35: 0510: 8100 9500 a500 d200 e202 008c 0730 3630
19:08:59 SRV 35: 0520: 3035 3234 0247 5002 4750 064f 5244 2d53
19:08:59 SRV 35: 0530: 4f00 0002 8074 0280 2f00 0002 4541 0247
19:08:59 SRV 35: 0540: 4f08 5331 3234 3332 3537 0853 3132 3433
19:08:59 SRV 35: 0550: 3235 3700 0831 3030 3833 3338 3200 0000
19:08:59 SRV 35: 0560: 0484 4097 3900 0003 8019 7f03 1bb1 8300
19:08:59 SRV 35: 0570: 0000 0000 fd00 0779 3535 3070 7877 0002
19:08:59 SRV 35: 0580: 4750 0441 3232 3400 0000 0001 0100 0003
19:08:59 SRV 35: 0590: 5553 4402 801f 0002 7a9a 0004 3336 3030
19:08:59 SRV 35: 05a0: 0000 fd00 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 05b0: 0000 0000 fdfd fdfd fd00 0000 0000 0000
19:08:59 SRV 35: 05c0: 0000 0873 6f73 6f6d 742e 7000 0000 0000
19:08:59 SRV 35: 05d0: fd00 fa00 1308 5435 3534 3134 3131 0444
19:08:59 SRV 35: 05e0: 4953 5400 0247 47ff 0000 0000 0000 0000
19:08:59 SRV 35: 05f0: 0000 0000 00fd 0000 0000 00fd 0000 a8e7
19:08:59 SRV 35: 0600: 000c 001c 0038 004e 005e 0085 0097 0153
19:08:59 SRV 35: 0610: 0303 fe23 0444 4f57 4e02 474b 0154 0000
19:08:59 SRV 35: 0620: 0002 8115 0000 0000 0000 0003 8212 2606
19:08:59 SRV 35: 0630: 3839 3939 3931 0000 0000 0000 0003 8212
19:08:59 SRV 35: 0640: 2600 0000 0000 0000 0437 3836 3800 0000
19:08:59 SRV 35: 0650: 0002 4750 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0660: 0000 0000 0000 0000 0000 fdfd 0000 0004
19:08:59 SRV 35: 0670: 3336 3030 0a73 666f 7074 7230 342e 7002
19:08:59 SRV 35: 0680: 7a93 0779 3535 306c 7862 0000 0000 0000
19:08:59 SRV 35: 0690: 0000 0000 0000 0000 0000 0280 1f00 0000
19:08:59 SRV 35: 06a0: 0000 00fd 0000 0000 53e7 0002 0047 0200
19:08:59 SRV 35: 06b0: af08 3738 3939 3731 3530 0437 3836 3808
19:08:59 SRV 35: 06c0: 3738 3130 3831 3530 0437 3836 3803 03fe
19:08:59 SRV 35: 06d0: 2303 8212 260e 574f 3030 3034 3230 3633
19:08:59 SRV 35: 06e0: 3737 3031 084c 4252 2d32 3030 3000 0001
19:08:59 SRV 35: 06f0: 0801 0700 0003 0147 6f00 0000 00a8 e700
19:08:59 SRV 35: 0700: 0c00 1c00 3800 4e00 5e00 8500 9701 5303
19:08:59 SRV 35: 0710: 03fe 2204 444f 574e 0247 4b01 5400 0000
19:08:59 SRV 35: 0720: 0281 1500 0000 0000 0000 0382 1226 0638
19:08:59 SRV 35: 0730: 3939 3939 3100 0000 0000 0000 0382 1226
19:08:59 SRV 35: 0740: 0000 0000 0000 0004 3738 3638 0000 0000
19:08:59 SRV 35: 0750: 0247 5000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0760: 0000 0000 0000 0000 00fd fd00 0000 0433
19:08:59 SRV 35: 0770: 3630 300a 7366 6f70 7472 3034 2e70 027a
19:08:59 SRV 35: 0780: 8e07 7935 3530 6c78 6200 0000 0000 0000
19:08:59 SRV 35: 0790: 0000 0000 0000 0000 0002 801f 0000 0000
19:08:59 SRV 35: 07a0: 0000 fd00 0000 0053 e700 0200 4702 00af
19:08:59 SRV 35: 07b0: 0837 3839 3937 3135 3004 3738 3638 0837
19:08:59 SRV 35: 07c0: 3831 3038 3135 3004 3738 3638 0303 fe22
19:08:59 SRV 35: 07d0: 0382 1226 0e57 4f30 3030 3432 3036 3337
19:08:59 SRV 35: 07e0: 3730 3108 4c42 522d 3230 3030 0000 0108
19:08:59 SRV 35: 07f0: 0107 0000 0301 476c 0000 0000 a8e7 000c
19:08:59 SRV 35: 0800: 001c 0038 004e 005e 0085 0097 0153 0303
19:08:59 SRV 35: 0810: fe21 0444 4f57 4e02 474b 0154 0000 0002
19:08:59 SRV 35: 0820: 8115 0000 0000 0000 0003 8212 2606 3839
19:08:59 SRV 35: 0830: 3939 3931 0000 0000 0000 0003 8212 2600
19:08:59 SRV 35: 0840: 0000 0000 0000 0437 3836 3800 0000 0002
19:08:59 SRV 35: 0850: 4750 0000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0860: 0000 0000 0000 0000 fdfd 0000 0004 3336
19:08:59 SRV 35: 0870: 3030 0a73 666f 7074 7230 342e 7002 7a87
19:08:59 SRV 35: 0880: 0779 3535 306c 7862 0000 0000 0000 0000
19:08:59 SRV 35: 0890: 0000 0000 0000 0000 0280 1f00 0000 0000
19:08:59 SRV 35: 08a0: 00fd 0000 0000 53e7 0002 0047 0200 af08
19:08:59 SRV 35: 08b0: 3738 3939 3731 3530 0437 3836 3808 3738
19:08:59 BROKER 0: Begin ABNORMAL shutdown code 2 (2249)
19:08:59 SRV 35: 08c0: 3130 3831 3530 0437 3836 3803 03fe 2103
19:08:59 SRV 35: 08d0: 8212 260e 574f 3030 3034 3230 3633 3737
19:08:59 SRV 35: 08e0: 3031 084c 4252 2d32 3030 3000 0001 0801
19:08:59 SRV 35: 08f0: 0700 0003 0147 6a00 0000 00a8 e700 0c00
19:08:59 SRV 35: 0900: 1c00 3800 4e00 5e00 8500 9701 5303 03fe
19:08:59 SRV 35: 0910: 2004 444f 574e 0247 4b01 5400 0000 0281
19:08:59 SRV 35: 0920: 1500 0000 0000 0000 0382 1226 0638 3939
19:08:59 SRV 35: 0930: 3939 3100 0000 0000 0000 0382 1226 0000
19:08:59 SRV 35: 0940: 0000 0000 0004 3738 3638 0000 0000 0247
19:08:59 SRV 35: 0950: 5000 0000 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0960: 0000 0000 0000 00fd fd00 0000 0433 3630
19:08:59 SRV 35: 0970: 300a 7366 6f70 7472 3034 2e70 027a 8207
19:08:59 SRV 35: 0980: 7935 3530 6c78 6200 0000 0000 0000 0000
19:08:59 SRV 35: 0990: 0000 0000 0000 0002 801f 0000 0000 0000
19:08:59 SRV 35: 09a0: fd00 0000 0053 e700 0200 4702 00af 0837
19:08:59 SRV 35: 09b0: 3839 3937 3135 3004 3738 3638 0837 3831
19:08:59 SRV 35: 09c0: 3038 3135 3004 3738 3638 0303 fe20 0382
19:08:59 SRV 35: 09d0: 1226 0e57 4f30 3030 3432 3036 3337 3730
19:08:59 SRV 35: 09e0: 3108 4c42 522d 3230 3030 0000 0108 0107
19:08:59 SRV 35: 09f0: 0000 0301 4767 0000 0000 e9e7 000e 002e
19:08:59 SRV 35: 0a00: 004c 007a 0092 00a2 00c6 00d6 0200 8c07
19:08:59 SRV 35: 0a10: 3535 3433 3135 3002 4750 fd07 5243 542d
19:08:59 SRV 35: 0a20: 4348 4c04 7034 626e 0003 8013 0000 0280
19:08:59 SRV 35: 0a30: 5500 0245 4102 4750 0000 0000 0000 0004
19:08:59 SRV 35: 0a40: 8415 3355 0000 0003 1bb1 8200 0000 0732
19:08:59 SRV 35: 0a50: 3033 3530 3439 00fd 0280 5507 7935 3530
19:08:59 SRV 35: 0a60: 7467 6808 3832 6367 3030 3937 0247 5004
19:08:59 SRV 35: 0a70: 4136 3136 0000 0000 0000 0003 5553 4402
19:08:59 SRV 35: 0a80: 801f 0002 7a69 0004 3336 3030 0266 6700
19:08:59 SRV 35: 0a90: 0248 a400 0000 0000 0000 0000 0000 0000
19:08:59 SRV 35: 0aa0: 0000 0000 fdfd fdfd fd00 0000 0000 0000
19:08:59 SRV 35: 0ab0: 0000 0969 6369 746c 6d74 2e70 0004 3336
19:08:59 SRV 35: 0ac0: 3030 0000 00fd 00fa 0005 0000 0000 ff00
19:08:59 SRV 35: 0ad0: 0000 0000 0000 0000 0000 0000 fd00 0000
19:08:59 SRV 35: 0ae0: 0000 fd00 0028 e700 0200 1b02 00ea 0000
19:08:59 SRV 35: 0af0: 0000 0000 031b b182 0752 4354 2d43 484c
19:08:59 SRV 35: 0b00: 0000 0000 0000 0401 ac86 5700 0000 0028
19:08:59 SRV 35: 0b10: e700 0200 1b02 00ea 0000 0000 0000 031b
19:08:59 SRV 35: 0b20: b181 0749 5353 2d43 484c 0000 0000 0000
19:08:59 SRV 35: 0b30: 0401 ac86 5400 0000 00ef e700 0e00 2e00
19:08:59 SRV 35: 0b40: 4c00 7a00 9800 a800 cc00 dc02 008c 0735
19:08:59 SRV 35: 0b50: 3534 3331 3530 0247 50fd 0749 5353 2d43
19:08:59 SRV 35: 0b60: 484c 0470 3462 6e02 8055 0380 1355 0000
19:08:59 SRV 35: 0b70: 0002 4541 0247 5000 0000 0000 0000 0484
19:08:59 SRV 35: 0b80: 1533 5500 0000 031b b181 0000 0007 3230
19:08:59 SRV 35: 0b90: 3335 3034 3900 fd02 0055 0779 3535 3074
19:08:59 SRV 35: 0ba0: 6768 0838 3263 6730 3039 3702 4750 0441
19:08:59 BIW 41: Stopped. (2520)
9:08:59 SRV 35: 0bb0: 3631 3600 0000 0000 0000 0355 5344 0280
19:08:59 SRV 35: 0bc0: 1f00 027a 6900 0433 3630 3008 5141 2051
19:08:59 SRV 35: 0bd0: 5541 524e 0002 48a4 0000 0000 0000 0000
19:08:59 SRV 35: 0be0: 0000 0000 0000 0000 00fd fdfd fdfd 0000
19:08:59 SRV 35: 0bf0: 0000 0000 0000 0009 6963 6974 6c6d 742e
19:08:59 SRV 35: 0c00: 7000 0433 3630 3000 0000 fd00 fa00 0500
19:08:59 SRV 35: Stopped. (2520)
19:08:59 SRV 9: Stopped. (2520)
19:08:59 SRV 7: Stopped. (2520)
19:08:59 SRV 16: Stopped. (2520)
19:08:59 SRV 10: Stopped. (2520)
19:08:59 SRV 30: Stopped. (2520)
19:08:59 SRV 2: Stopped. (2520)
19:08:59 SRV 11: Stopped. (2520)
19:08:59 SRV 20: Stopped. (2520)
19:08:59 SRV 8: Stopped. (2520)
19:08:59 SRV 24: Stopped. (2520)
19:08:59 SRV 13: Stopped. (2520)
19:08:59 SRV 12: Stopped. (2520)
19:08:59 SRV 17: Stopped. (2520)
19:08:59 SRV 25: Stopped. (2520)
19:08:59 SRV 14: Stopped. (2520)
19:08:59 SRV 28: Stopped. (2520)
19:08:59 SRV 18: Stopped. (2520)
19:08:59 SRV 15: Stopped. (2520)
19:08:59 SRV 21: Stopped. (2520)
19:08:59 SRV 29: Stopped. (2520)
19:08:59 SRV 19: Stopped. (2520)
19:08:59 APW 36: Stopped. (2520)
19:08:59 AIW 42: Stopped. (2520)
19:08:59 SRV 32: Stopped. (2520)
19:08:59 SRV 22: Stopped. (2520)
19:08:59 SRV 23: Stopped. (2520)
19:08:59 SRV 33: Stopped. (2520)
19:08:59 SRV 26: Stopped. (2520)
19:08:59 SRV 31: Stopped. (2520)
19:08:59 SRV 27: Stopped. (2520)
19:08:59 SRV 34: Stopped. (2520)
19:08:59 APW 37: Stopped. (2520)
19:08:59 APW 38: Stopped. (2520)
19:09:00 WDOG 40: Stopped. (2520)
19:09:01 BROKER : Removed shared memory with segment_id: 262191
19:09:01 BROKER : Removed shared memory with segment_id: 262156
19:09:01 BROKER : Removed shared memory with segment_id: 262175
19:09:01 BROKER : Removed shared memory with segment_id: 262192
19:09:01 BROKER : Removed shared memory with segment_id: 262176
19:09:01 BROKER : Removed shared memory with segment_id: 262193
19:09:01 BROKER : Multi-user session end. (334)
19:09:24 Usr 54: Logout by pgresdba on batch. (453)
19:09:29 Usr 50: Logout by pgresdba on batch. (453)
19:09:33 Usr 43: Logout by pgresdba on batch. (453)
19:09:33 Usr 48: Logout by pgresdba on batch. (453)
19:09:33 Usr 49: Logout by pgresdba on batch. (453)
19:09:42 Usr 45: Logout by pgresdba on batch. (453)
> On Apr 11, 2018, at 9:06 AM, durgaprasad m wrote:
>
> 19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
Seen these many times before. These crashes are caused by very old bugs in the OpenEdge SQL Server code. You need to upgrade. Version 9.1E04 was released in 2006 and is now 12 years old.
19:08:59 SRV 35: Block buffer dump follows, from address 0x442be444: 19:08:59 SRV 35: 0000: 00ff fa00 05fd fdfd fdff 0000 0000 4fe7 19:08:59 SRV 35: 0010: 0002 0042 0200 ea08 3135 3330 3031 3530 19:08:59 SRV 35: 0020: 0431 3030 3608 3135 3331 3031 3530 0004 19:08:59 SRV 35: 0030: 8223 329f 0e49 4330 3030 3532 3534 3030 19:08:59 SRV 35: 0040: 3330 3203 1d94 b306 4953 532d 5452 0000
It's indeed a Progress data block but it's not a beginning of block (as it should be).
For example:
08 3135 3330 3031 3530
means a field value of 8 characters: "15300150"
Conclusion: some process left in db buffer pool the correct data but at wrong offset. I would suspect the processes that issued the memory violation error.
There is no option to release shared memory segments from the server level.
I thought newer versions of the utilities would clean up shared memory segments ? I thought I remember that being mentioned in the past.
Been a long time since we had to run ipcrm.
> There is no option to release shared memory segments from the server level.
This problem has nothing to do with shared memory segments.
You seem to think that "Memory violation" has sometime to do with "full shared memory segments". As if you think they are running out of space or something.
Nothing like that is happening.
It is completely and perfectly normal for your shared memory segments to look the way they do. When the db starts up it allocates and initializes shared memory to store -B and various other data structures. The OS may limit the size of any one chunk of shared memory - thus there can be multiple "segments". Ideally they will all be full. That way you are not wasting any. But depending on how much you have asked for there can be rounding of sizes and Progress reserves a bit extra for some things. So they are not all always completely full. But there is nothing wrong with any of the shared memory utilization data that you have shown.
It is normal.
You do not have a problem with full shared memory segments.
A "memory violation" is not a capacity problem. It is an *access* problem. The db server attempted to access a memory location that it does not have permission to access. The location that it attempted to access *might* have been in shared memory. Or it might not have -- we don't know, the bogus address is not reported in the error message.
As George points out there is a bad address in the corrupt block that is reported *later*. That may, or may not, be a useful clue.
As Gus points out the SQL92 engine in v9 has many bugs of this sort.
Your best course of action is to upgrade to a modern release.
If the hardware and operating system are as old as v9 then you should also make sure that they get upgraded.
Your worst course of action is to continue to think that "full shared memory segments" is in any way related to your db crash.
As far as "root cause" goes -- it was probably either a hardware problem or one of those bugs that Gus refers to. If it is hardware then I would expect it to continue to happen and that errors will appear in the system logs. But it may not be obvious that they are related to this.
If it is a Progress bug you are pretty much out of luck. Support for v9 mostly consists of sympathy for your predicament.
Hi Tom,
Today also we received same memory violation error in our database log and database went down.
Again we restarted database its came up after that 2 mins mfg/pro application received "server or system ha no more resource" connect .
After few mins issue resolved on it owns mfg/pro application we are able to connect.
06:26:49 SRV 7: vv_flush:I/O error 5 on fd 1
06:26:49 SRV 7: Previous message sent on behalf of user 1071. (5512)
06:26:49 SRV 7: HANGUP signal received. (562)
06:26:49 SRV 7: Previous message sent on behalf of user 1071. (5512)
06:26:49 SRV 7: Begin transaction backout. (2252)
06:26:49 SRV 7: Transaction backout completed. (2253)
06:26:49 SRV 7: Logout usernum 1071, userid y020txe, on p5lp5 122. (739)
06:26:50 SRV 7: HANGUP signal received. (562)
06:26:50 SRV 7: Previous message sent on behalf of user 1072. (5512)
06:26:50 SRV 7: Logout usernum 1072, userid y020txe, on p5lp5 51. (739)
06:27:28 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)
06:27:29 SRV 6: SYSTEM ERROR: Attempt to read block 0 which does not exist. (210)
06:27:29 SRV 6: ** Save file named core for analysis by Progress Software Corporation. (439)
06:27:29 SRV 6: Logout usernum 1085, userid y020txe, on p5lp5 42. (739)
06:27:29 SRV 6: Logout usernum 1081, userid fsobien, on mhlctxqad07a. (739)
06:27:29 SRV 6: Logout usernum 1083, userid y020axh, on p5lp5 1. (739)
06:27:29 SRV 6: Begin transaction backout. (2252)
06:27:29 SRV 6: Transaction backout completed. (2253)
06:27:29 SRV 6: Logout usernum 1080, userid rhughes, on mhlctxqad08a. (739)
06:27:29 SRV 6: Logout usernum 1082, userid y020dyv, on p5lp5 123. (739)
06:27:29 SRV 6: Stopped. (2520)
06:27:38 SRV 8: Corrupt block detected when attempting to release a buffer. (4232)
06:27:38 SRV 8: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
06:27:38 SRV 8: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
06:27:38 SRV 8: SYSTEM DEBUG: Database buffer block
06:27:38 SRV 8: pbktbl = 0x30a0401c
06:27:38 SRV 8: pbktbl->qself = 0x58a0401c
06:27:38 SRV 8: XBKBUF(pbktbl->qself) = 0x30a0401c
06:27:38 SRV 8: pbktbl->bt_qbuf = 0x72a30bb4
06:27:38 SRV 8: XBKBUF(pbktbl->bt_qbuf) = 0x62a30bb4
06:27:38 SRV 8: pbkbuf = 0x62a30bb4
06:27:38 SRV 8: Block dbkey = 0x1b5b7980 bt_offset = 0x0
06:27:38 SRV 8: File: /progdata/prd/slcdb6/prdslc.d55
06:27:38 SRV 8: 0000: 0000 0000 0000 08e4 58a0 40cc 72a3 0bb4
06:27:38 SRV 8: 0010: 0000 0001 0100 042d 0000 23fc 001b 0000
06:27:38 SRV 8: 0020: 1b5b 7980 0000 0006 67ff f3f8 0000 0000
06:27:38 SRV 8: 0030: 0000 0000 3dc3 8000 ffff ffff 0000 0000
06:27:38 SRV 8: 0040: 0000 0000 0000 0000 0000 0000 58a2 d62c
06:27:38 SRV 8: 0050: 5859 3c8c 0000 0000 0000 0000 0000 0000
06:27:38 SRV 8: 0060: 0000 0000 0000 0000 0001 0100 0000 0000
06:27:38 SRV 8: 0070: 006a 0000 0000 0000 0000 0000 0000 0000
06:27:38 SRV 8: 0080: 0000 0000 0000 0000 0000 0000 0000 0000
06:27:38 SRV 8: 0090: 0000 0000 0000 0000 0000 0000 0000 0000
06:27:38 SRV 8: 00a0: 58a0 401c 0000 0000 0000 0000 0000 0000
06:27:38 SRV 8: Block buffer dump follows, from address 0x62a30bb4:
06:27:38 SRV 8: 0000: 0401 ab6f 5e06 4953 532d 534f 0000 0102
....
06:27:38 SRV 8: 1ff0: 487f 0000 0004 01a6 8ae5 0000 0000 00fd
06:27:38 SRV 8: End of buffer
06:27:38 SRV 8: SYSTEM ERROR: wrong dbkey in block. Found 67218287, should be 458979712 in area 6 (1124)
06:27:38 SRV 8: ** Save file named core for analysis by Progress Software Corporation. (439)
06:27:38 SRV 8: Stopped. (2520)
06:27:38 SRV 7: Stopped. (2520)
06:27:38 SRV 2: Stopped. (2520)
06:27:38 BIW 41: Stopped. (2520)
06:27:38 AIW 42: Stopped. (2520)
06:27:38 APW 36: Stopped. (2520)
06:27:38 APW 37: Stopped. (2520)
06:27:38 APW 39: Stopped. (2520)
06:27:38 APW 38: Stopped. (2520)
06:27:39 BROKER 0: Begin ABNORMAL shutdown code 2 (2249)
06:27:40 BROKER 0: Disconnecting dead server -1. (2525)
06:27:41 BROKER 0: Disconnecting dead server -1. (2525)
06:27:42 BROKER 0: Disconnecting dead server -1. (2525)
06:27:46 WDOG 40: Stopped. (2520)
06:27:47 BROKER : Removed shared memory with segment_id: 131105
06:27:47 BROKER : Removed shared memory with segment_id: 131083
06:27:47 BROKER : Removed shared memory with segment_id: 131121
06:27:47 BROKER : Removed shared memory with segment_id: 131120
06:27:47 BROKER : Removed shared memory with segment_id: 131079
06:27:47 BROKER : Removed shared memory with segment_id: 131081
06:27:47 BROKER : Multi-user session end. (334)
Now we are able to see shared memory status for database which went down only.
PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
393216 9136 3 - /progdata/prd/slcdb1/prdslc.db
393217 9136 4 - /progdata/prd/slcdb1/prdslc.db
You have mail in /usr/spool/mail/pgresdba
please provide your suggestion or thoughts on this issue.
Adding to that two core and protrace files got generated during this issue.
-rw-r--r-- 1 pgresdba progress 698177682 Apr 12 06:27 core.83750.12102729
-rw-r--r-- 1 pgresdba progress 698177682 Apr 12 06:27 core.146828.12102738
-rw-r--r-- 1 pgresdba progress 751 Apr 12 06:27 protrace.62782
-rw-r--r-- 1 pgresdba progress 755 Apr 12 06:27 protrace.583520
-rw-r--r-- 1 pgresdba progress 751 Apr 12 06:27 protrace.51322
-rw-r--r-- 1 pgresdba progress 1005 Apr 12 06:27 protrace.83750
-rw-r--r-- 1 pgresdba progress 1010 Apr 12 06:27 protrace.146828
-rw-r--r-- 1 pgresdba progress 83 Apr 12 06:31 protrace.434066
4/12/18 Status: Shared Memory Segments
07:33:20
Seg Id Size Used Free
1 393225 134217728 134217724 4
2 131125 134217728 134217724 4
3 131124 134217728 134217716 12
4 393216 134217728 134217716 12
5 393217 134217728 120247956 13969772
6 262151 25251840 396 25251444
Enter <return>, R, P, T, or X (? for help):
Faced with this situation I'd run as diagnostics for hardware failures, particularly a faulty disc.
If you find this it, will need to be resolved asap.
Not sure what diagnostics to run on AIX but I'm sure others / the internet will know this.
Then I think your options for the database are either to go back to the latest backup from before the issues started or if this is not an option then do a dump and load to hopefully repair the corruption that has occured in area 6 as per the error.
If there are hardware issues there's not much point doing the above until after these issues are fixed as the same could just happen again.
Finally post
Due to memory violation and connection time out errors user are not able to access mfg/pro applications