Shared Memory segments Full.

Posted by durgaprasad m on 11-Apr-2018 08:05

Hi All,

In our database went down due to memory violation error.I checked in promon > 1> Status: Shared Memory Segments
It shows all memory utilized.
Now database and application up and running but shared memory segments is full for all database.

what action i need to take to resolve this issue.

 Status: Shared Memory Segments

Seg Id Size Used Free

1 393265 134217728 134217724 4
2 393248 134217728 134217724 4
3 393264 134217728 134217716 12
4 393247 134217728 134217716 12
5 393228 134217728 120246044 13971684
6 393263 25251840 396 25251444

Enter <return>, R, P, T, or X (? for help):

19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)
19:08:47 SQLSRV2 3: SYSTEM ERROR: Memory violation. (49)
19:08:50 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)
19:08:57 SQLSRV2 4: SYSTEM ERROR: Memory violation. (49)
19:08:59 SRV 35: Corrupt block detected when attempting to release a buffer. (4232)
19:08:59 SRV 35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)
19:08:59 SRV 35: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block


PROGRESS Version 9.1E as of Tue Oct 12 17:19:22 EDT 2004

PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
131072 - - - (not PROGRESS)
131073 - - - (not PROGRESS)

We tried rebooting server option also but few seconds it is worked fine, again it showing shared memory segments full.

OS: Aix 5.3
Progress version : 9.1e

Application : Mfg/pro 8.3

Posted by gus bjorklund on 11-Apr-2018 11:58

> On Apr 11, 2018, at 9:06 AM, durgaprasad m wrote:

>

> 19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)

Seen these many times before. These crashes are caused by very old bugs in the OpenEdge SQL Server code. You need to upgrade. Version 9.1E04 was released in 2006 and is now 12 years old.

Posted by lbryan on 12-Apr-2018 07:15

Faced with this situation I'd run as diagnostics for hardware failures, particularly a faulty disc.

If you find this it, will need to be resolved asap.

Not sure what diagnostics to run on AIX but I'm sure others / the internet will know this.

Then I think your options for the database are either to go back to the latest backup from before the issues started or if this is not an option then do a dump and load to hopefully repair the corruption that has occured in area 6 as per the error.

If there are hardware issues there's not much point doing the above until after these issues are fixed as the same could just happen again.

Posted by durgaprasad m on 12-Apr-2018 09:33

Finally post

Due to memory violation  and connection time out errors user are not able to access mfg/pro applications

All Replies

Posted by George Potemkin on 11-Apr-2018 08:18

> but shared memory segments is full for all database.

Last segments are NOT full

>  Corrupt block detected when attempting to release a buffer. (4232)

Did you check the system memory during server's reboot? I guess it's a failure of RAM.

Posted by durgaprasad m on 11-Apr-2018 08:32

Present situation of the database

04/11/18        Status: Shared Memory Segments

09:25:07

Seg     Id      Size      Used      Free

 1     22 134217728 134217724         4

 2     23 134217728 134217724         4

 3     24 134217728 134217716        12

 4     25 134217728 134217716        12

 5     26 134217728 120258516  13959212

 6     27  25251840       396  25251444

Total Paging Space   Percent Used

     4608MB               1%

IPCS comand output :

p-10004*:/progdata/prd/slcdb1: ipcs -am

IPC status from /dev/mem as of Wed Apr 11 09:29:21 EDT 2018

T        ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NATTCH     SEGSZ  CPID  LPID   ATIME    DTIME    CTIME

Shared Memory:

m    131072 0x78000033 --rw-rw-rw-     root   system     root   system      1  33554432 41288 434112  6:55:29  9:21:16  2:05:50

m    131073 0x7800002e --rw-rw-rw-     root   system     root   system      1  33554432 41288 434112  6:55:29  9:21:16  2:05:50

m    131074 0x0d05332f --rw-rw----     root   system     root   system    173      1440 16552 432508  9:29:16  9:29:21  2:05:02

m         3 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 40370 40370  2:07:16 no-entry  2:07:16

m         4 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 53958 294206  7:00:03  7:00:03  2:07:16

m         5 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 40006 198430  7:00:02  7:00:02  2:07:16

m         6 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   3678208 54462 54462  2:07:16 no-entry  2:07:16

m         7 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   6307840 40778 365608  8:26:08  8:26:08  2:07:16

m         8 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      1   2830336 44140 44140  2:07:16 no-entry  2:07:16

m         9 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   6426624 41602 357358  8:25:27  8:25:27  2:07:16

m        10 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      4 128491520 56762 130058  6:43:55  6:44:09  2:07:16

m        11 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   6246400 57020 365092  8:26:07  8:26:07  2:07:16

m        12 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     14 134217728 57794 148452  7:00:01  7:00:01  2:07:16

m        13 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     14 134217728 57794 148452  7:00:01  7:00:01  2:07:16

m        14 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     14 134217728 57794 148452  7:00:01  7:00:01  2:07:16

m        15 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     14 134217728 57794 148452  7:00:01  7:00:01  2:07:16

m        16 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     14 126713856 57794 148452  7:00:01  7:00:01  2:07:16

m        17 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 58310 155544  7:00:02  7:00:02  2:07:16

m        18 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 58310 155544  7:00:02  7:00:02  2:07:16

m        19 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 58310 155544  7:00:02  7:00:02  2:07:16

m        20 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9 134217728 58310 155544  7:00:02  7:00:02  2:07:16

m        21 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      9  37187584 58310 155544  7:00:02  7:00:02  2:07:16

m        22 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     36 134217728 45974 485942  9:25:00  9:25:37  2:07:16

m        23 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     36 134217728 45974 485942  9:25:00  9:25:37  2:07:16

m        24 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     36 134217728 45974 485942  9:25:00  9:25:37  2:07:16

m        25 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     36 134217728 45974 485942  9:25:00  9:25:37  2:07:16

m        26 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     36 134217728 45974 485942  9:25:00  9:25:37  2:07:16

m        27 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     36  25251840 45974 485942  9:25:00  9:25:37  2:07:16

m        28 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     27 134217728 56506 310142  9:29:15  9:29:15  2:07:16

m        29 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     27 134217728 56506 310142  9:29:15  9:29:15  2:07:16

m        30 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     27 134217728 56506 310142  9:29:15  9:29:15  2:07:16

m        31 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     27 134217728 56506 310142  9:29:15  9:29:15  2:07:16

m        32 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     27  37175296 56506 310142  9:29:15  9:29:15  2:07:16

m        33 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     47 134217728 23630 472222  9:15:00  9:15:05  2:07:16

m        34 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     47 134217728 23630 472222  9:15:00  9:15:05  2:07:16

m        35 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     47 134217728 23630 472222  9:15:00  9:15:05  2:07:16

m        36 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     47  26300416 23630 472222  9:15:00  9:15:05  2:07:16

m        37 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     52   7151616 58570 476054  9:23:42  9:23:42  2:07:16

m        38 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      5   6672384 59860 160942  4:38:02  4:38:43  2:07:16

m        39 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     30 134217728 42856 450010  9:20:01  9:20:01  2:07:16

m        40 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     30 134217728 42856 450010  9:20:01  9:20:01  2:07:16

m        41 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     30 134217728 42856 450010  9:20:01  9:20:01  2:07:16

m        42 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     30 134217728 42856 450010  9:20:01  9:20:01  2:07:16

m        43 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     30 126652416 42856 450010  9:20:01  9:20:01  2:07:16

m        44 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      5  43663360 60890 263182  4:37:28  4:37:49  2:07:16

m        45 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     22 134217728 59090 478866  9:20:07  9:20:07  2:07:16

m        46 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     22 134217728 59090 478866  9:20:07  9:20:07  2:07:16

m        47 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     22 134217728 59090 478866  9:20:07  9:20:07  2:07:16

m        48 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     22 134217728 59090 478866  9:20:07  9:20:07  2:07:16

m        49 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress     22  38039552 59090 478866  9:20:07  9:20:07  2:07:16

m        50 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      5  41148416 61408 465230  9:05:00  9:05:06  2:07:16

m        51 0xffffffff --rw-r--r-- pgresdba progress pgresdba progress      4   2891776 61924 320412  7:00:02  7:00:02  2:07:16

System configuration: lcpu=6 mem=22528MB ent=2.90

kthr    memory              page              faults              cpu

----- ----------- ------------------------ ------------ -----------------------

r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec

1  0 1821474 70132   0   0   0   0    0   0 425 127514 9687 26 13 60  1  1.43  49.2

0  0 1821324 70097   0   0   0   0    0   0 611 19809 6391  6  7 84  3  0.47  16.2

1  0 1820366 71023   0   0   0   0    0   0 544 76921 4776 10 11 74  5  0.72  24.8

0  0 1819724 71637   0   0   0   0    0   0 305 15837 4518  3  5 91  1  0.30  10.3

1  0 1819998 70824   0   0   0   0    0   0 1070 227439 5869 12 11 77  1  0.81  27.8

PROGRESS Version 9.1E as of Tue Oct 12 17:19:22 EDT 2004

PROGRESS SHARED MEMORY STATUS

     ID ShMemVer Seg# InUse Database

 131072       -    - -      (not PROGRESS)

 131073       -    - -      (not PROGRESS)

*svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'

    Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB

 196652 _sqlsrv2       2858997    34581        0   218297      N     Y     N

 188694 _sqlsrv2       2858248    34577        0   217548      N     Y     N

 105538 _progres       2858101    34569        0   216373      N     N     N

  92372 _progres       2858101    34569        0   216373      N     N     N

 125142 _progres       2858101    34569        0   216373      N     N     N

 106838 _progres       2858101    34569        0   216373      N     N     N

 189660 _sqlsrv2       2858034    34576        0   217334      N     Y     N

 259304 _mprosrv       2856955    34569        0   215892      N     N     N

  58062 _mprosrv       2856940    34569        0   215877      N     N     N

 137794 _mprosrv       2856936    34569        0   215873      N     N     N

 170554 _mprosrv       2856936    34569        0   215873      N     N     N

 386766 _mprosrv       2856935    34569        0   215872      N     N     N

  94514 _mprosrv       2856935    34569        0   215872      N     N     N

 354772 _mprosrv       2856933    34569        0   215870      N     N     N

 412070 _mprosrv       2856933    34569        0   215870      N     N     N

Posted by durgaprasad m on 11-Apr-2018 08:34

svmon -G

              size       inuse        free         pin     virtual

memory      5767168     5693016       74152      275290     1818627

pg space    1179648        5645

              work        pers        clnt       other

pin          199722           0           0       75568

in use      1818627     3305255      569134

Posted by ChUIMonster on 11-Apr-2018 09:08

Your problem is not what you think it is.

Shared  memory use is expected behavior and is completely normal.  It has nothing to do with the memory violation error, the corrupt block or the db crash.

The fact that you are running version 9 is a much bigger problem than the fact that you can display the shared memory segments that are in use.

Version 9 was conceived and coded in the 1990s.  The environments that is was designed to run in have changed considerably.  Many, many things are dramatically different in todays world -- security and multi-core systems memory to name just a couple of the more obvious changes.

Is using v9 responsible for your crash?  Probably not.  It looks more like a problem with faulty hardware. If the hardware is  as old and unsupported as v9 that isn't very surprising.

Posted by durgaprasad m on 11-Apr-2018 09:29

Thanks Tom

yes we know vesion is out of support.

May be in latest version this issue got fixed.

can i know why we are falling into this issue everytime.

Posted by ChUIMonster on 11-Apr-2018 09:35

"everytime"?  You are posting that shared memory is being used.  That is normal.  There is nothing to solve about that.

You have reported one db crash so far.

Have you had more than 1 db crash?  If so - do the log messages always refer to block 717557?

Have you opened a support call about:

19:08:59 SRV 35: Corrupt block detected when attempting to release a buffer. (4232)

19:08:59 SRV 35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)

19:08:59 SRV 35: Writing block 7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)

19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block

Posted by durgaprasad m on 11-Apr-2018 10:09

Hi Tom,

We don't have vendor support and already we contacted for the same with progress already said they are out of support also.

Posted by George Potemkin on 11-Apr-2018 10:11

> 19:08:59 SRV 35: SYSTEM DEBUG: Database buffer block

Post the beginning of block dump.

Posted by durgaprasad m on 11-Apr-2018 10:41

Please find below for our database log history events during the issue

18:52:17 SRV     2: HANGUP signal received. (562)

18:52:17 SRV     2: Previous message sent on behalf of user 905. (5512)

18:52:17 SRV     2: Logout usernum 905, userid y550sxn, on p5lp5 281. (739)

18:52:20 SRV    14: HANGUP signal received. (562)

18:52:20 SRV    14: Previous message sent on behalf of user 1026. (5512)

18:52:20 SRV    14: Logout usernum 1026, userid y550sxn, on p5lp5 210. (739)

18:52:34 SRV     2: Login usernum 1084, userid y550axp, on p5lp5 72. (742)

18:53:26 SRV     2: HANGUP signal received. (562)

18:53:26 SRV     2: Previous message sent on behalf of user 1084. (5512)

18:53:26 SRV     2: Logout usernum 1084, userid y550axp, on p5lp5 72. (739)

18:53:37 SRV     2: Login usernum 1084, userid y550axp, on p5lp5 72. (742)

18:53:38 SRV     2: HANGUP signal received. (562)

18:53:38 SRV     2: Previous message sent on behalf of user 1084. (5512)

18:53:38 SRV     2: Logout usernum 1084, userid y550axp, on p5lp5 72. (739)

18:55:01 SRV     2: Login usernum 1084, userid pgresdba, on p5lp5 batch. (742)

18:55:01 SRV     2: Logout usernum 1084, userid pgresdba, on p5lp5 batch. (739)

18:56:37 SRV     8: Logout usernum 1068, userid cquinn, on mhlctxqad07a. (739)

18:58:00 Usr    44: Login by pgresdba on batch. (452)

18:58:05 SRV    11: HANGUP signal received. (562)

18:58:05 SRV    11: Previous message sent on behalf of user 930. (5512)

18:58:05 SRV    10: HANGUP signal received. (562)

18:58:05 SRV    10: Previous message sent on behalf of user 931. (5512)

18:58:05 SRV    11: Logout usernum 930, userid y020syw, on p5lp5 247. (739)

18:58:05 SRV    10: Begin transaction backout. (2252)

18:58:05 SRV    10: Transaction backout completed. (2253)

18:58:05 SRV    10: Logout usernum 931, userid y020syw, on p5lp5 246. (739)

18:58:25 SRV     2: Login usernum 1084, userid y020syw, on p5lp5 18. (742)

18:58:35 SRV    33: Logout usernum 947, userid eylst, on mhlctxqad02a. (739)

18:58:51 SRV     7: Login usernum 1083, userid y020syw, on p5lp5 60. (742)

18:59:13 SRV    21: HANGUP signal received. (562)

18:59:13 SRV    21: Previous message sent on behalf of user 1007. (5512)

18:59:13 SRV    21: Begin transaction backout. (2252)

18:59:13 SRV    21: Transaction backout completed. (2253)

18:59:13 SRV    21: Logout usernum 1007, userid y550jyw, on p5lp5 179. (739)

18:59:49 SRV    16: Usernum 1034 terminated abnormally. (794)

18:59:49 SRV    16: Begin transaction backout. (2252)

18:59:49 SRV    16: Transaction backout completed. (2253)

18:59:49 SRV    16: Logout usernum 1034, userid aaustin, on mhlctxqad08a. (739)

18:59:51 SRV    21: HANGUP signal received. (562)

18:59:51 SRV    21: Previous message sent on behalf of user 1009. (5512)

18:59:51 SRV    21: Begin transaction backout. (2252)

18:59:51 SRV    21: Transaction backout completed. (2253)

18:59:51 SRV    21: Logout usernum 1009, userid y550jyw, on p5lp5 177. (739)

19:00:00 Usr    46: Login by pgresdba on batch. (452)

19:00:00 SRV     7: Login usernum 1080, userid pgresdba, on p5lp5 batch. (742)

19:00:01 RFUTIL 47: Login by pgresdba on batch. (452)

19:00:01 RFUTIL 47: Logout by pgresdba on batch. (453)

19:00:01 SRV     8: Login usernum 1075, userid pgresdba, on p5lp5 batch. (742)

19:00:01 SRV     8: Logout usernum 1075, userid pgresdba, on p5lp5 batch. (739)

19:00:05 SRV     7: Logout usernum 1080, userid pgresdba, on p5lp5 batch. (739)

19:00:05 Usr    46: Logout by pgresdba on batch. (453)

19:00:30 SRV    24: Logout usernum 1064, userid npassey, on mhlctxqad02a. (739)

19:00:32 Usr    44: Logout by pgresdba on batch. (453)

19:01:35 SRV     7: Login usernum 1080, userid jonewman, on mhlctxqad08a. (742)

19:02:37 SRV     9: Logout usernum 1070, userid mthorne, on mhlctxqad08a. (739)

19:03:26 SRV    17: Logout usernum 1029, userid aaldrich, on MHLCTXQAD03A. (739)

19:05:00 Usr    44: Login by pgresdba on batch. (452)

19:05:00 Usr    46: Login by pgresdba on batch. (452)

19:05:00 Usr    47: Login by pgresdba on batch. (452)

19:05:01 SRV     8: Login usernum 1075, userid pgresdba, on p5lp5 batch. (742)

19:05:01 SRV     8: Logout usernum 1075, userid pgresdba, on p5lp5 batch. (739)

19:05:13 Usr    44: Logout by pgresdba on batch. (453)

19:05:16 Usr    46: Logout by pgresdba on batch. (453)

19:06:07 Usr    47: Logout by pgresdba on batch. (453)

19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)

19:08:47 SQLSRV2 3: SYSTEM ERROR: Memory violation. (49)

19:08:50 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)

19:08:57 SQLSRV2 4: SYSTEM ERROR: Memory violation. (49)

19:08:59 SRV    35: Corrupt block detected when attempting to release a buffer. (4232)

19:08:59 SRV    35: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)

19:08:59 SRV    35: Writing block  7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)

19:08:59 SRV    35: SYSTEM DEBUG: Database buffer block

19:08:59 SRV    35: pbktbl = 0x3050bb3c

19:08:59 SRV    35: pbktbl->qself = 0x5850bb3c

19:08:59 SRV    35: XBKBUF(pbktbl->qself) = 0x3050bb3c

19:08:59 SRV    35: pbktbl->bt_qbuf = 0x642be444

19:08:59 SRV    35: XBKBUF(pbktbl->bt_qbuf) = 0x442be444

19:08:59 SRV    35: pbkbuf = 0x442be444

19:08:59 SRV    35: Block dbkey = 0x1b5b7980   bt_offset = 0x0

19:08:59 SRV    35: File: /progdata/prd/slcdb6/prdslc.d55

19:08:59 SRV    35: 0000:  5846 624c 0000 08e4 5850 bbec 642b e444

19:08:59 SRV    35: 0010:  0000 0001 0100 038a 0000 a26b 0019 0000

19:08:59 SRV    35: 0020:  1b5b 7980 0000 0006 67ff f3f8 0000 0000

19:08:59 SRV    35: 0030:  0000 0000 3dc3 8000 ffff ffff 0000 0000

19:08:59 SRV    35: 0040:  0000 0000 0000 0000 0000 0000 584d 49dc

19:08:59 SRV    35: 0050:  5878 ea4c 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0060:  0000 0000 0000 0000 0001 0100 0000 0000

19:08:59 SRV    35: 0070:  1e7f 0000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0080:  0000 0000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0090:  0000 0000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 00a0:  5850 bb3c 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: Block buffer dump follows, from address 0x442be444:

19:08:59 SRV    35: 0000:  00ff fa00 05fd fdfd fdff 0000 0000 4fe7

19:08:59 SRV    35: 0010:  0002 0042 0200 ea08 3135 3330 3031 3530

19:08:59 SRV    35: 0020:  0431 3030 3608 3135 3331 3031 3530 0004

19:08:59 SRV    35: 0030:  8223 329f 0e49 4330 3030 3532 3534 3030

19:08:59 SRV    35: 0040:  3330 3203 1d94 b306 4953 532d 5452 0000

19:08:59 SRV    35: 0050:  0102 0101 0000 0401 ac86 6500 0000 0114

19:08:59 SRV    35: 0060:  e700 0e00 3800 6900 9700 ba00 ca00 f101

19:08:59 SRV    35: 0070:  0102 008c 0735 3539 3531 3530 0247 6efd

19:08:59 SRV    35: 0080:  0649 5353 2d54 5207 7265 7475 726e 7302

19:08:59 SRV    35: 0090:  806f 0380 153f 0000 0002 4541 0247 6e08

19:08:59 SRV    35: 00a0:  636c 696e 6970 6164 0000 0000 0000 0585

19:08:59 SRV    35: 00b0:  3851 008f 0486 0025 5304 8602 3625 0687

19:08:59 SRV    35: 00c0:  3888 0845 2f03 1d94 b304 8223 329f 0000

19:08:59 SRV    35: 00d0:  0739 3338 3631 312a 00fd 0200 6f07 7935

19:08:59 SRV    35: 00e0:  3530 7078 7208 3336 616b 3030 3333 0247

19:08:59 SRV    35: 00f0:  6e04 4136 3536 0000 0000 0000 0003 5553

19:08:59 SRV    35: 0100:  4402 801f 0002 6beb 0587 3445 872f 0433

19:08:59 SRV    35: 0110:  3630 3008 7175 6172 6e6d 6f76 0002 48c2

19:08:59 SRV    35: 0120:  0000 0000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0130:  00fd fdfd fdfd 0000 0000 0000 0000 0214

19:08:59 SRV    35: 0140:  700a 6963 6c6f 7472 3032 2e70 0004 3336

19:08:59 SRV    35: 0150:  3030 0000 00fd 00fa 0005 0000 0000 ff00

19:08:59 SRV    35: 0160:  0000 0000 0000 0000 0000 0000 fd00 0000

19:08:59 APW    39: Stopped. (2520)

19:08:59 SRV    35: 0170:  0000 fd00 00ea e700 0e00 3100 4d00 7400

19:08:59 SRV    35: 0180:  9000 a000 c700 d702 008c 0937 3030 3032

19:08:59 SRV    35: 0190:  3442 3031 0247 6efd 0652 4354 2d54 5205

19:08:59 SRV    35: 01a0:  5354 4147 4500 0380 107f 0003 8010 0f00

19:08:59 SRV    35: 01b0:  0245 4102 476e 0000 0000 0000 0003 8223

19:08:59 SRV    35: 01c0:  7200 0003 8223 7203 1d94 b203 8023 7200

19:08:59 SRV    35: 01d0:  0000 00fd 0380 100f 0779 3535 3062 766b

19:08:59 SRV    35: 01e0:  0002 476e 0441 3035 3400 0000 0000 0000

19:08:59 SRV    35: 01f0:  0355 5344 0280 1f00 026b dd00 0433 3031

19:08:59 SRV    35: 0200:  3208 494e 2053 5441 4745 00fd 0000 0000

19:08:59 SRV    35: 0210:  0000 0000 0000 0000 0000 0000 00fd fdfd

19:08:59 SRV    35: 0220:  fdfd 0000 0000 0000 0000 0214 700a 6963

19:08:59 SRV    35: 0230:  6c6f 7472 3032 2e70 0004 3330 3132 0000

19:08:59 SRV    35: 0240:  00fd 00fa 0005 0000 0000 ff00 0000 0000

19:08:59 SRV    35: 0250:  0000 0000 0000 0000 fd00 0000 0000 fd00

19:08:59 SRV    35: 0260:  004e e700 0200 4102 00ea 0831 3533 3130

19:08:59 SRV    35: 0270:  3135 3000 0831 3531 3130 3135 3004 3130

19:08:59 SRV    35: 0280:  3030 0380 2372 0e49 4330 3030 3532 3534

19:08:59 SRV    35: 0290:  3031 3230 3103 1d94 b206 5243 542d 5452

19:08:59 SRV    35: 02a0:  0000 0107 0101 0000 0401 ac86 6200 0000

19:08:59 SRV    35: 02b0:  004e e700 0200 4102 00ea 0831 3531 3130

19:08:59 SRV    35: 02c0:  3135 3004 3130 3030 0831 3533 3130 3135

19:08:59 SRV    35: 02d0:  3000 0380 2372 0e49 4330 3030 3532 3534

19:08:59 SRV    35: 02e0:  3031 3230 3103 1d94 b106 4953 532d 5452

19:08:59 SRV    35: 02f0:  0000 0106 0105 0000 0401 ac86 5e00 0000

19:08:59 SRV    35: 0300:  00eb e700 0e00 3200 4e00 7500 9100 a100

19:08:59 SRV    35: 0310:  c800 d802 008c 0937 3030 3032 3442 3031

19:08:59 SRV    35: 0320:  0247 6efd 0649 5353 2d54 5203 5152 4d03

19:08:59 SRV    35: 0330:  8010 0f03 8020 7f00 0300 100f 0002 4541

19:08:59 SRV    35: 0340:  0247 6e00 0000 0000 0000 0382 2372 0000

19:08:59 SRV    35: 0350:  0382 2372 031d 94b1 0380 2372 0000 0000

19:08:59 SRV    35: 0360:  fd03 0010 0f07 7935 3530 6276 6b00 0247

19:08:59 SRV    35: 0370:  6e04 4130 3534 0000 0000 0000 0003 5553

19:08:59 SRV    35: 0380:  4402 801f 0002 6bdd 0004 3330 3132 0849

19:08:59 SRV    35: 0390:  4e20 5354 4147 4500 fd00 0000 0000 0000

19:08:59 SRV    35: 03a0:  0000 0000 0000 0000 0000 fdfd fdfd fd00

19:08:59 SRV    35: 03b0:  0000 0000 0000 0002 1470 0a69 636c 6f74

19:08:59 SRV    35: 03c0:  7230 322e 7000 0433 3031 3200 0000 fd00

19:08:59 SRV    35: 03d0:  fa00 0500 0000 00ff 0000 0000 0000 0000

19:08:59 SRV    35: 03e0:  0000 0000 00fd 0000 0000 00fd 0000 4fe7

19:08:59 SRV    35: 03f0:  0002 0042 0200 ea08 3135 3331 3031 3530

19:08:59 SRV    35: 0400:  0008 3135 3330 3031 3530 0431 3030 3604

19:08:59 SRV    35: 0410:  8211 664f 0e49 4330 3030 3532 3534 3030

19:08:59 SRV    35: 0420:  3330 3203 1d94 b006 5243 542d 5452 0000

19:08:59 SRV    35: 0430:  0104 0103 0000 0401 ac86 5b00 0000 0062

19:08:59 SRV    35: 0440:  e700 0400 3b00 5801 3b05 3738 3130 3804

19:08:59 SRV    35: 0450:  3738 3430 0257 4f02 4750 0247 4f04 8211

19:08:59 SRV    35: 0460:  164f 0e57 4f30 3030 3432 3436 3337 3730

19:08:59 SRV    35: 0470:  3100 0007 7935 3530 6c78 6200 0000 0355

19:08:59 SRV    35: 0480:  5344 0004 3130 3130 0101 0280 1f00 0000

19:08:59 SRV    35: 0490:  0000 0001 3103 3135 3000 0002 801f 0000

19:08:59 SRV    35: 04a0:  0000 0062 e700 0400 3b00 5801 3b05 3738

19:08:59 SRV    35: 04b0:  3939 3704 3738 3638 0257 4f02 4750 0247

19:08:59 SRV    35: 04c0:  4b04 0212 055f 0e57 4f30 3030 3432 3036

19:08:59 SRV    35: 04d0:  3337 3730 3100 0007 7935 3530 6c78 6200

19:08:59 SRV    35: 04e0:  0000 0355 5344 0004 3130 3130 0108 0280

19:08:59 SRV    35: 04f0:  1f00 0000 0000 0001 3103 3135 3000 0002

19:08:59 SRV    35: 0500:  801f 0000 0000 00f5 e700 0e00 3a00 5c00

19:08:59 SRV    35: 0510:  8100 9500 a500 d200 e202 008c 0730 3630

19:08:59 SRV    35: 0520:  3035 3234 0247 5002 4750 064f 5244 2d53

19:08:59 SRV    35: 0530:  4f00 0002 8074 0280 2f00 0002 4541 0247

19:08:59 SRV    35: 0540:  4f08 5331 3234 3332 3537 0853 3132 3433

19:08:59 SRV    35: 0550:  3235 3700 0831 3030 3833 3338 3200 0000

19:08:59 SRV    35: 0560:  0484 4097 3900 0003 8019 7f03 1bb1 8300

19:08:59 SRV    35: 0570:  0000 0000 fd00 0779 3535 3070 7877 0002

19:08:59 SRV    35: 0580:  4750 0441 3232 3400 0000 0001 0100 0003

19:08:59 SRV    35: 0590:  5553 4402 801f 0002 7a9a 0004 3336 3030

19:08:59 SRV    35: 05a0:  0000 fd00 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 05b0:  0000 0000 fdfd fdfd fd00 0000 0000 0000

19:08:59 SRV    35: 05c0:  0000 0873 6f73 6f6d 742e 7000 0000 0000

19:08:59 SRV    35: 05d0:  fd00 fa00 1308 5435 3534 3134 3131 0444

19:08:59 SRV    35: 05e0:  4953 5400 0247 47ff 0000 0000 0000 0000

19:08:59 SRV    35: 05f0:  0000 0000 00fd 0000 0000 00fd 0000 a8e7

19:08:59 SRV    35: 0600:  000c 001c 0038 004e 005e 0085 0097 0153

19:08:59 SRV    35: 0610:  0303 fe23 0444 4f57 4e02 474b 0154 0000

19:08:59 SRV    35: 0620:  0002 8115 0000 0000 0000 0003 8212 2606

19:08:59 SRV    35: 0630:  3839 3939 3931 0000 0000 0000 0003 8212

19:08:59 SRV    35: 0640:  2600 0000 0000 0000 0437 3836 3800 0000

19:08:59 SRV    35: 0650:  0002 4750 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0660:  0000 0000 0000 0000 0000 fdfd 0000 0004

19:08:59 SRV    35: 0670:  3336 3030 0a73 666f 7074 7230 342e 7002

19:08:59 SRV    35: 0680:  7a93 0779 3535 306c 7862 0000 0000 0000

19:08:59 SRV    35: 0690:  0000 0000 0000 0000 0000 0280 1f00 0000

19:08:59 SRV    35: 06a0:  0000 00fd 0000 0000 53e7 0002 0047 0200

19:08:59 SRV    35: 06b0:  af08 3738 3939 3731 3530 0437 3836 3808

19:08:59 SRV    35: 06c0:  3738 3130 3831 3530 0437 3836 3803 03fe

19:08:59 SRV    35: 06d0:  2303 8212 260e 574f 3030 3034 3230 3633

19:08:59 SRV    35: 06e0:  3737 3031 084c 4252 2d32 3030 3000 0001

19:08:59 SRV    35: 06f0:  0801 0700 0003 0147 6f00 0000 00a8 e700

19:08:59 SRV    35: 0700:  0c00 1c00 3800 4e00 5e00 8500 9701 5303

19:08:59 SRV    35: 0710:  03fe 2204 444f 574e 0247 4b01 5400 0000

19:08:59 SRV    35: 0720:  0281 1500 0000 0000 0000 0382 1226 0638

19:08:59 SRV    35: 0730:  3939 3939 3100 0000 0000 0000 0382 1226

19:08:59 SRV    35: 0740:  0000 0000 0000 0004 3738 3638 0000 0000

19:08:59 SRV    35: 0750:  0247 5000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0760:  0000 0000 0000 0000 00fd fd00 0000 0433

19:08:59 SRV    35: 0770:  3630 300a 7366 6f70 7472 3034 2e70 027a

19:08:59 SRV    35: 0780:  8e07 7935 3530 6c78 6200 0000 0000 0000

19:08:59 SRV    35: 0790:  0000 0000 0000 0000 0002 801f 0000 0000

19:08:59 SRV    35: 07a0:  0000 fd00 0000 0053 e700 0200 4702 00af

19:08:59 SRV    35: 07b0:  0837 3839 3937 3135 3004 3738 3638 0837

19:08:59 SRV    35: 07c0:  3831 3038 3135 3004 3738 3638 0303 fe22

19:08:59 SRV    35: 07d0:  0382 1226 0e57 4f30 3030 3432 3036 3337

19:08:59 SRV    35: 07e0:  3730 3108 4c42 522d 3230 3030 0000 0108

19:08:59 SRV    35: 07f0:  0107 0000 0301 476c 0000 0000 a8e7 000c

19:08:59 SRV    35: 0800:  001c 0038 004e 005e 0085 0097 0153 0303

19:08:59 SRV    35: 0810:  fe21 0444 4f57 4e02 474b 0154 0000 0002

19:08:59 SRV    35: 0820:  8115 0000 0000 0000 0003 8212 2606 3839

19:08:59 SRV    35: 0830:  3939 3931 0000 0000 0000 0003 8212 2600

19:08:59 SRV    35: 0840:  0000 0000 0000 0437 3836 3800 0000 0002

19:08:59 SRV    35: 0850:  4750 0000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0860:  0000 0000 0000 0000 fdfd 0000 0004 3336

19:08:59 SRV    35: 0870:  3030 0a73 666f 7074 7230 342e 7002 7a87

19:08:59 SRV    35: 0880:  0779 3535 306c 7862 0000 0000 0000 0000

19:08:59 SRV    35: 0890:  0000 0000 0000 0000 0280 1f00 0000 0000

19:08:59 SRV    35: 08a0:  00fd 0000 0000 53e7 0002 0047 0200 af08

19:08:59 SRV    35: 08b0:  3738 3939 3731 3530 0437 3836 3808 3738

19:08:59 BROKER  0: Begin ABNORMAL shutdown code 2 (2249)

19:08:59 SRV    35: 08c0:  3130 3831 3530 0437 3836 3803 03fe 2103

19:08:59 SRV    35: 08d0:  8212 260e 574f 3030 3034 3230 3633 3737

19:08:59 SRV    35: 08e0:  3031 084c 4252 2d32 3030 3000 0001 0801

19:08:59 SRV    35: 08f0:  0700 0003 0147 6a00 0000 00a8 e700 0c00

19:08:59 SRV    35: 0900:  1c00 3800 4e00 5e00 8500 9701 5303 03fe

19:08:59 SRV    35: 0910:  2004 444f 574e 0247 4b01 5400 0000 0281

19:08:59 SRV    35: 0920:  1500 0000 0000 0000 0382 1226 0638 3939

19:08:59 SRV    35: 0930:  3939 3100 0000 0000 0000 0382 1226 0000

19:08:59 SRV    35: 0940:  0000 0000 0004 3738 3638 0000 0000 0247

19:08:59 SRV    35: 0950:  5000 0000 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0960:  0000 0000 0000 00fd fd00 0000 0433 3630

19:08:59 SRV    35: 0970:  300a 7366 6f70 7472 3034 2e70 027a 8207

19:08:59 SRV    35: 0980:  7935 3530 6c78 6200 0000 0000 0000 0000

19:08:59 SRV    35: 0990:  0000 0000 0000 0002 801f 0000 0000 0000

19:08:59 SRV    35: 09a0:  fd00 0000 0053 e700 0200 4702 00af 0837

19:08:59 SRV    35: 09b0:  3839 3937 3135 3004 3738 3638 0837 3831

19:08:59 SRV    35: 09c0:  3038 3135 3004 3738 3638 0303 fe20 0382

19:08:59 SRV    35: 09d0:  1226 0e57 4f30 3030 3432 3036 3337 3730

19:08:59 SRV    35: 09e0:  3108 4c42 522d 3230 3030 0000 0108 0107

19:08:59 SRV    35: 09f0:  0000 0301 4767 0000 0000 e9e7 000e 002e

19:08:59 SRV    35: 0a00:  004c 007a 0092 00a2 00c6 00d6 0200 8c07

19:08:59 SRV    35: 0a10:  3535 3433 3135 3002 4750 fd07 5243 542d

19:08:59 SRV    35: 0a20:  4348 4c04 7034 626e 0003 8013 0000 0280

19:08:59 SRV    35: 0a30:  5500 0245 4102 4750 0000 0000 0000 0004

19:08:59 SRV    35: 0a40:  8415 3355 0000 0003 1bb1 8200 0000 0732

19:08:59 SRV    35: 0a50:  3033 3530 3439 00fd 0280 5507 7935 3530

19:08:59 SRV    35: 0a60:  7467 6808 3832 6367 3030 3937 0247 5004

19:08:59 SRV    35: 0a70:  4136 3136 0000 0000 0000 0003 5553 4402

19:08:59 SRV    35: 0a80:  801f 0002 7a69 0004 3336 3030 0266 6700

19:08:59 SRV    35: 0a90:  0248 a400 0000 0000 0000 0000 0000 0000

19:08:59 SRV    35: 0aa0:  0000 0000 fdfd fdfd fd00 0000 0000 0000

19:08:59 SRV    35: 0ab0:  0000 0969 6369 746c 6d74 2e70 0004 3336

19:08:59 SRV    35: 0ac0:  3030 0000 00fd 00fa 0005 0000 0000 ff00

19:08:59 SRV    35: 0ad0:  0000 0000 0000 0000 0000 0000 fd00 0000

19:08:59 SRV    35: 0ae0:  0000 fd00 0028 e700 0200 1b02 00ea 0000

19:08:59 SRV    35: 0af0:  0000 0000 031b b182 0752 4354 2d43 484c

19:08:59 SRV    35: 0b00:  0000 0000 0000 0401 ac86 5700 0000 0028

19:08:59 SRV    35: 0b10:  e700 0200 1b02 00ea 0000 0000 0000 031b

19:08:59 SRV    35: 0b20:  b181 0749 5353 2d43 484c 0000 0000 0000

19:08:59 SRV    35: 0b30:  0401 ac86 5400 0000 00ef e700 0e00 2e00

19:08:59 SRV    35: 0b40:  4c00 7a00 9800 a800 cc00 dc02 008c 0735

19:08:59 SRV    35: 0b50:  3534 3331 3530 0247 50fd 0749 5353 2d43

19:08:59 SRV    35: 0b60:  484c 0470 3462 6e02 8055 0380 1355 0000

19:08:59 SRV    35: 0b70:  0002 4541 0247 5000 0000 0000 0000 0484

19:08:59 SRV    35: 0b80:  1533 5500 0000 031b b181 0000 0007 3230

19:08:59 SRV    35: 0b90:  3335 3034 3900 fd02 0055 0779 3535 3074

19:08:59 SRV    35: 0ba0:  6768 0838 3263 6730 3039 3702 4750 0441

19:08:59 BIW    41: Stopped. (2520)

9:08:59 SRV    35: 0bb0:  3631 3600 0000 0000 0000 0355 5344 0280

19:08:59 SRV    35: 0bc0:  1f00 027a 6900 0433 3630 3008 5141 2051

19:08:59 SRV    35: 0bd0:  5541 524e 0002 48a4 0000 0000 0000 0000

19:08:59 SRV    35: 0be0:  0000 0000 0000 0000 00fd fdfd fdfd 0000

19:08:59 SRV    35: 0bf0:  0000 0000 0000 0009 6963 6974 6c6d 742e

19:08:59 SRV    35: 0c00:  7000 0433 3630 3000 0000 fd00 fa00 0500

19:08:59 SRV    35: Stopped. (2520)

19:08:59 SRV     9: Stopped. (2520)

19:08:59 SRV     7: Stopped. (2520)

19:08:59 SRV    16: Stopped. (2520)

19:08:59 SRV    10: Stopped. (2520)

19:08:59 SRV    30: Stopped. (2520)

19:08:59 SRV     2: Stopped. (2520)

19:08:59 SRV    11: Stopped. (2520)

19:08:59 SRV    20: Stopped. (2520)

19:08:59 SRV     8: Stopped. (2520)

19:08:59 SRV    24: Stopped. (2520)

19:08:59 SRV    13: Stopped. (2520)

19:08:59 SRV    12: Stopped. (2520)

19:08:59 SRV    17: Stopped. (2520)

19:08:59 SRV    25: Stopped. (2520)

19:08:59 SRV    14: Stopped. (2520)

19:08:59 SRV    28: Stopped. (2520)

19:08:59 SRV    18: Stopped. (2520)

19:08:59 SRV    15: Stopped. (2520)

19:08:59 SRV    21: Stopped. (2520)

19:08:59 SRV    29: Stopped. (2520)

19:08:59 SRV    19: Stopped. (2520)

19:08:59 APW    36: Stopped. (2520)

19:08:59 AIW    42: Stopped. (2520)

19:08:59 SRV    32: Stopped. (2520)

19:08:59 SRV    22: Stopped. (2520)

19:08:59 SRV    23: Stopped. (2520)

19:08:59 SRV    33: Stopped. (2520)

19:08:59 SRV    26: Stopped. (2520)

19:08:59 SRV    31: Stopped. (2520)

19:08:59 SRV    27: Stopped. (2520)

19:08:59 SRV    34: Stopped. (2520)

19:08:59 APW    37: Stopped. (2520)

19:08:59 APW    38: Stopped. (2520)

19:09:00 WDOG   40: Stopped. (2520)

19:09:01 BROKER   : Removed shared memory with segment_id: 262191

19:09:01 BROKER   : Removed shared memory with segment_id: 262156

19:09:01 BROKER   : Removed shared memory with segment_id: 262175

19:09:01 BROKER   : Removed shared memory with segment_id: 262192

19:09:01 BROKER   : Removed shared memory with segment_id: 262176

19:09:01 BROKER   : Removed shared memory with segment_id: 262193

19:09:01 BROKER   : Multi-user session end. (334)

19:09:24 Usr    54: Logout by pgresdba on batch. (453)

19:09:29 Usr    50: Logout by pgresdba on batch. (453)

19:09:33 Usr    43: Logout by pgresdba on batch. (453)

19:09:33 Usr    48: Logout by pgresdba on batch. (453)

19:09:33 Usr    49: Logout by pgresdba on batch. (453)

19:09:42 Usr    45: Logout by pgresdba on batch. (453)

Posted by gus bjorklund on 11-Apr-2018 11:58

> On Apr 11, 2018, at 9:06 AM, durgaprasad m wrote:

>

> 19:08:39 SQLSRV2 6: SYSTEM ERROR: Memory violation. (49)

Seen these many times before. These crashes are caused by very old bugs in the OpenEdge SQL Server code. You need to upgrade. Version 9.1E04 was released in 2006 and is now 12 years old.

Posted by George Potemkin on 11-Apr-2018 12:04

19:08:59 SRV    35: Block buffer dump follows, from address 0x442be444:
19:08:59 SRV    35: 0000:  00ff fa00 05fd fdfd fdff 0000 0000 4fe7
19:08:59 SRV    35: 0010:  0002 0042 0200 ea08 3135 3330 3031 3530
19:08:59 SRV    35: 0020:  0431 3030 3608 3135 3331 3031 3530 0004
19:08:59 SRV    35: 0030:  8223 329f 0e49 4330 3030 3532 3534 3030
19:08:59 SRV    35: 0040:  3330 3203 1d94 b306 4953 532d 5452 0000

It's indeed a Progress data block but it's not a beginning of block (as it should be).

For example:

08 3135 3330 3031 3530

means a field value of 8 characters: "15300150"

Conclusion: some process left in db buffer pool the correct data but at wrong offset. I would suspect the processes that issued the memory violation error.

Posted by durgaprasad m on 11-Apr-2018 13:01

There is no option to release shared memory segments from the server level.

Posted by cjbrandt on 11-Apr-2018 13:32

I thought newer versions of the utilities would clean up shared memory segments ?  I thought I remember that being mentioned in the past.  

Been a long time since we had to run ipcrm.

Posted by ChUIMonster on 11-Apr-2018 14:09

> There is no option to release shared memory segments from the server level.

This problem has nothing to do with shared memory segments.

You seem to think that  "Memory violation" has sometime to do with "full shared memory segments".  As if you think they are running out of space or something.

Nothing like that is  happening.

It is completely and perfectly normal for your shared memory segments to look the way they do.  When the db starts up it allocates and initializes shared memory to store  -B and various other data structures.  The OS may limit the size of any one chunk of shared memory - thus there can be multiple "segments".  Ideally they will all be full.  That way you are not wasting any.  But depending on how much you have asked for there can be rounding of sizes and Progress reserves a bit extra for some things.  So they are not all always completely full.  But there is nothing wrong with any of the shared memory utilization data that you have shown.

It is normal.

You do not have a problem with full shared memory segments.

A "memory violation" is not a capacity problem.  It is an *access* problem.  The db server attempted to access a memory location that it does not have permission to access.  The location that it attempted to access *might*  have been in shared memory.  Or it might not have -- we don't know, the bogus address is not reported in the error message.

As George points out there is a bad address in the corrupt block that is reported *later*.  That may, or may not, be a useful clue.

As Gus points out the SQL92 engine in v9 has many bugs of this sort.

Your best course of action is to upgrade to a modern release.

If the hardware and operating system are as old as v9 then you should also make sure that they get upgraded.

Your worst course of action is to continue to think that "full shared memory segments" is in any way related to your db crash.

As far as "root cause" goes -- it was probably either a hardware problem or one of those bugs that Gus refers to.  If it is hardware then I would expect it to continue to happen and that errors will appear in the system logs.  But it may not be obvious that they are related to this.

If it is a Progress bug you are pretty much out of luck.  Support for v9 mostly consists of sympathy for your predicament.

Posted by durgaprasad m on 12-Apr-2018 06:31

Hi Tom,

Today also we received same memory violation error in our database log and database went down.

Again we restarted database its came up after that 2 mins mfg/pro application received "server or system ha no more resource" connect .

After few mins issue resolved on it owns mfg/pro application we are able to connect.

06:26:49 SRV     7: vv_flush:I/O error 5 on fd 1

06:26:49 SRV     7: Previous message sent on behalf of user 1071. (5512)

06:26:49 SRV     7: HANGUP signal received. (562)

06:26:49 SRV     7: Previous message sent on behalf of user 1071. (5512)

06:26:49 SRV     7: Begin transaction backout. (2252)

06:26:49 SRV     7: Transaction backout completed. (2253)

06:26:49 SRV     7: Logout usernum 1071, userid y020txe, on p5lp5 122. (739)

06:26:50 SRV     7: HANGUP signal received. (562)

06:26:50 SRV     7: Previous message sent on behalf of user 1072. (5512)

06:26:50 SRV     7: Logout usernum 1072, userid y020txe, on p5lp5 51. (739)

06:27:28 SQLSRV2 5: SYSTEM ERROR: Memory violation. (49)

06:27:29 SRV     6: SYSTEM ERROR: Attempt to read block 0 which does not exist. (210)

06:27:29 SRV     6: ** Save file named core for analysis by Progress Software Corporation. (439)

06:27:29 SRV     6: Logout usernum 1085, userid y020txe, on p5lp5 42. (739)

06:27:29 SRV     6: Logout usernum 1081, userid fsobien, on mhlctxqad07a. (739)

06:27:29 SRV     6: Logout usernum 1083, userid y020axh, on p5lp5 1. (739)

06:27:29 SRV     6: Begin transaction backout. (2252)

06:27:29 SRV     6: Transaction backout completed. (2253)

06:27:29 SRV     6: Logout usernum 1080, userid rhughes, on mhlctxqad08a. (739)

06:27:29 SRV     6: Logout usernum 1082, userid y020dyv, on p5lp5 123. (739)

06:27:29 SRV     6: Stopped. (2520)

06:27:38 SRV     8: Corrupt block detected when attempting to release a buffer. (4232)

06:27:38 SRV     8: bmReleaseBuffer: Error occurred in area 6, block number: 7171557, extent: /progdata/prd/slcdb1/prdslc.d1. (10560)

06:27:38 SRV     8: Writing block  7171557 to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)

06:27:38 SRV     8: SYSTEM DEBUG: Database buffer block

06:27:38 SRV     8: pbktbl = 0x30a0401c

06:27:38 SRV     8: pbktbl->qself = 0x58a0401c

06:27:38 SRV     8: XBKBUF(pbktbl->qself) = 0x30a0401c

06:27:38 SRV     8: pbktbl->bt_qbuf = 0x72a30bb4

06:27:38 SRV     8: XBKBUF(pbktbl->bt_qbuf) = 0x62a30bb4

06:27:38 SRV     8: pbkbuf = 0x62a30bb4

06:27:38 SRV     8: Block dbkey = 0x1b5b7980   bt_offset = 0x0

06:27:38 SRV     8: File: /progdata/prd/slcdb6/prdslc.d55

06:27:38 SRV     8: 0000:  0000 0000 0000 08e4 58a0 40cc 72a3 0bb4

06:27:38 SRV     8: 0010:  0000 0001 0100 042d 0000 23fc 001b 0000

06:27:38 SRV     8: 0020:  1b5b 7980 0000 0006 67ff f3f8 0000 0000

06:27:38 SRV     8: 0030:  0000 0000 3dc3 8000 ffff ffff 0000 0000

06:27:38 SRV     8: 0040:  0000 0000 0000 0000 0000 0000 58a2 d62c

06:27:38 SRV     8: 0050:  5859 3c8c 0000 0000 0000 0000 0000 0000

06:27:38 SRV     8: 0060:  0000 0000 0000 0000 0001 0100 0000 0000

06:27:38 SRV     8: 0070:  006a 0000 0000 0000 0000 0000 0000 0000

06:27:38 SRV     8: 0080:  0000 0000 0000 0000 0000 0000 0000 0000

06:27:38 SRV     8: 0090:  0000 0000 0000 0000 0000 0000 0000 0000

06:27:38 SRV     8: 00a0:  58a0 401c 0000 0000 0000 0000 0000 0000

06:27:38 SRV     8: Block buffer dump follows, from address 0x62a30bb4:

06:27:38 SRV     8: 0000:  0401 ab6f 5e06 4953 532d 534f 0000 0102

....

06:27:38 SRV     8: 1ff0:  487f 0000 0004 01a6 8ae5 0000 0000 00fd

06:27:38 SRV     8: End of buffer

06:27:38 SRV     8: SYSTEM ERROR: wrong dbkey in block. Found 67218287, should be 458979712 in area 6 (1124)

06:27:38 SRV     8: ** Save file named core for analysis by Progress Software Corporation. (439)

06:27:38 SRV     8: Stopped. (2520)

06:27:38 SRV     7: Stopped. (2520)

06:27:38 SRV     2: Stopped. (2520)

06:27:38 BIW    41: Stopped. (2520)

06:27:38 AIW    42: Stopped. (2520)

06:27:38 APW    36: Stopped. (2520)

06:27:38 APW    37: Stopped. (2520)

06:27:38 APW    39: Stopped. (2520)

06:27:38 APW    38: Stopped. (2520)

06:27:39 BROKER  0: Begin ABNORMAL shutdown code 2 (2249)

06:27:40 BROKER  0: Disconnecting dead server -1. (2525)

06:27:41 BROKER  0: Disconnecting dead server -1. (2525)

06:27:42 BROKER  0: Disconnecting dead server -1. (2525)

06:27:46 WDOG   40: Stopped. (2520)

06:27:47 BROKER   : Removed shared memory with segment_id: 131105

06:27:47 BROKER   : Removed shared memory with segment_id: 131083

06:27:47 BROKER   : Removed shared memory with segment_id: 131121

06:27:47 BROKER   : Removed shared memory with segment_id: 131120

06:27:47 BROKER   : Removed shared memory with segment_id: 131079

06:27:47 BROKER   : Removed shared memory with segment_id: 131081

06:27:47 BROKER   : Multi-user session end. (334)

Now we are able to see shared memory status for database which went down only.

PROGRESS SHARED MEMORY STATUS

     ID ShMemVer Seg# InUse Database

 393216     9136    3 -     /progdata/prd/slcdb1/prdslc.db

 393217     9136    4 -     /progdata/prd/slcdb1/prdslc.db

You have mail in /usr/spool/mail/pgresdba

please provide your suggestion or thoughts on this issue.

Posted by durgaprasad m on 12-Apr-2018 06:38

Adding to that two core and protrace files got generated during this issue.

-rw-r--r--    1 pgresdba progress  698177682 Apr 12 06:27 core.83750.12102729

-rw-r--r--    1 pgresdba progress  698177682 Apr 12 06:27 core.146828.12102738

-rw-r--r--    1 pgresdba progress        751 Apr 12 06:27 protrace.62782

-rw-r--r--    1 pgresdba progress        755 Apr 12 06:27 protrace.583520

-rw-r--r--    1 pgresdba progress        751 Apr 12 06:27 protrace.51322

-rw-r--r--    1 pgresdba progress       1005 Apr 12 06:27 protrace.83750

-rw-r--r--    1 pgresdba progress       1010 Apr 12 06:27 protrace.146828

-rw-r--r--    1 pgresdba progress         83 Apr 12 06:31 protrace.434066

4/12/18        Status: Shared Memory Segments

07:33:20

Seg     Id      Size      Used      Free

 1 393225 134217728 134217724         4

 2 131125 134217728 134217724         4

 3 131124 134217728 134217716        12

 4 393216 134217728 134217716        12

 5 393217 134217728 120247956  13969772

 6 262151  25251840       396  25251444

Enter <return>, R, P, T, or X (? for help):

Posted by lbryan on 12-Apr-2018 07:15

Faced with this situation I'd run as diagnostics for hardware failures, particularly a faulty disc.

If you find this it, will need to be resolved asap.

Not sure what diagnostics to run on AIX but I'm sure others / the internet will know this.

Then I think your options for the database are either to go back to the latest backup from before the issues started or if this is not an option then do a dump and load to hopefully repair the corruption that has occured in area 6 as per the error.

If there are hardware issues there's not much point doing the above until after these issues are fixed as the same could just happen again.

Posted by durgaprasad m on 12-Apr-2018 09:33

Finally post

Due to memory violation  and connection time out errors user are not able to access mfg/pro applications

This thread is closed