Container failure: no space left on device

Posted by appmath on 28-Jun-2012 18:54

I have ran into this issue with a huge queue and I can't restart the container:

[12/06/27 19:36:47] ID=Broker02 (info) TCP_ACCEPTOR: accepting connections on tcp://b3isbqperfsonic02:2516

[12/06/27 19:36:47] ID=Broker02 (info) SonicMQ Broker started

com.odi.AbortException: An I/O exception occurred.

Stack trace at point of original failure:

*****************************************

java.io.IOException: No space left on device

        at java.io.RandomAccessFile.writeBytes(Native Method)

        at java.io.RandomAccessFile.write(Unknown Source)

        at com.sonicsw.storage.impl.CachedPagesFile.writeDataPage(CachedPagesFile.java:289)

        at com.sonicsw.storage.impl.CachedPagesFile.writePages(CachedPagesFile.java:282)

        at com.sonicsw.storage.impl.PageManager.forceToDisk(PageManager.java:587)

        at com.sonicsw.storage.impl.Storage.commit(Storage.java:285)

        at com.odi.imp.sonic.StorageObject.commit(StorageObject.java:177)

        at com.odi.imp.sonic.ObjectTable.commit(ObjectTable.java:95)

        at com.odi.imp.sonic.Transaction.serverCommit(Transaction.java:61)

        at com.odi.imp.Transaction.doPhasedCommit(Transaction.java:262)

        at com.odi.imp.Transaction.doCommitOrCheckpoint(Transaction.java:203)

        at com.odi.imp.Transaction.commitInternal(Transaction.java:166)

        at com.odi.imp.Transaction.commit(Transaction.java:159)

        at progress.message.db.pse.PSEDbContext.commit(PSEDbContext.java:295)

        at progress.message.dbq.pse.DBQPSEBase.commit(DBQPSEBase.java:52)

        at progress.message.broker.BrokerDatabase.commitQueueTran(BrokerDatabase.java:1252)

        at progress.message.broker.QueueMsgSaver.threadMain(QueueMsgSaver.java:1198)

        at progress.message.zclient.DebugThread.run(DebugThread.java:226)

*****************************************

        at com.odi.imp.sonic.Server.IOFailure(Server.java:365)

        at com.odi.imp.sonic.Transaction.serverCommit(Transaction.java:63)

        at com.odi.imp.Transaction.doPhasedCommit(Transaction.java:262)

        at com.odi.imp.Transaction.doCommitOrCheckpoint(Transaction.java:203)

        at com.odi.imp.Transaction.commitInternal(Transaction.java:166)

        at com.odi.imp.Transaction.commit(Transaction.java:159)

        at progress.message.db.pse.PSEDbContext.commit(PSEDbContext.java:295)

        at progress.message.dbq.pse.DBQPSEBase.commit(DBQPSEBase.java:52)

        at progress.message.broker.BrokerDatabase.commitQueueTran(BrokerDatabase.java:1252)

        at progress.message.broker.QueueMsgSaver.threadMain(QueueMsgSaver.java:1198)

        at progress.message.zclient.DebugThread.run(DebugThread.java:226)

IO Exception flushing log file

PSEException: com.odi.AbortException: An I/O exception occurred.

Stack trace at point of original failure:

*****************************************

java.io.IOException: No space left on device

        at java.io.RandomAccessFile.writeBytes(Native Method)

        at java.io.RandomAccessFile.write(Unknown Source)

        at com.sonicsw.storage.impl.CachedPagesFile.writeDataPage(CachedPagesFile.java:289)

        at com.sonicsw.storage.impl.CachedPagesFile.writePages(CachedPagesFile.java:282)

        at com.sonicsw.storage.impl.PageManager.forceToDisk(PageManager.java:587)

        at com.sonicsw.storage.impl.Storage.commit(Storage.java:285)

        at com.odi.imp.sonic.StorageObject.commit(StorageObject.java:177)

        at com.odi.imp.sonic.ObjectTable.commit(ObjectTable.java:95)

        at com.odi.imp.sonic.Transaction.serverCommit(Transaction.java:61)

        at com.odi.imp.Transaction.doPhasedCommit(Transaction.java:262)

        at com.odi.imp.Transaction.doCommitOrCheckpoint(Transaction.java:203)

        at com.odi.imp.Transaction.commitInternal(Transaction.java:166)

        at com.odi.imp.Transaction.commit(Transaction.java:159)

        at progress.message.db.pse.PSEDbContext.commit(PSEDbContext.java:295)

        at progress.message.dbq.pse.DBQPSEBase.commit(DBQPSEBase.java:52)

        at progress.message.broker.BrokerDatabase.commitQueueTran(BrokerDatabase.java:1252)

        at progress.message.broker.QueueMsgSaver.threadMain(QueueMsgSaver.java:1198)

        at progress.message.zclient.DebugThread.run(DebugThread.java:226)

All Replies

Posted by davidg on 28-Jun-2012 19:31

So I don't believe your message database or recovery logs are corrupt so I think you should be able to copy the files to a disk location with more space and restart the container/broker there. The recovery should then work.

If you are able to have the same pathing then it makes it easier. However if you change the pathing for the DB and recovery logs you will have to change the broker configuration before restarting.

Posted by appmath on 29-Jun-2012 15:55

We are currently trying to allocate more disk space. The (bogus) messages are just for testing. What files should we delete to have a fresh restart.

Thanks,

Aziz

Posted by davidg on 02-Jul-2012 12:17

You can delete the broker DB (default ./SonicMQStore) and recovery logs (./log) directories if you don't mind loosing any messages that are trapped there .. the broker will recreate the files when it restarts.

This thread is closed