dot.r Stomp announcement

Posted by jmls on 07-Jul-2012 01:15

Following on from presentations in Boston and Finland, dot.r is pleased to announce the open source Stomp project, available immediately.

Download  from either http://www.dotr.com or https://bitbucket.org/jmls/stomp , the dot.r stomp programs allow you to connect your progress session to any other application or service

that is connected to the same message broker.

Open source , free message brokers that support Stomp are :

Fuse (http://fusesource.com/products/fuse-mq-enterprise/) [a Progress company now owned by Red Hat inc]

====

Fuse MQ Enterprise is a standards-based, open source messaging platform that deploys with a very small footprint. The lack of license fees combined with high-performance, reliable messaging that can be

used with any development environment provides a solution that supports integration everywhere

ActiveMQ (http://activemq.apache.org/)

=======

Apache ActiveMQ (tm) is the most popular and powerful open source messaging and Integration Patterns server. Apache ActiveMQ is fast, supports many Cross Language Clients and

Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4.

Apache ActiveMQ is released under the Apache 2.0 License

RabbitMQ

=======

RabbitMQ is a message broker. The principal idea is pretty simple: it accepts and forwards messages. You can think about it as a post office: when you send mail to the post box you're pretty sure that Mr.

Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.

The major difference between RabbitMQ and the post office is the fact that it doesn't deal with paper, instead it accepts, stores and forwards binary blobs of data - messages.

--------------------------------

Please feel free to log any issues on the https://bitbucket.org/jmls/stomp issue system, and fork the project in order to commit back all those new features that you are going to add ...

dot.r Stomp uses the permissive MIT licence (http://en.wikipedia.org/wiki/MIT_License)

Have fun, enjoy !

Julian

All Replies

Posted by dmci2309 on 25-Jul-2012 05:34

Hi Julian,

I had troubles installing ActiveMQ 5.6.0 on Windows 7 (64 bit).

You may want to also include this link "http://stackoverflow.com/questions/4626824/how-do-i-install-activemq-as-a-64-bit-service-on-windows" in your notes, it helped overcome this issue.

Cheers

David

P.S. Thanks for sharing... time to play!

Posted by jmls on 25-Jul-2012 06:26

I will do so, many thanks for the heads up.

Any issues, drop me a line.

Julian

Posted by dmci2309 on 26-Jul-2012 06:09

Hi Julian,

Found a couple of issues and have some suggestions for improvement. Whats the best way to send that info to you?

Thanks

David

Posted by jmls on 26-Jul-2012 06:46

fantastic.

By far, the best way would be to fork, make your changes to your

repository, and then create a pull request for me to merge into the

main repository

have a look at https://confluence.atlassian.com/display/BITBUCKET/Forkingabitbucket+Repository

Thanks very much !

Julian

Posted by dmci2309 on 26-Jul-2012 07:48

Done.

Be gentle, its my first fork! (haha sorry, coldn't help myself! )

David.

Posted by jmls on 26-Jul-2012 07:51

grin

Posted by rbf on 26-Aug-2012 06:05

Hi Julian,

This is interesting stuff indeed!

Some minor suggestions:

- Add a ROUTINE-LEVEL statement to StompClient.cls

- Add a bracket to the following example on the web page: dotr.Stomp.Helper.SendMessage:ToPersistentQueue("MyQueue","a test message".

Thank you for your contribution to the OpenEdge community!

-peter

Posted by rbf on 11-Oct-2012 07:03

jmls wrote:

I will do so, many thanks for the heads up.

Any issues, drop me a line.

Julian

Today I got the 'oooop' message:

IF SocketHandle:BYTES-WRITTEN NE lNumBytes THEN MESSAGE "oooop" VIEW-AS ALERT-BOX.

What does that indicate?

Posted by jmls on 11-Oct-2012 07:13

Heh. That means that for some reason, not all of the data was written

to the server. Very unusual. It can happen if the server crashes or

shuts down with active clients, or if the server can't process all the

data in one hit. What size chunks are you sending ? There are various

options in the ActiveMQ configuration to increase socket data size etc

I should write some error handling code round that to handle this .

I will at some stage

Julian

On 11 October 2012 13:03, Peter van Dam

Posted by rbf on 11-Oct-2012 07:18

jmls wrote:

Heh. That means that for some reason, not all of the data was written

to the server. Very unusual. It can happen if the server crashes or

shuts down with active clients, or if the server can't process all the

data in one hit. What size chunks are you sending ? There are various

options in the ActiveMQ configuration to increase socket data size etc

I should write some error handling code round that to handle this .

I will at some stage

Julian

On 11 October 2012 13:03, Peter van Dam

The chunks are small, and I am only receiving. But there were over 400 queued messages waiting on the server and for some reason they were not being processed by the client.

The server never faltered. When I closed the client because I noticed it was not processing messages I got the ooops message. I rely on GC when closing the client.

When I say closing the client I am not closing the OE session, just the menu program that also invokes the listener.

Posted by jmls on 11-Oct-2012 08:26

Hmm. That error is definitely in the sendframe method of StompConnection.cls

It could it be that the client connected , tried to subscribe to a queue and then lost the connection before it had a chance to send the subscribe frame. Or that it did connect , and you're sending receipt frames , and it got disconnected at some stage.

That would explain the server not sending the messages (the client is no longer connected) and the alert-box doesn't allow the client to be GC'd

try replacing the error message with this block of code and see what happens

if SocketHandle:bytes-written ne lNumBytes then

do:

     case true:

          when not valid-handle(SocketHandle) then assign err = "Socket handle invalid".

          when not SocketHandle:connected() then assign err = "Socket is disconnected".

        

          otherwise assign err = substitute("Unexpected write error on socket Bytes written [&1], expected to write [&2:]",SocketHandle:bytes-written, lNumBytes).

     end case.

         

     message err view-as alert-box warning.

end.

Posted by rbf on 11-Oct-2012 08:53

OK did that.

Now we wait

Posted by Admin on 12-Oct-2012 00:58

this is how the write method looks like in my own socketoutputbuffer... maybe can be easily adapted there

method override public void write( input bytes as memptr, input offset as integer, input len as integer ):
      if bytes eq ? or offset eq ? or len eq ? then
         undo, throw new NullPointerException().
        
      if len eq 0 then
         return.
     
      if offset lt 0 or len lt 0 or offset + len gt get-size(bytes) then
         undo, throw new ArrayIndexOutOfBoundsException().
     
      checkClosed().
      do while len gt 0:
         outSocket:write(bytes, offset + 1, len) no-error.
         if outSocket:bytes-written eq 0 then
            undo, throw new IOException().
         assign
            offset = offset + outSocket:bytes-written
            len    = len    - outSocket:bytes-written.
      end.
end method.

Posted by jmls on 12-Oct-2012 01:18

thanks Marian

I will do that .

Posted by rbf on 12-Oct-2012 10:57

OK received the 'socket is disconnected' message today 3 times in a row after closing the menu.

I suppose socket disconnections can happen when there is a glitch on the network (this is actually running over the internet so no wonder).

However, what is strange is that I am getting these only after closing the menu and several times.

If it happened earlier I could try a (silent) reconnect before displaying an error message.

Any ideas?

Posted by jmls on 12-Oct-2012 11:18

Are you running on EC2 ?

We have had some major issues with the network "glitching" , so

clients like putty get disconnected after a very short time .

Stomp also has the problem, but so does ActiveMQ using jms clients, so

it's not progress

Also, which "menu" are you talking about ? Can you manually close the

client before you close the app ? GC does not always delete objects in

the "right" order, so sometimes we manually delete objects in the

order that we want them destroyed

On 12 October 2012 16:58, Peter van Dam

Posted by jmls on 12-Oct-2012 12:49

marianedu wrote:

this is how the write method looks like in my own socketoutputbuffer... maybe can be easily adapted there

method override public void write( input bytes as memptr, input offset as integer, input len as integer ):
      if bytes eq ? or offset eq ? or len eq ? then
         undo, throw new NullPointerException().
        
      if len eq 0 then
         return.
     
      if offset lt 0 or len lt 0 or offset + len gt get-size(bytes) then
         undo, throw new ArrayIndexOutOfBoundsException().
     
      checkClosed().
      do while len gt 0:
         outSocket:write(bytes, offset + 1, len) no-error.
         if outSocket:bytes-written eq 0 then
            undo, throw new IOException().
         assign
            offset = offset + outSocket:bytes-written
            len    = len    - outSocket:bytes-written.
      end.
end method.



Error handling code similar to the above has been implemented and in the latest push. Thanks Marian.

Posted by jmls on 12-Oct-2012 12:53

thanks very much for the great comments.

Both of your suggestions are implemented in the latest version

Thanks for trying it out !

Posted by Admin on 12-Oct-2012 12:55

You're welcome Sir.

See you have more focus on that one, maybe something you can pick-up from the unfinished attempt I had on the subject http://sourceforge.net/p/progress-jms/code/102/tree/net.sourceforge.progress.jms/

Posted by jmls on 12-Oct-2012 13:03

I'm sure that I will be able to steal lots of ideas from it

I am but a "dwarf standing on the shoulders of giants" http://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants

Posted by rbf on 13-Oct-2012 04:07

Are you running on  EC2  ?

We have had some major issues with the network "glitching" , so

clients like putty get disconnected after a very short time .

Stomp also has the problem, but so does ActiveMQ using jms clients, so

it's not progress

Also, which "menu" are you talking about ? Can you manually close the

client before you close the app ? GC does not always delete objects in

the "right" order, so sometimes we manually delete objects in the

order that we want them destroyed

I am running over a normal internet connection (just for testing purposes).

I have noticed that every morning the socket connection seems to have been lost overnight. This means that I don't know which messages I missed.

The problem is that there is no 'disconnected' event I guess.

So I am thinking of reconnecting the client every 10 minutes or so to avoid this issues. Is that normal practice?

Posted by jmls on 13-Oct-2012 08:09

I am running over a normal internet connection (just for testing purposes).

I have noticed that every morning the socket connection seems to have been lost overnight. This means that I don't know which messages I missed.

The problem is that there is no 'disconnected' event I guess.

So I am thinking of reconnecting the client every 10 minutes or so to avoid this issues. Is that normal practice?

Is the server on the internet, or both client and server ?

If you use the latest version on bitbucket, I've added some error handling for socket disconnection. Stomp will now post an "error" message with the body of the message stating that the socket has been disconnected. You can now choose how to handle this (reconnect etc)

let me know if it helps.

If you are using ActiveMQ 5.7.0 or above, you could try adding the heartbeat option to the server, which "pings" the client every x milliseconds. This may help to maintain the connection as it is active.

If you need to know which messages that you have "missed" you can always use a queue for the one subscriber - the messages will build up until that consumer joins

Also, have a look at the Apache Apollo project (http://activemq.apache.org/apollo/documentation/user-manual.html) , which is going to the base of ActiveMQ 6 - there is an option to "retain" topic messages, which means that the last message sent to the topic is sent to each new subscriber

Posted by jmls on 16-Oct-2012 01:13

Peter, did you get a chance to look at the updates ?

We're running with ApacheMQ 5.8 snapshot, seems very sold pumping hundreds of thousands of messages through per day

Posted by jmls on 22-Oct-2012 07:19

Just thought I'd post some details of stomp in real world use

A snapshot of the ActiveMQ in operation at our call center.

The "queue" in "pbx,queue" is not a stomp queue, it's an inbound call queue

Each topic is automatically deleted after 30s inactivity

NameNumber Of ConsumersMessages EnqueuedMessages Dequeued
ActiveMQ.Advisory.Connection025152320
ActiveMQ.Advisory.Consumer.Queue.ActiveMQ.DLQ0100
ActiveMQ.Advisory.Consumer.Topic.pbx.queue.info...014570
ActiveMQ.Advisory.Consumer.Topic.pbx.queue.warn...014570
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.i...0130
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.i...054430
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.i...04230
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.i...03070
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.i...01140
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.w...0100
ActiveMQ.Advisory.MessageDLQd.Topic.pbx.queue.w...046420
ActiveMQ.Advisory.Queue010
ActiveMQ.Advisory.Topic0150
debtnet.info.db.update.Campaign0520
debtnet.info.db.update.CampaignData06026180
debtnet.info.db.update.CampaignNumber05417430
debtnet.info.db.update.ExtraData06430
debtnet.info.db.update.Plan01470
debtnet.info.db.update.Queue0300
debtnet.info.db.update.SkillMatrix030
pbx.queue.info.agentconnected53335714
pbx.queue.info.hangup534552385
pbx.queue.info.newcall534700309
pbx.queue.info.team.removed53250
pbx.queue.info.tryagent53509577929
pbx.queue.info.voicemail5345867
pbx.queue.warning.noagent5347885247530
pbx.queue.warning.timedout532220

and the broker details

Broker

Namelocalhost
Version5.8-SNAPSHOT
IDID:prometheus-1069-1350291226059-0:1
Uptime7 days 3 hours
Store percent used0
Memory percent used1
Temp percent used0

Posted by rbf on 28-Nov-2012 10:20

Hi Julian,

I would like to use synchronous (request-response) messaging. Therefore support for properties such as 'correlation-id' and 'reply-to' should be added.

Currently this is not quite possible since although there is a StompMessage class, this class is not used for sending messages. The current header is constructed hardcoded in the Send() method in StompClient.cls as a string.

I would rather start using the StompMessage class so I can add those properties and be more flexible in the future.

As I am writing this I realize I suppose I could implement that without breaking the code by adding overloads.

What do you think?

-peter

Posted by jmls on 28-Nov-2012 12:01

Hi Peter - have a look at the WithHeader method on the stompclient. This method allows you to add headers of your own choice

Julian

Posted by rbf on 28-Nov-2012 15:53

jmls wrote:

Hi Peter - have a look at the WithHeader method on the stompclient. This method allows you to add headers of your own choice

Julian

OK I guess that will work.

Posted by rbf on 01-Dec-2012 12:02

rbf wrote:

jmls wrote:

Hi Peter - have a look at the WithHeader method on the stompclient. This method allows you to add headers of your own choice

Julian

OK I guess that will work.

I am trying to do this but it does not seem to work.

I think the headers are OK but there is this thing called creating a Temporary Queue that I don't seem to be able to achieve.

Do you know how to do that?

Posted by jmls on 01-Dec-2012 13:45

right, I see part of the problem

given this scenario

a) producer subscribes to /temp-queue/response
b) then sends a message with reply-to header set to "/temp-queue/response"
c) the consumer gets the message , with reply-to set to "/remote-temp-queue/ID\cjmls-study-49161-1353295140603-2\c463\c1"
d) and responds to that reply-to address
e) sure enough, message appears on the queue list of ActiveMQ , but with 0 consumers, so the message just sits there waiting
f) the producer is also just sitting around, waiting for the message as well, but is obviously not subscribed to the queue it needs to be

I seem to smell a bug somewhere. ActiveMQ is meant to automatically subscribe the producer to the /remote-temp-queue/ID\cjmls-study-49161-1353295140603-2\c463\c1 queue, but either it is not, or the subscription is terminated before the response.

I have asked the ActiveMQ folks for their read on the situation.

In the meantime, we don't use temporary queues

Each client that uses a stomp connection gets it's own guid using guid(generate-uuid) (let's call it "myguid") , and subscribes to that queue (/queue/myguid) . If we want a reqeust-response, set the reply-to to be "/queue/myguid", and manipulate the other headers as required.

The consumer of the message should use the reply-to header to send the information back to the orginal producer (in this case, using the /queue/myguid queue )

HTH

Julian

Posted by rbf on 01-Dec-2012 14:21

jmls wrote:

In the meantime, we don't use temporary queues

That is probably not an option, as the other party is and NMS client that conforms to the rules http://blog.cylewitruk.com/2011/08/synchronous-request-response-messaging-with-nms-and-activemq/

Posted by jmls on 01-Dec-2012 15:12

right, but if the other party simply responds to the queue specified

in the reply-to header, then won't it get back to your abl client ?

ActiveMQ will only change the reply-to queue name if you specify

/temp-queue/yourqueue instead of /yourqueue/. If you specify

"reply-to: /myguid" then the nms client reply should be sent down the

myguid subscription.

There are settings in the ActiveMQ configuration to automatically

delete queues and topics that are no longer in use, something that

also helps provide temp-queue functionality

In the meantime, I'll let you know what I hear back

On 1 December 2012 20:21, Peter van Dam

Posted by rbf on 10-Dec-2012 04:44

Hi Julian,

The only way I could get this working was by officially creating a temporary queue in the broker.

I added the code below in StompClient.cls to achieve this.

I am happy now since this works fine so you might consider adding this to your code base.

  METHOD PUBLIC CHARACTER SubscribeToTempQueue():
    RETURN SubscribeToTemp("queue"). 
  END.
 
  METHOD PUBLIC CHARACTER SubscribeToTempTopic():
    RETURN SubscribeToTemp("topic"). 
  END.  
 
  METHOD PUBLIC CHARACTER SubscribeToTemp (p_type AS CHAR):
   
    DEFINE VARIABLE nfram# AS CHARACTER.
    DEFINE VARIABLE TempQueue AS CHARACTER.
   
    ASSIGN
      TempQueue = GUID(GENERATE-UUID).
     
    CREATE TTDestination.
    ASSIGN
      TTDestination.Destination = SUBSTITUTE("/temp-&1/ID:&2", p_type, TempQueue)
      TTDestination.ID          = GUID(GENERATE-UUID).
            
    ASSIGN nfram# = SUBSTITUTE("SUBSCRIBE~nid:&1~ndestination:&2~nack:auto~n~n", TTDestination.ID,TTDestination.Destination).
     
    StompConnection:sendFrame(nfram#).
   
    RETURN TempQueue.
              
  END METHOD. 

Posted by rbf on 22-Jan-2013 15:12

Hi Julian,

After an extensive investigation EMEA (kudos to Rob Debbage!) figured out that this is an OpenEdge bug that only hits you when you  set -cpinternal UTF-8 -cpstream UTF-8 which I was doing in my test environment but luckily not (yet) in the production environment.

The issue is logged as bug# OE00230773.

Posted by jmls on 22-Jan-2013 15:16

Yeah, I got a message from Psc informing me that someone had logged this . Kudos, again, to Psc for letting me know.

And, I have to say, well done to you for finding this !

 * jmls is just really glad it's not my code

Now I hope that you're finding it useful !

Posted by rbf on 22-Jan-2013 15:20

jmls wrote:

Now I hope that you're finding it useful !

Definitely because now I know I can confidently put this code in production as long as we don't use UTF-8 which luckily is not the case.

I have noticed that the error had not occurred on the two pilot sites but could not explain that. Now I can.

Your code is excellent!

Keep up the good work!

-peter

Posted by jmls on 22-Jan-2013 15:46

your cheque is in the post ..

On 22 January 2013 21:20, Peter van Dam

Posted by Paul Radbone on 17-Dec-2014 08:14

Sorry for digging up an old post, but are there any examples with message selectors?  also can the message selector be changed without starting a new session?

Thanks,

Paul.

Posted by jmls on 18-Dec-2014 03:07

never them myself, but from these docs (activemq.apache.org/.../stomp-manual.html) all I think you need to do is add a header with the appropriate selector before you subscribe

Posted by PhilF on 02-Nov-2017 18:50

> I have noticed that every morning the socket connection seems to have been lost overnight.

I know that this is far too late to be of any use to you -- but I thought I would add a comment for others who might find this on a search.  We ran into a similar problem -- our connection to ActiveMQ would regularly hang for no obvious reason.  What we found was this:

1)  There was a state-aware firewall between the client and the server.  A connection would be opened between the two, but after about 90 minutes the firewall would notice that there had been no traffic, and would drop the connection.  Neither end was notified.  

2)  The ABL STOMP client system is installed on Cent/OS 7 system with the default TCP/IP configuration.  That means that after 2 hours the TCP/IP stack noticed the lack of traffic.  It  would try to contact the remote system 10 times, separated by 75 seconds.  Since the firewall had closed the connection, these attempts failed and the TCP/IP stack closed the local connection.  Our ABL Stomp client was so informed, and it shut down.

This problem was resolved (in a manner of speaking) by adding

   heart-beat:0,60000

to the (hard-coded) connect frame.  This tells the ActiveMQ system to send a heart-beat every minute, keeping the network connection alive.

I don't know what was happening on the ActiveMQ server at the same time -- but it never dropped the stale consumer from the queue.  Heartbeats would probably resolve that issue too -- but that's a separate conversation.

This thread is closed