Compression for large (memptr) messages sent from an ABL cli

Posted by slacroixak on 11-Oct-2019 15:48

Dear forum,        doing with 11.7.5

In a recent thread about sending a large binary file to an AppServer, I was advised to adopt the Web transport with WebHandlers.  At a point I thought I might first compress my file before sending it.  Now that I am about doing it, I recall http has native compression capabilities...

This KBase says how to enable compression for the REST Transport :

https://knowledgebase.progress.com/articles/Article/How-to-enable-HTTP-Compression-for-an-OpenEdge-REST-web-application

Q0 : Having played with different levels of compression with 7zip, I found out the default compression rate was a good compromise between the time it takes to make it and the obtained compression.  Based on that, would you rather compress your file locally before sending a zipped version of it, or would just rely on the native compression capabilities of http to do it on the fly, hoping it would negotiate the rate adaptively?

Q1 : I suppose the KBase works for the WEB transport as well as ofr REST, correct?

Q2 : How about the APSV transport?  Does Tomcat enable compression for it as well ?

Q3 : Does the -mc startup parameter work for the PASOE, or does it apply only for the old classical AppServer?

Q4: Aside, with the APSV transport and the Classic AppServer, I do not recall if we can send a MEMPTR to an AppServer session (it may make no sense...) and cannot find it in the Doc.   If it is possible, would it better to send a large binary message with APSV transport and a MEMPTR param with -mc, or with the WEB transport with the enable compression mechanism described in the KBase?

Posted by gus bjorklund on 15-Oct-2019 14:14

> On Oct 14, 2019, at 4:44 AM, Jean-Christophe Cardot wrote:

>

> I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

>

>

the compression code used by the app server is zlib. the zlib shared library is included with openedge.

in my own test harness, i found that the zlib compression makes little difference on fast networks but was useful in slow networks.

Posted by Peter Judge on 15-Oct-2019 13:56

Q6.  -mc should  work with PASOE. Please log a bug if it doesn't.

Q7.  The HTTP client doesn't do anything special for Content-Encoding (either for requests or responses).

Untested approach below but ...

The OpenEdge.Net.HTTP.Filter.Payload.DefaultRequestFilter class is where you would add support for Content-Encoding for requests and for responses, in the OpenEdge.Net.HTTP.Lib.ABLSockets.ABLSocketLibrary class. In both cases it'd be in/after the ExtractEntity method. You could write a class that extends the above, and register them as the appropriate handlers.

For requests

OpenEdge.Net.HTTP.Filter.Writer.RequestWriterBuilder:Registry:Put('HTTP/1.1':u,         get-class(Your.Class.That.Extends.DefaultRequestFilter)).

For responses

OpenEdge.Net.HTTP.Lib.ClientLibraryBuilder:Registry:Put(get-class(IHttpClientLibrary):TypeName, get-class(Your.Class.That.Extends.ABLSocketLibrary)).

If you'd like to log a bug for this too, please do. It might end up being an enhancement but I'd think that it's worth logging. As I mentioned earlier, the PASOE server does the Content-Encoding in the Tomcat layer.

Posted by Jean-Christophe Cardot on 14-Oct-2019 08:40

Hi Sébastien

I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

The dll exists in both 32 and 64 bits, Windows and Linux (and other unices too) and can be easily used from ABL.

Regards

JC

All Replies

Posted by David Cleary on 11-Oct-2019 16:36
Four properties control compression on PASOE, and they are transport independent. Below are the defaults.
 
-Dpsc.as.http.compress=on
-Dpsc.as.https.compress=on
-Dpsc.as.compress.min=2048
-Dpsc.as.compress.types=text/html,text/xml,text/javascript,text/css,application/json
 
So, compression is enabled for payloads > 2048 and for the mime types specified. To enable compression in APSV, you need to add application/octet-stream (please double check this first) to the compress types. There is an addition setting that isn’t exposed as a property, but is what Tomcat provides as a default.
 
noCompressionUserAgents="gozilla,traviata"
 
Dave
 
Posted by Peter Judge on 11-Oct-2019 16:42
In PASOE , there are properties in conf/catalina.properties to control this. The values below are defaults in 11.7.something (but don't ask which version I used to create the instance).
# HTTP & HTTPS  Compression
psc.as.http.compress=on
psc.as.https.compress=on
psc.as.compress.min=2048
psc.as.compress.types=text/html,text/xml,text/javascript,text/css,application/json
 
Since the config is for the HTTP and HTTPS connectors, and since all client traffic flows through one of these schemes/connectors, I'd expect that all the transports were covered.
 
That said, I don't know how you'd get an ABL client (ie APSV transport) to compress the HTTP request.  The answer is probably your musings around Q4 . I don't know where the "-mc" compression is de-compressed, but if it's done in the MS-Agent then you may get a benefit from it since the data would be passed from the Tomcat process to the MS-Agent.
 
 
Posted by David Cleary on 11-Oct-2019 16:44
Just to clarify, the KBase you reference is for the Classic REST Adapter on Tomcat 5.5.
 
Posted by David Cleary on 11-Oct-2019 16:47
Forgot about the client. Yes, for APSV and an ABL Client, you should use -mc. The OpenClients can probably support HTTP Compression but it isn’t anything we investigated.
 
Dave
 
Posted by slacroixak on 11-Oct-2019 17:21

Thank you David and Peter.  I was using the httpClient (OpenEdge.Net.*) to use the WEB transport from an ABL Client.  

Q5 Can it support compression enablement?

To what I understand, I'd better simply use the APSV transport with -mc and you seem to answer Q4 saying we can use a MEMPTR input param to send something.  Correct?

Posted by Peter Judge on 11-Oct-2019 17:32
No. Not natively - you have to compress (and de-compress) the body yourself with the HTTP client. I thought there was a bug logged for this but apparently not. So feel free to log one.
 
 
 
Posted by slacroixak on 11-Oct-2019 21:12

Thank you Peter, I will log one but could you please tell me a out Q4?  Can a memptr be sent via APSV transport, and can compression. Occur?

Posted by slacroixak on 11-Oct-2019 21:19

to remove any confusion : I am interested in the new PASOE appserver, not the classic one to be deprecated. I'm interested in sending large stuff from an ABL client, with either WEB or APSV transport.

Posted by slacroixak on 11-Oct-2019 21:19

(post removed because duplicated for obscur reasons)

Posted by slacroixak on 14-Oct-2019 07:46

Just did a few tests : Yes we can simply pass a MEMPTR to an AppServer, with the APSV Transport.  But -mc seems to have no effect.  

I can test it by sending a 10MBytes file filled with the character 'A'  (so 10 millions 'A' s) that can be highly compressed (down to 12kBytes with 7zip)

With Wireshark, I can see a the long message sent to AppServer is cut in slices of 8402 bytes, with a content full of 'A', the same way with or without -mc.

I've tried that with both MEMPTR and LONGCHAR parameter types with same result.

I am surprised -mc has no effect.  According to this KBase, I have enabled it the right way on the client side (I even message SESSION:STARTUP-PARAMETERS to make sure the launcher inserts -mc) :

knowledgebase.progress.com/.../P69169.  

Q6 : Did I miss something?  Would -mc work only for temp-table parameters?

Q7 : Perhaps a question for Peter, so I could keep using the WEB Transport and the native decompression capabilities mentioned above : could you give me a hint to compress my http bodies myself on the ABL side as suggested ?  (Any native .Net System method?).

Posted by Jean-Christophe Cardot on 14-Oct-2019 08:40

Hi Sébastien

I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

The dll exists in both 32 and 64 bits, Windows and Linux (and other unices too) and can be easily used from ABL.

Regards

JC

Posted by Peter Judge on 15-Oct-2019 13:56

Q6.  -mc should  work with PASOE. Please log a bug if it doesn't.

Q7.  The HTTP client doesn't do anything special for Content-Encoding (either for requests or responses).

Untested approach below but ...

The OpenEdge.Net.HTTP.Filter.Payload.DefaultRequestFilter class is where you would add support for Content-Encoding for requests and for responses, in the OpenEdge.Net.HTTP.Lib.ABLSockets.ABLSocketLibrary class. In both cases it'd be in/after the ExtractEntity method. You could write a class that extends the above, and register them as the appropriate handlers.

For requests

OpenEdge.Net.HTTP.Filter.Writer.RequestWriterBuilder:Registry:Put('HTTP/1.1':u,         get-class(Your.Class.That.Extends.DefaultRequestFilter)).

For responses

OpenEdge.Net.HTTP.Lib.ClientLibraryBuilder:Registry:Put(get-class(IHttpClientLibrary):TypeName, get-class(Your.Class.That.Extends.ABLSocketLibrary)).

If you'd like to log a bug for this too, please do. It might end up being an enhancement but I'd think that it's worth logging. As I mentioned earlier, the PASOE server does the Content-Encoding in the Tomcat layer.

Posted by gus bjorklund on 15-Oct-2019 14:14

> On Oct 14, 2019, at 4:44 AM, Jean-Christophe Cardot wrote:

>

> I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

>

>

the compression code used by the app server is zlib. the zlib shared library is included with openedge.

in my own test harness, i found that the zlib compression makes little difference on fast networks but was useful in slow networks.

Posted by slacroixak on 15-Oct-2019 16:32

Thank you Jean-Christophe, Peter and Gus.

Peter, I'll try to soon report a bug for -mc.  Regarding Q7, I will have to do it quick.  Compression is important for my case because my calls go over WAN.  The messages are so large that I decided to send them by Chunks so it offers monitoring and stop + continue possiblities.  I was interested in native compression of my chunks at the transfer level if it was available, but since it is not and I am getting short in time, I am wondering if I'd better produce a zip file of what I want to send, then send that zip by chunks rather than taking a risk at implementing the compression of payload on the fly for now.

Now, I agree this is an important feature.

Posted by Jeff Ledbetter on 15-Oct-2019 17:13

We send/receive large files in chunks from/to our Java (passed to server via MEMPTR) and have been happy with the results. It's been a while, but we tried various sizes and landed on 10MB chunks being the optimal size.

This thread is closed