Get data from Multipart/related response

Posted by jsandrea on 19-Jun-2018 18:20

Hi everyone, I have a problem with a service consumption, the service response is Multipart/Related. 

This is the full response:

------=_Part_8444890_801700095.1529449048289
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"

<SOAP-ENV:Envelope xmlns:SOAP-ENV="schemas.xmlsoap.org/.../"><SOAP-ENV:Header><wsse:Security xmlns:wsse="docs.oasis-open.org/.../oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="docs.oasis-open.org/.../oasis-200401-wss-wssecurity-utility-1.0.xsd" SOAP-ENV:mustUnderstand="1"><ds:Signature xmlns:ds="www.w3.org/.../xmldsig Id="SIG-6397428"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="www.w3.org/.../xml-exc-c14n xmlns:ec="www.w3.org/.../xml-exc-c14n PrefixList="SOAP-ENV"/></ds:CanonicalizationMethod><ds:SignatureMethod Algorithm="www.w3.org/.../xmldsig URI="#id-6397427"><ds:Transforms><ds:Transform Algorithm="www.w3.org/.../xml-exc-c14n xmlns:ec="www.w3.org/.../xml-exc-c14n PrefixList=""/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="www.w3.org/.../xmldsig Id="KI-37F78DBBC79211384615294490482784798070"><wsse:SecurityTokenReference wsu:Id="STR-37F78DBBC79211384615294490482784798071"><ds:X509Data><ds:X509IssuerSerial><ds:X509IssuerName>CN=AC SUB CERTICAMARA,O=CERTICAMARA S.A,OU=NIT 830084433-7,C=CO,ST=DISTRITO CAPITAL,L=BOGOTA,STREET=www.certicamara.com</ds:X509IssuerName><ds:X509SerialNumber>91436926122049973593519406185072624124</ds:X509SerialNumber></ds:X509IssuerSerial></ds:X509Data></wsse:SecurityTokenReference></ds:KeyInfo></ds:Signature><wsu:Timestamp wsu:Id="TS-6397426"><wsu:Created>2018-06-19T22:57:28.277Z</wsu:Created><wsu:Expires>2018-06-19T23:02:28.277Z</wsu:Expires></wsu:Timestamp><wsse11:SignatureConfirmation xmlns:wsse11="docs.oasis-open.org/.../oasis-wss-wssecurity-secext-1.1.xsd" wsu:Id="SC-6397425"/></wsse:Security></SOAP-ENV:Header><SOAP-ENV:Body xmlns:wsu="docs.oasis-open.org/.../oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="id-6397427"><ns2:EnvioFacturaElectronicaRespuesta xmlns:ns2="www.dian.gov.co/.../ReportarFactura" xmlns:ns3="www.dian.gov.co/.../ConsultaDocumentos" xmlns:ns4="www.dian.gov.co/.../VersionDespliegue"><ns2:Version>Componente DIAN</ns2:Version><ns2:ReceivedDateTime>2018-06-19T17:57:28.154-05:00</ns2:ReceivedDateTime><ns2:ResponseDateTime>2018-06-19T17:57:28.276-05:00</ns2:ResponseDateTime><ns2:Response>200</ns2:Response><ns2:Comments>Ejemplar recibido exitosamente pasará a verificación.</ns2:Comments></ns2:EnvioFacturaElectronicaRespuesta></SOAP-ENV:Body></SOAP-ENV:Envelope>
------=_Part_8444890_801700095.1529449048289--

I already tried with:

oResponseMultipartEntity = CAST(oResponse:Entity,OpenEdge.Net.MultipartEntity).
oMessagePart = oResponseMultipartEntity:GetPart(oResponseMultipartEntity:Size).
oByteBucket = CAST(oMessagePart:Body,OpenEdge.Core.ByteBucket).
oResponseMemptrEntity = oByteBucket:GetBytes().

but the oResponseMultipartEntity:Size return 0. 

Any other idea?

Thank you.

Posted by Peter Judge on 20-Jun-2018 12:26

The challenge is that the conversion from bytes to a MultipartEntity happens before you get control back from the Execute() method.
 
A workaround for this is
  1. Create an extension to the MultipartEntityWriter that does the right thing with the Boundary
  2. Register it for use with multipart messages
 
I’ve attached a FixBoundaryWriter class that un-quotes the Boundary.  It has code like the below in it (for 3 Write() methods).
    method override public int64 Write( input pmData as memptr ):
        // we consume the boundary in an unquoted manner
        if not OpenEdge.Core.String:IsNullOrEmpty(this-object:Boundary)
           and OpenEdge.Core.String:IsQuoted(this-object:Boundary, StringConstant:DOUBLE_QUOTE)
        then
            assign this-object:Boundary = substring(this-object:Boundary, 2, length(this-object:Boundary) - 2).
       
        return super:Write(input pmData).
    end method.
 
To register this writer for multipart messages, add this line somewhere in your session startup (or similar).
OpenEdge.Net.HTTP.Filter.Writer.EntityWriterRegistry:Registry:Put('multipart/*':u, get-class(FixBoundaryWriter)).
 
You only need to do this once per session since this Registry is kept for the life of the session.
Now all HTTP Client (and webhandler) requests that are multipart will use this writer for turning HTTP message bodies into MultipartEntity objects.
 
I would suggest doing this since it means that you are not changing shipped OE code.
 
Hth,
-- peter
 
 
 

All Replies

Posted by Tony on 19-Jun-2018 18:58

Not sure if this has any bearing, but the actual SOAP payload is malformed.

Posted by jsandrea on 19-Jun-2018 19:11

I'm sorry, I can't see where is my mistake, could you be a little bit more explicit please?

Posted by Tony on 19-Jun-2018 19:51

I'm not refering to you code snipet, but to the actual SOAP XML you're trying to extract.

All of <SOAP-ENV:Envelope> .......... </SOAP-ENV:Envelope>

If you copy the XML payload into an online XML validator, it will show you that the XML is structure/syntax is flawed.

Try this one:   https://www.xmlvalidation.com/

Posted by jsandrea on 19-Jun-2018 20:05

I see ... you're right, but this is the response from a government tax office web service, so I'm not able to change it, unfortunately, I have to work with it. I think my main problem is figure out how to read the response from the "multipart/related" to a variable or something like this.

Posted by Tony on 19-Jun-2018 21:14

It sounds like you are reading the raw response from the web service rather than just the service response. This could be because of how you are talking to the service.

Posted by jsandrea on 20-Jun-2018 06:37

Hi Tony, yes, read the raw response could be an option, but I could not read it.

Posted by Peter Judge on 20-Jun-2018 10:42

You’re  running into the same issue as community.progress.com/.../26828 . There is a bug in the HTTP client that has not been fixed yet.
 
The root cause is that the boundary in the response’s Content-Type header has quotes and the HTTP client is not removing them appropriately so the body-parser thinks that the boundary is
“----=_Part_8444890_801700095.1529449048289”
Instead of
----=_Part_8444890_801700095.1529449048289
Unfortunately there are no simple workaround at the moment; code changes are required to the OpenEdge.Net.HTTP.ContentTypeHeader class.
 
 
 
 
 

Posted by Mike Fechner on 20-Jun-2018 10:50

We ran into this issue yesterday (and logged a bug with PSC). While debugging, we found of that we can workaround this issue (or the issue we've had with a multipart response), by adding

ASSIGN THIS-OBJECT:Boundary = TRIM (THIS-OBJECT:Boundary, "~"") .

at the beginning of this method

method override public int64 Write(input pmData as memptr):

in the class MultipartEntityWriter github.com/.../MultipartEntityWriter.cls

Make sure you use the correct version of that file (it was updated in 11.6.1, 11.7.2 and 11.7.3). We've added a modified version of that file in OpenEdge\Net\HTTP\Filter\Payload\MultipartEntityWriter.cls / .r in the PROPATH prior to OpenEdge.Net.pl.

Posted by jsandrea on 20-Jun-2018 11:53

Thank you. I tried to do a cast to others objects type but the response is the same, "Invalid cast from OpenEdge.Net.MultipartEntity to 'some object type'".  Is there some workaround in this case? Doesn't matter if I have to read line by line. Or maybe I'll have to use some third party tool instead?

Posted by Peter Judge on 20-Jun-2018 12:26

The challenge is that the conversion from bytes to a MultipartEntity happens before you get control back from the Execute() method.
 
A workaround for this is
  1. Create an extension to the MultipartEntityWriter that does the right thing with the Boundary
  2. Register it for use with multipart messages
 
I’ve attached a FixBoundaryWriter class that un-quotes the Boundary.  It has code like the below in it (for 3 Write() methods).
    method override public int64 Write( input pmData as memptr ):
        // we consume the boundary in an unquoted manner
        if not OpenEdge.Core.String:IsNullOrEmpty(this-object:Boundary)
           and OpenEdge.Core.String:IsQuoted(this-object:Boundary, StringConstant:DOUBLE_QUOTE)
        then
            assign this-object:Boundary = substring(this-object:Boundary, 2, length(this-object:Boundary) - 2).
       
        return super:Write(input pmData).
    end method.
 
To register this writer for multipart messages, add this line somewhere in your session startup (or similar).
OpenEdge.Net.HTTP.Filter.Writer.EntityWriterRegistry:Registry:Put('multipart/*':u, get-class(FixBoundaryWriter)).
 
You only need to do this once per session since this Registry is kept for the life of the session.
Now all HTTP Client (and webhandler) requests that are multipart will use this writer for turning HTTP message bodies into MultipartEntity objects.
 
I would suggest doing this since it means that you are not changing shipped OE code.
 
Hth,
-- peter
 
 

Posted by jsandrea on 20-Jun-2018 13:39

Hi [mention:9e4ee96fac634b8f91b580e1fb4f7e71:e9ed411860ed4f2ba0265705b8793d05] ,

I am very grateful, your solution works great.  Thank you.

Posted by west on 28-Sep-2018 08:27

Peter,

I'm having the same issues as jsandrea had, size = 0 (Part num must be positive), but adding the FixBoundaryWriter class did not seem to fix it. I'm still learning how to work with web services so maybe I'm doing something wrong on my end. I saved the FixBoundaryWriter.cls to a file and added a propath directory to it so the program can access it. One thing I noticed is that the Write method that is passed in an memptr as a parameter throws an error. I believe this is because the MultipartEntityWriter class does not have a Write method that takes in a memptr parameter so it can't override it. If I take out the override keyword however, it compiles fine.

I am registering the writer at the top of the program I run, is that correct?

DISPLAY "{&LINE-NUMBER} oResponse:ContentLength=" STRING(oResponse:ContentLength).

oResponseMultipartEntity = cast(oResponse:Entity,OpenEdge.Net.MultipartEntity).

DISPLAY "{&LINE-NUMBER} oResponseMultipartEntity:Size=" STRING(oResponseMultipartEntity:Size). <-- Returns error

I'm using version 11.7.3.

Thanks in advance,

West

Posted by Peter Judge on 28-Sep-2018 11:48

What's the error?

And yes, it sounds like you're doing the registration at the right place.

Posted by west on 28-Sep-2018 12:28

The error is "Part num must be positive" It's returning a 0. This occurs after the Execute() method.

oResponse = ClientBuilder:Build()

                       :Client:Execute(oRequest).

DISPLAY "{&LINE-NUMBER} oResponse:ContentLength=" STRING(oResponse:ContentLength).

oResponseMultipartEntity = cast(oResponse:Entity,OpenEdge.Net.MultipartEntity).

DISPLAY "{&LINE-NUMBER} oResponseMultipartEntity:Size=" STRING(oResponseMultipartEntity:Size). /***Error "Part num must be positive" is thrown on this line and the program fails***/

oResponseMultipartEntity:Size returns 0 for some reason and causes the program to fail. Do I have to reference the write method anywhere? I just want to make sure I'm not missing a line that's causing this issue.

Posted by Peter Judge on 05-Oct-2018 10:37

You shouldn't need to call Write() anywhere - the HTTP client will do that.

So you are getting an instance of MultipartEntity returned?

I'd also check the response-data-received.txt file (you can enable that by turning on logging at a level of 5+).

Posted by west on 09-Oct-2018 09:47

Yes I am getting an instance of MultipartEntity returned. I am about to successfully call oResponseMultipartEntity:Boundary and get a value. I set some outputs in the program and it fails when trying to access the size of the oResponseMultipartEntity object.

I turned on logging and set it to 5 via LOG-MANAGER (Thanks for that as I was unaware of it). Below is what is returned for response-data-received.txt when my program runs.  Do you see anything out of the ordinary?

HTTP/1.1 200 OK

Content-Type: multipart/related; type="application/xop+xml";start="<tempuri.org/.../xml"

Server: Microsoft-IIS/8.5

MIME-Version: 1.0

X-Powered-By: ASP.NET

Date: Tue, 09 Oct 2018 14:25:50 GMT

Cteonnt-Length: 1736

Cache-Control: private

Content-Encoding: gzip

Content-Length:        847

‹      •Um›8þ)ÿÁÚ§«*Ö’nÈz‘HÂnr—’(pj÷£&±D²Mwóïo€»Vmöø™™g^<ô{†Q–<™X©ã8‰c˜ƒáÈƶeìÆIbØéîÁÞ™æ°ÑGž<YýÞ,š m,çDZŒ5;¥ä÷¹ÜcÓ½b"I…J™4|ç û ï¸î N› Z©æ¹Àïyññý˜=Æ*ÓO¥Nñ£ÜӝfïÃá]¿×ï5ñÅ7–åC j¢žîÎ|T|`GªîA®rZÔ¼ªfg

|ç‚þ4ON.yazF›3My¦¦§fAµÊãŽÉ-SE.ÔÙÅÅA7`0õ3e¦ÏùÅÄÛÛÛý›][˜¦…¿~^…5mƒ¥©ˆžAöLµÞ/—¬Ó”¯9Õ̵±5 #ÖYƒ‰i‹¼Ïw!d“1ꎰSáÆ9ª$ȇ:H–\ŸOJ¦4vÉœCÜŠW%qÑUmCOhÆ¿ñm˜ ™>¡—’Ç஠¡¶" C¡¦º¬8ZŸî?YÃ?Gneç`fy)´kÜݶŠs¶—Œ¹À·Y!-E*i\yº˜:£Z¥ƒ2»‚›ýÕ¤Š%/jªÑÖ@¡ïEhê¯"¬#ôe½

Ðôm¼0ôƒ‹þØëGÑ¿nÎ덖øe¡ ÝTj~S¸JÝV–s×6!#¶9€ÌŸ%$ Gæ†/ˆü

Ë¿—‘÷e±Œ|Áõ1©ª¹N§\êƒkAU°åŒnëÞ…ê†¹sP¡ ®÷ÄÓ:—‚ @írsÈEU°†ÏøÚÿãy@pýŒžWëírîý€+®=o2Ê…æiúLê/$ú.U¥vMkÓŒ0gÒ”Ç7ÑbÏ<cÉU­º,†iÖCTç

Þ&o5Î%«¼GWµÌ¨Hrh¿hKp{ÞPjîù;Wö£­iÛcghš¦·þÚj.ù«Lö(œ1Z0*a@"¸´<f’àrîÙNͲ\U÷µ‘ƒ`ò_ž,µno å¢ WÓSm‡2¶ì.

ñsîPõ@\;€aÑNrøwüîÉ0ú½Œ-ÓàÈ                  

Posted by Peter Judge on 17-Oct-2018 10:28

it's the fact that it's coming zipped that's breaking things, as indicated by the Content-Encoding: gzip header.

(I also see 2 Content-Length headers and no boundary parameter on the Content-Type)

The way to influence this is via the Accept-Encoding header that the request sends. See tools.ietf.org/.../rfc7231 for some details.

You can try Accept-Encoding with a value of "identity" which should tell the server not to zip the response.

Posted by Matt Baker on 17-Oct-2018 10:36

For anyone wondering why the whacky spelling in some of the headers...

www.nextthing.org/.../fun-with-http-headers

This thread is closed