PASOE REST call returns JSON string with iso8859 encoding

Posted by wvandergraaf on 14-Jun-2018 10:34

REST calls to our PASOE server keep returning JSON strings that are encoded with iso8859-1, but the client treats the string as utf-8.

Example: 

{
"result": {
"ttmerk": [
{
"merk_in_eid": 1,
"merk_in_num": 1335,
"merk_ch_oms": "Moët & Chandon",
"meso_in_eid": 1,
"meso_ch_oms": "A-Merk",
"rRowid": "AAAAAAAXBAE="
}]}}

Write-json from dataset to file is correct, but to longchar which is then returned to the client app will give above result.

Test client is postman.

In catalina.properties I use the following setting:

psc.as.http.uriencoding=UTF-8

Progress version 11.7.2 (build 1497)

IS there a setting I'm missing?

All Replies

Posted by gdb390 on 14-Jun-2018 10:37

before writing the longchar have you set the codepage correct ?

fix-codepage (<variable name longchar>) = "UTF-8".

this can only be done on an empty string

Posted by wvandergraaf on 14-Jun-2018 10:44

I tried that, but since dataset:write-json will always use UTF-8 this has no effect. The encoding IS actually UTF-8, but somehow it is marked as iso8859-1.

Posted by Matt Baker on 14-Jun-2018 10:48

 
What is Content-Type header in the response to the request?
 
Should be something like
 
application/json; charset=UTF-8
 
If it is only
 
application/json
 
the charset is missing, and the client is free to interpret it as per the default which is ISO8859-1.
 

Posted by wvandergraaf on 14-Jun-2018 11:17

It's only application/json, but I can't find a way to set the charset on the reply.

It's a REST service that returns the dataset in a longchar after doing a simple write-json to the longchar.

How can I add the header for charset? I thought it would be enough to have it in the uriencoding setting.

Posted by Matt Baker on 14-Jun-2018 11:28

The "psc.as.http.uriencoding=UTF-8" system property is instructions to the Tomcat connector how to interpret incoming requests.  Outgoing requests are owned by whatever servlet is generating the response.

If not specified, the HTTP spec requires the client to assume ISO8859-1.  But JSON is required to be UNICODE (i.e. UTF-8 or UTF-16), so the response must include it for the client to interpret it properly.

A lot of responses can get away without specifying it because they only contain ascii, but as soon as you wander outside of that charset, bad things happen.

Posted by Matt Baker on 14-Jun-2018 11:29

How is the response being generated.  Is this REST transport or WEB transport?

Posted by wvandergraaf on 14-Jun-2018 11:36

It's REST transport.

This is my call to the REST service which will generate the json answer.

localhost:8120/.../merk

Previously I had the same setup on another server using the "classic" appserver. This one has no problems. The returned json file is identical, without "charset" in the header. Only thing is, Postman is showing it correct.

Posted by wvandergraaf on 18-Jun-2018 08:43

I tried another approach. After dataset:write-json to longchar I performed a codepage-convert to iso8859-1.

It's ridiculous as far as I'm concerned, but it does work!

So first: dataset ds:write-dataset("longchar",lcResponse,true). /* this is always UTF-8 */

then: lcResponse = codepage-convert(lcResponse,"iso8859-1","UTF-8").

Now the SoapUI and Postman testclients view the characters correct.

This thread is closed