Incorrect computation when using decimal fields with 9 decim

Posted by Rollbase User on 07-Mar-2013 22:02

Our client has a requirement to compute decimal fields and round off the results to the nearest 9th place. However, they've noticed a slight error in the computation of Rollbase. Please see screenshot below:  Here's the the result when we debug the formula: 001 function wrapper() { 002 sum = 18.0 + 285321.18 + 23180.47 + 73615.7 + 1.0593191E7 + 24411.39 + 10165.98 + 60000.0; 004 } 005 wrapper(); We believe that the error is caused by the decimal field with a lengthy value of 10,593,191.230000000. Changing the value to a shorter length will give the correct results. In addition, we noticed an unusual behavior when updating the value to 10,593,191.230000001 - the value becomes 10,593,191.230000004 instead. We have also tested this in version 4.0.

All Replies

Posted by Admin on 10-Mar-2013 13:17

Hi Godfrey,



This looks like pure JavaScript behavior, Rollbase does not manipulate numeric values directly, they are sent as is to the JavaScript engine we use on the server-side (Rhino). Can you reproduce this using client-side JavaScript?



Thanks,

Matt

Posted by Admin on 10-Mar-2013 22:03

I replicated this on client-side JavaScript and got the same precision issue. This is not an issue with Rollbase. Thanks!

Posted by Admin on 11-Mar-2013 00:51

Great,

Thanks for following up here. Let us know what your solution is when you find it.



Regards,

Matt

This thread is closed