Openedge 11.6 JSDO Webspeed Generic Business Entity example

Hi Carl,

Could you check out the following post?

community.progress.com/.../19521

Edsel suggested a code change there, maybe you can check whether it suits your case.

Had a look at this and the suggestion only helps where the first page returns less that the batch size.

What is needed is for the current filtering to be stored client side and only if the filtering changes should the count function be run on the server. If whilst paging we reach a page that returns less than the page size then we no we have reached the end. If on the last page we find there are yet more records to read then we run the count again.

From my tests it is often better to return the entire dataset (30,000 records) and do client side sorting/filtering, etc unless the dataset is very large. If 30,000 server filtering then the count takes 0.5 seconds on each next page. If it was 300,000 records this would make the grid very slow on page changing, approx. 5 seconds per page.

Hello,

I also found that for some dataset sizes, using local paging, filtering, and sorting is fine.

Others would benefit from a server side operations.

Other may require custom implementation of query (READ) and count.

It would be interesting to see where the 0.5 delay comes from. Whether it is the network or the query/count resolution.

Your suggestion of the delay being 5 seconds with a 10x dataset, would imply that it is caused by the query/count resolution.

In some cases, the sample count implementation performs fast enough (with indexes).

Even though it would be possible to save the filter and only call count if the filter has changed, this might not be sufficient if records have been added to the server and the client does not perform the next page request because it is logically already on the last page.

It would all depend on whether you would want count to be able to change based on the records on the server or use count to only mean the number of records found on the first count request.

Also, changes to fields involved in the filter by the client may result in a different count.

Something to consider is that you can change the implementation of count (and read/query as well) in the Business Entity to improve its performance.

If you are working with a large number of records, you could cache in a database table the filter and the count. The next request from any user with the same filter would only need to get the value for the count rather that calculating it.

When records are created/updated/removed, you could update the count on the table, asynchronously and independently of the request from the client.

Another alternative, could be to use the "Session Context" functionality. We added a mechanism where you can send properties from the client to the server and also receive them from the server. (Implemented via HTTP headers which can be used with WebSpeed.)

For example, you could create an ID for the filter, either on the client or the server, pass it using the session context and use the ID as the key to get the count of records.

You could also override the count method on the JSDO and use a value for count return with the READ request. In this way saving a network request.

Please let me know if you would like more detail on this.

I hope this helps.

Thanks for the reply, thanks for the example code and for the workshops at the EMEA PUG last year.

It is the count function taking the time, a simple for each product with 30k products and no filtering that takes 0.5 seconds. I was thinking of the JSDO keeping track of the last filter used (where clause) and only calling count if the filter changes.

If we are on the last page and peeking ahead we see there more records (products added) yet to come then add an entire page worth of records to the count, eg. 3000 becomes 3020 with a page-size of 20. Thus adding another page to the grid.

If during paging we reach a page where it is evidently the last page (less than a page of products returned as products deleted) then we set the count to be number of whole pages + records on this last page.

What do you think?  

Hi Carl,

That's not a bad idea on datasets that are relatively static.  I suppose the count is refreshed each time to match the latest data returned by the filter and to stop paging problems if records are created before moving to the next/prev page.  It might be worth having the option to enable this feature though.

Hello Carl,

You are welcome. I am glad to share the knowledge.

I think that the approach that you mentioned seems possible. My concern is that only refreshing the count at when reaching the last page might not be sufficient. In some cases, you might want to know that the number of records changed on the server.

Please feel free to submit an idea for enhancing the count operation.

Considering that you would like to try this out and see how effective it would be, I would like to share some code that you can try out:

            var totalRecords;
            customerJSDO.myCount = function () {
                var request = {}, promise;

                if (totalRecords) {
                    request.deferred = $.Deferred();
                    request.response = { numRecs: totalRecords };
                    request.deferred.resolve(customerJSDO, true, request);
                    return request;
                }
                else {
                    request = customerJSDO.count.apply(this, arguments);
                    request.deferred.promise().done(function (jsdo, success, request) {
                        // console.log("DEBUG: " + request.response.numRecs);
                        totalRecords = request.response.numRecs;
                    });
                    return request;
                }
            };


You can try see this in the example at oemobiledemo.progress.com/.../example014p.html .

The logic on detecting the number of records is greater than the page size is not hard... you can call getData() on the JSDO instance.

I look forward to see how this works for your scenario.

That certainly works for reducing the count overhead. Many thanks. The other 3 points I mentioned can you point me in the right direction code wise please:

- If the filter changes then refresh the record count

- If we are on the last page and peeking ahead we see there more records (products added) yet to come then add an entire page worth of records to the count, eg. 3000 becomes 3020 with a page-size of 20.

- If during paging we reach a page where it is evidently the last page (less than a page of products returned as products deleted) then we set the count to be number of whole pages + records on this last page.

This seems to handle the filtering so any filter change recalculates the count:

var totalRecords,
    currentFilter,
    newFilter;
    productJSDO.myCount = function () {

    var request = {}, promise;

    newFilter = JSON.stringify(productDataSource.filter());

    if (newFilter != currentFilter) {
        totalRecords = undefined;
    }
    currentFilter = JSON.stringify(productDataSource.filter());

    if (totalRecords) {
        request.deferred = $.Deferred();
        request.response = { numRecs: totalRecords };
        request.deferred.resolve(productJSDO, true, request);
        return request;
    }
    else {
        request = productJSDO.count.apply(this, arguments);
        request.deferred.promise().done(function (jsdo, success, request) {
            totalRecords = request.response.numRecs;
        });

        return request;
    }
};

So just the last page, premature end page could do with some help. Thanks.

Added to the example code a recalculation of count when the filter changes. Anyone help with the last page, premature end page  Thanks