Cache web service data and response

Posted by atuldalvi123 on 20-Sep-2016 09:41

Hi All

Is there a way for caching in memory a web service data and response ?

And if exists a nice way for managing data in memory.

Actually, while creating a policy from the portal, we have 8 to 9 webpages and for each webpage we have different different web service call, which hits DB, creates few DB transactions and returns the response.  But because of this process we are facing some serious performance issues at DB levels.

I am looking something related to the cache memory or memory pointer which will keep all the data in memory and finally in a single shot will update the DB tables.

All Replies

Posted by bronco on 20-Sep-2016 10:33

Well, there could be various ways, one way is looking at the HTTP level (see: developers.google.com/.../http-caching. Another way can be done in OE. Assuming your data is not changing every second (then you *must* hit the db every time), the response should be a function of the request. So: f(request) = response. If you somehow concatenate the request parameters and hash this you have a unique number (the hash) representing the request parameters. Based on the hash you can cache your data either in memory of on disk (the latter makes it possible to share the result among all AppServer agents). hashing can be done with the message-digest function. On the every request you hash the request parameters, check if the hash is in the cache etc...

Posted by bronco on 20-Sep-2016 10:37

and use interfaces and dependency injection so you can easily switch from a in-memory cache to a database cache or whatever you come with. (just an advice)

Posted by marian.edu on 20-Sep-2016 11:00

Caching otherwise dynamic content is most of the time not a good idea but depending on the specific content of one page you might decide to let the browser cache the content for one hour/day or more - there are http headers for that (see cache-control on Bronco’s link).


If the content is actually dynamic but you still decide to cache the response then you might need to consider request path, query string, header information (cookies, accepted language, etc)… making a hash of all that is might take some extra processing and then you need to match that agains a potentially large (or growing) cache hash table. That need to be some extra cache management that handles expiration and clean-up of expired cache entries and if you’re on the appsrv and don’t use a database for that each agent will have it’s own cache hash-table so be prepare to throw in some extra RAM, on the other hand if you do use a database table for cache management you might have that I/O issue that was the reason you started to use cache come back and hunt you.

Better just see why do you need the cache in the first place, reading a few records it should not be too much of a penalty if the database is modelled/set-up correctly (you’ll probably have all those records in memory anyway or you should)… if you need to have transactions then take a closer look and see why do you need to have updates, using a cache will defeat the purpose of those transactions anyway.

  
Marian Edu

Acorn IT 
+40 740 036 212

Posted by marian.edu on 20-Sep-2016 11:02

ah yes, have that modelled as business entities that uses a data access layer and then be sure you group them together in business tasks for a proper OERA sandwich ;)


Marian Edu

Acorn IT 
+40 740 036 212

Posted by bronco on 20-Sep-2016 11:48

[quote user="marian.edu"]each agent will have it’s own cache hash-table so be prepare to throw in some extra RAM

[/quote]

I would say: use PASOE and if Progress is kind enough to implement define universal shared temp-table

then the RAM issue is solved :-)

But you're right, writing a good cache isn't easy (it is fun though).

Posted by atuldalvi123 on 20-Sep-2016 14:02

Actually the old programs are designed in such a way where they are unnecessary hitting DB & creating records in the tables in each web service call even before the actual policy creation just to show some information on the next page. For example, on the 1st page itself when we put customer information and proceed for the next page, data gets created in the tables such as account information and many more just to show information on the next page and just like this for every next page we have web service calls and DB hits till the last page which is causing the issues.

These DB hits are not only for the couple of table data population but to read couple of master tables as well. So we are planning to restructure the progress code but it is not easy for us as of now and so we are looking for the memory related concept in progress programming ( Not HTTP level )  where we can skip these DB hits in between.

Posted by AdrianJones on 21-Sep-2016 10:27

if you are operating over multiple web pages, gathering data, you could maybe look at the web storage api and cache it in the browser (see developer.mozilla.org/.../Using_the_Web_Storage_API )

there are a few javascript wrapper libs out there too providing fall back for older browsers etc.

This thread is closed