The size of library procedures

I have read through a post by someone else in the OpenEdge forum and he seems to be asking the same questions as me.

He, for instance, had a huge file that dealt with everything to do with revenue.  When they needed some revenue information they would run that huge file persistently and then run an IP inside it.

That is exactly what we have at the moment.

The ridiculously huge file that I mentioned in the OP is everything to do with orders.  Creating them, despatching them, returning them, cancelling them, reporting on them etc...

With the prospect of moving to a OERA set up I think we have the opportunity to break down these massive files.

In my head I imagine a collection of small (possibly even single, non persistent) procedure files in the DA layer that provide temp-tables or datasets to the BL layer.  The BL layer would then have larger files (maybe one per front end section i.e. creating an order) that then access the DA layer when needed.

Is my train of thought along the right path?

Thanks!

So, this is a persistent procedure ... not a program library per se?

With a PP, there is a cost to run it initially, but once run there should not be a different cost for running an IP within the PP ... unless, of course, it is so huge that you are paging.  Have you looked at your client parameters to see if there is reasonable working room?

Which said, it does sound as if there is a possible benefit from doing some decomposition.  Again, I would look for inspiration to OOAD principles.  It sounds like what you have now is the kitchen sink approach to object design .... throw everything into one bucket, no matter how loosely connected.  I don't find that a good design approach, but by the same token I wouldn't like to see you go the splitter approach of making everything its own .p .  There is a good role for PPs and SPs or corresponding objects and the difference in instantiation cost between a reasonably rich library and individual routines should be minor and actually in favor of the library in the end since there is only one instantiation cost.

Hi Thomas,

Thanks for the reply.

Yes this is a persistent procedure.  I'm not entirely sure what a program library is.  I call this a library procedure because that is what all the other guys here call it.

Each time the GUI wants something from it it runs the procedure persistently on the AppServer and then runs an IP from inside it.  At the moment this particular procedure has about 300 internal procedures and most of the time when it is run persistently only 1 or 2 of those 300 are run before it gets deleted.

Once it has finished with it it then deletes the persistent procedure from the AppServer.

I definitely agree with the smaller persistent procedures point though.  Besides, having 300 small .p files is probably just as bad as having one huge .p with 300 IPs.

The OOAD direction is definitiely something I would love to do coming from a Java background.  At the moment though we have 6 other developers who have been using Porgress for up to 20 years but have no OO experience so I think some training needs to be attended before we go in that direction.

Thanks for the help!

I forgot to say that this procedure could potentially be called thousands of times a day.

Ideally I would like to set up parts of it as SSPs so that it just runs and can be called without having to load massive amounts of program.  Unfortunately the program is not designed to be run that way and I'm almost certain there are temp-tables and variables that are saved in between procedures.

ojfoggin schrieb:
I'm not entirely sure what a program library is.

A program (or procedure) library has nothing to do with the design of the application / architectur. It's an archive (similar to a Java jar file) containing a number of procedures (ususally in compiled form). Access to R-Code in a .pl file is usually faster than accessing each individual .r file using the file system, especially with .r-code on a network share.

ojfoggin schrieb:
Once it has finished with it it then deletes the persistent procedure from the AppServer.

That's probably the issue. Changing this may or may not be simple. It depends on if and what kind of context information is stored inside that persistent procedure (variable values, temp-table contents, etc.). Whenever the PP is loaded and un-loaded this context get's reset to a know state. When it remains in memory (as a session super procedure) it will be the easiest if there is no context at all kept between two calls into the library procedure. If you need to store context, you may need a strategy for that context.

Ah, an AppServer boundary ... new facts!

You might want to read up on procedure libraries.  They bundle together a bunch of code into a single file which then can be memory mapped and shared.

But, now with a clearer picture, let's return to your problem.  Yes, running a PP of large size, executing one or two IPs, and then deleting the whole thing is not very efficient.  But, fixing it may require some rethinking about how you are using AppServer.  This sounds like a perfect setup for creating a Order _service_, whether on an AppServer or Sonic, where there is one or more PPs or SPs or .cls run at startup and then service requests can be run against this stateless, preexisting routine.  In the AppServer case, you would have small facade procedures that were run by the client and which in turn executed the complex logic in the PP.  Note the need to be stateless.

As for transitioning your team ... just because someone has been writing in ABL for many years doesn't mean they are hopeless.  My recommendation would be a combination of consulting and mentoring ... consulting to help you define your architecture so that you knew what the target was that you were shooting for and then mentoring to help guide along the existing developers so that they can map their existing skills on to producing work consistent with the target architecture.  The definition of the target architecture would include a transition plan for how you were going to move from where you are to where you want to be.  You just need the right guide (hint, hint).

Maintaining state across multiple AppServer calls is a no no.  That itself is a real performance killer.  Step One is to make this stateless.

ojfoggin schrieb:
Each time the GUI wants something from it it runs the procedure persistently on the AppServer and then runs an IP from inside it.  At the moment this particular procedure has about 300 internal procedures and most of the time when it is run persistently only 1 or 2 of those 300 are run before it gets deleted.

I've missed that the first time!

That's really bad... Running a persistent procedure from the client on the appserver costs a minimum of 3 requests! One request for the initialization, another one for calling into the internal procedure that does the work and a third one to close it. With a minimum of 2 network roundtrips per AppServer call (call to the broker and then call to the agent) this has a minimum cost of 6 network messages. A killer for a distributed (WebClient) application.

It's better to create small interface procedures (one for all of your 300 internal procedures) that is run non-persistently from the client and runs the library (persistently) on the appserver and from the appserver and returns the data back to the client.

Sounds like a huge efford, but most of these procedures (.p files) can be generated from the signature information of the internal procedures (read on the GET-SIGNATURE method of the procedure handle).

Of course modifying the calling procedures is a bigger deal.... But it might help to create an persistent procedure that runs on the client containing 300 internal procedures that know how to call their counterpart on the appserver (using the non persistent procedures). Then you just have to replace the RUN .... ON hAppServer PERSISTENT SET h  with RUN .... ON SESSION:HANDLE PERSISTENT SET h . The RUN internalprocedure IN h (param1, param2) does not require any change.

As a reference, take Progress' approaches to increase the performance within Dynamics. At a time they made this shift and it created a huge (positive) performance difference!

Never, really never, call a PP from the client on the AppServer. The other negative side-effect is, that the appserver agent is bound to that client for the life-time of the PP.

Architecture change may be charming, but not when the initial pain is performance. In such a situation I recommed fixing the current issue and then concentrate on the new (better) architecture...

So instead of doing this on the GUI...

Run OrderLibrary.p persistent on AppServer set handle.

Run GetOrder in handle.

Delete Procedure handle.

You would do this...

Run GetOrder.p on AppServer.

Then the AppServer does this...

Run OrderLibrary.p persistent set handle.

Run GetOrder in handle.

Delete Procedure handle.

and then passes the output from GetOrder to the GUI?

I am giong back to basics on this and reading the AppServer manual.  The setup we have now, whilst still not ideal, is a whole lot better than it was about 2 months ago.

Thanks for the help.

Hmm... just reading the AppServer manual and one of the things it suggestst and shows how to code in the sample code is running remote persistent procedures on the AppServer.  I understand the performance issues with this but it's fairly conflicting information.

Infact, the manual only says "do not run remote persistent procedures" when referring to a state-free AppServer.  At the moment we are running a stateless AppServer but apart from the PPs we could switch to state-free.  At the moment I am struggling to see how we can sensibily move towards and state-free AppServer and have no remote persistent procedures.  From reading the AppServer manual I can now see why the connection was set up the way it was before I changed it.  It actually shows sample code where each time something is required from the AppServer a connection is made, then a remote PP run, then an IP of the PP run, then the PP deleted, then the AppServer disconnected.

I have taken that and made it so that the connection is set up and start up and disconnected and shut down of the ABL client.

Looking at the trade-offs table of the Operating Modes it looks likee state-free is the one we would benefit from most.

One more question.

What is a "large amount of context"?

The manual states that "if your application has a large amount of context to manage between each request, then stat-aware or state-reset are appropriate choices".

There is never any statement that quantifies what a large amount of context is.

Thanks again.

ojfoggin schrieb:
So instead of doing this on the GUI...

Run OrderLibrary.p persistent on AppServer set handle.

Run GetOrder in handle.

Delete Procedure handle.
You would do this...
Run GetOrder.p on AppServer.
Then the AppServer does this...
Run OrderLibrary.p persistent set handle.

Run GetOrder in handle.

Delete Procedure handle.

and then passes the output from GetOrder to the GUI?

Basically for the server side, that's it. I'd try to auto-generate most of the .p files. And by the way: Those .p files may be pretty good candidates for inclusion in any kind of proxy (Web Services, .NET, Sonic, Java).

On the UI side, I'd try to create a persistent procedure that has the same internal procedures than the service side one. That will reduce the number of required changes to the GUI to to a minimum and it will give you a central place (that new persistent procedure) where you can change the AppServer connection.. Maybe some day you need to introduce Sonic for the communication, then that would be a central change in the PP. You can also implement some caching in the PP where it makes sense (cache salesrep master data for 1 hour, etc.).

I'd try this change on a few places first, see if it solves the performance issues or part of it. Then you can decide to do it for all. But my feeling is, that is's well spend time, especially if you succeed to generate part of the server side .p and the client side PP.

Hmm... just reading the AppServer manual and one of the things it suggestst and shows how to code in the sample code is running remote persistent procedures on the AppServer.  I understand the performance issues with this but it's fairly conflicting information.
Infact, the manual only says "do not run remote persistent procedures" when referring to a state-free AppServer.  

The purpose of the language reference is to tell you what's possible - and remote persistent procedures are possible (and there may be very few good use cases for them). It's different from a best or even good practices piece of information. (And that's also often the difference between a training and consulting, training is moderated language reference while the consultant should give you tailored best practices - at least a good one ).

The state-free AppServer does not support remote persistent procedures, that's why the language reference has to tell you "not to run them", because I assume it will result in an error.

Thanks again for the help.

The one thing I seem to be hearing consistently across all the forums is "get a consultant/mentor".

I think I know what my next recommendation is to my manager.

TBH I think the changes that can be made to our system in improving performance will probably bring about the same performance increase in putting our current system onto a new fangled server.  And a consultant/mentor is probably less expensive than the server.

So instead of doing this on the GUI...

...

You would do this...

Almost anything else would be an improvement ...

One of the things I would consider here is whether there are some concentrations of services which could be identified.  I.e., is there a collection of Order related services and a collection of Accounts Receivable services and a collection of Inventory services?  if so, you might want to consider using multiple AppServers where each AppServer runs some set of persistent procedures at startup so that they are already instantiated when the call comes from the client.

Second, consider what is actually useful to put into a common procedure.  PPs weren't designed for use in which you run the PP, run one IP in the PP, and then delete the PP.  They were designed to be run once and then to have a whole bunch of calls to the IPs.  The whole point, other than retaining state (which you don't want to be doing in an AppServer), is to eliminate the instantiation overhead of making the RUN IP.  Your current architecture has this backwards since instead of the instantiation penalty of running a small IP as a separate .p, you are paying the instantiation penalty of the huge PP all just to run one little routine.  You would be much better off with 100 separate routines.

Of course, the first this to sort out here is whether you are retaining state and why.  If you are making two IP calls and you need to keep state between them, then that is not a design that is suitable across an AppServer boundary regardless of what approach you use.  It is *possible* to keep state in a database on the server, but that certainly isn't the first thing I would do.

Actually, the first major watershed is *any* context.  This is one of the places where you need to step back from the manuals a bit and listen to the community.  There is a broad consensus that modern design of this type of server is stateless (meaning, has no state, not the specific meaning of the AppServer mode by that name).  So, just forget that part of the selection table and forget any mode which retains state.

The only difference between a little state and a lot of state is that it isn't very expensive with a little state to send the data back and forth.  I.e., if you have two operations which needs to retain some small amount of information from one to the other and it isn't possible to combine them, then the first operation can return the state to the client and the client can send the state back with the second call, thus making the AppServer stateless.  The more context there is, the less desirable this is.  When it becomes a lot, then one starts considering approaches like storing the state in a database on the server so that it can be retrieved by the second call.

With a stateless AppServer, each call potentially goes to a different agent so nothing should depend on hitting the same agent twice in a row.

The one thing I seem to be hearing consistently across all the forums is "get a consultant/mentor".

There are a lot of good reasons to do so ... and not just to save the price of a server.  The really important thing is to figure out where you want to go ... not next week, but where you want to be a couple of years down the line.  The impact of figuring out that target and working out a pathway that will get you there can easily have a business impact which dwarfs the costs of any hardware, possibly even dwarfs the cost of the development staff.

I think I know what my next recommendation is to my manager.

Here's a little something to get you thinking along the right lines.

http://www.cintegrity.com/sites/cintegrity.com/files/Transformation_NeedsAssessment.pdf