Balancing objects, object-orientation, and relational data i

Well, here's my contribution http://www.oehive.org/OERAStrategies

Thomas, your paper raises a number of interesting issues. To start to dig into some of the specifics, you mention in your contribution to this thread...

http://www.psdn.com/library/thread.jspa?threadID=7645&tstart=0

...that you are thinking / working in terms of treating a data entity such as a customer as strictly a set of properties in a true object, that is, an instance of a class that doesn't depend on manipulation of temp-tables and the like.

At the same time (as I understand the post), a collection of multiple data entities such as addresses for a customer, or orderlines for an order, could / should be represented as a temp-table (or even conceivably a ProDataSet?). And a single entity like an orderline could be represented as an object, while a collection of them could be represented as a table (or PDS?). Am I at all understanding this correctly? And is there a mapping between the two representations that allows the distinction to be completely masked?

And -- not to put too many questions into a single post -- I'm all in favor of encapsulating the data definitions into a single Business Entity (or whatever), understanding that this could in fact be a hierarchy of classes that define and refine subclasses of a general class and inherit its data definitions. So this ought to remove most of the problems of sharing tt and PDS definitions across unrelated objects. But the standard PDS / tt representation still allows the data to be passed in a very OE-specific way to another OE session (server to client) or to non-OE sessions and services. What's the approach to making a customer object available to the UI if there's no tt buffer?

What's

the approach to making a customer object available to

the UI if there's no tt buffer?

.NET for example enables data-binding to (almost) any class. Not just to dataset and datatable rows. Customer objects that do not inherit form any ADO.NET class may as well be browsed and updated in a Grid / Browser or a collection of fields.

The fact that this and serialization of objects accross the appserver boundary is not (yet ?) supported by the ABL makes such an implementation pretty hard - and I doubt if it would be maintainable.

But never the less from an architectural point of view such an aproach would be very desirable.

Something else I'll toss into the stew - what about load performance? My "managed SP" system - which is a kind of "procedural" twist on OO - is working really well for the customer I wrote it for, except for the r-code load times. Checking log files and such, and just doing a "RUN x.p" is taking an extraordinarily long amount of time before the program even does anything.

Will PSC classes & objects suffer from the same problem?

Thomas, your paper raises a number of interesting issues.

That was the general idea!

My current thinking is that one of the "twists" over standard OO practice in 3GLs that we should expect with a 4GL is that instead of having just an entity object which encapsulates data and behavior for a single instance and general purpose collection classes with which to handle groups of these entity objects, in OOABL we should have an entity object and one or more "set objects" which are specific to the entity object type. The set object uses a temp-table or PDS internally, but encapsulates it completely so its definition does not need to be shared. This set object would have methods to return and/or accept individual entity objects when some other object needed to deal with or produce one of the set. It would also have various business processes as appropriate to the set, e.g., total value or whatever.

Note that the key distinction here between the proposals offered thus far by PSC is that the TT or PDS is completely encapsulated and is accessed only by its methods.

I'm all in favor of encapsulating the data definitions into a single Business Entity (or whatever), understanding that this could in fact be a hierarchy of classes that define and refine subclasses of a general class and inherit its data definitions. So this ought to remove most of the problems of sharing tt and PDS definitions across unrelated objects.

The only way that I see that you avoid sharing TT definitions among multiple objects is if you make the DAO, BL, and UI classes all subclass from some common parent, which sounds very ugly to me. Otherwise, you will have to clarify what you mean.

As for the UI, I will confess to having a strong bias toward flyweight clients and so the issue of passing an ABL TT, PDS, or object doesn't really arise. Moreover, even if the client is a heavy client and non-ABL, one is probably going to be passing it XML so the issue of passing a ABL construct really doesn't arise. So, the only case we are left with is the ABL client ... not a context where I would expect to be seeing advanced OO OERA code, but even still I have a certain inclination to think in terms of still sending the data as XML. A couple of years ago, Greg Higgins reported on some experiments which suggested that he could serialize a TT to XML, transmit it, and deserialize it into another TT as fast or faster than he could send the TT itself across the wire.

Now, assuming that Greg's work still holds in the present day or at least close enough that it is not a major performance concern, using this approach has a number of pluses, not the least of which is that the server-side set object is agnostic about the UI which is consuming or supplying it. Today that UI could be another ABL program and tomorrow it might be .NET, but the server-side set object doesn't have to change. Sounds like a very big plus to me.

Moreover, regardless of the technology used at the client, there is no need for the receiving end to be of the same type as the source ... it merely needs to agree to a vague common signature about an XML parameter with certain characteristics. The XML file can acquire new elements which are not needed by the consumer and it simply ignores them. Sounds a lot like the arguments for XML messages in Sonic, doesn't it? And, its behavior can be whatever one wants. It can be read-only, for example, or only allow update of some limited information, e.g., a tracking number in a shipment, for example. In that context, if you want to have an ABL UI client, I see no reason why it can't consume and produce XML as well and then one is freed of this need to share TT definitions all over the place.

Now, I can't say what that ABL UI should look like since, for reasons stated above, it isn't an area I have been looking at, but I'm pretty sure that if you can do it passing in a TT, you can do it passing in XML.

You know John, if we had been thinking right a while back, this topic would have made an interesting joint panel discussion at Exchange, don't you think?

If the product transparently supported defining a complex data parameter on one side of a call (say, an AppServer-based business object) as a DataSet or as XML, and as either a DataSet or XML on the calling side (say, an OE client) in a mix-and-match form, it sounds as though that would help provide a step in the direction of eliminating the number of hard-wired dependencies between objects that share data definitions. So for example, a server-side business object could hold its data in a proDataSet instance and define various access points using that as an output parameter. Or it could hold its data as a set of object properties and be prepared to assemble it into an XML document and expose that as a parameter. An OpenEdge client -- or any other procedure or class running that entry point -- could receive the data as a PDS or as XML with any necessary conversions being done under the covers, much as we do already for Web service calls and calls from SOnic processes.

No promises, but does this sound like a step in the right direction?

As always, your mileage may vary depending on circumstances, but overall, an ABL class definition creates another form of .r file, and will not automatically be faster than executing a .r file compiled from a procedure. This is an area where the product group continues to work on optimizations, but where our high-level language does not (yet) compete with 3GLs in raw load or execution speed. That said, the ABL RUN statement is not that slow, and it certainly should not take "an extraordinary amount of time before the program does anything". So if that observation is not exaggerated, there must be something else going on. (When in doubt, blame the ProPath... ). ProPath or other configuration issues aside, is this for instance a persistent procedure that has a substantial main block that does a lot of setup before you can run its entry points? One general way to attack a load-and-initialize problem is to have some sort of simple service manager that keeps track of running procedure instances and allows them to be reused. (But maybe that is part of your "managed SP" system?)

Another couple of possible parts to the solution to shared hard-wired data definitions:

-- In many circumstances, even if the client (or other) requester wants to use the "heavyweight" DataSet, it may be quite reasonable to define these as dynamic objects (i.e handles) on the calling side, so that there is flexibility in what data and what data format is expected to come back. The request itself can include a description of what data is wanted and what is not needed, which the service provider can be responsive to.

-- I entirely agree that requesters should not generally have access to -- or need access to -- the actual data definition used by the object that actually manages the data. So outside whatever business object hierarchy can inherit a data definition, access should be provided through an API. If another object that is part of the larger business logic needs to know the inventory total for a customer, the Business Entity or whatever has responsibility for that data should provide an entry point to satisfy that request. This is not contrary to OERI examples, but perhaps outside the scope of what we have properly illustrated thus far.

-- The BE to DAO connection that shows how the BY-REFERENCE stuff works, and thereby requires a shared PDS definition, needs re-examination, I agree.

No promises, but does this sound like a step in the right direction?

I.e., automatic "type" conversion between PDS and XML?

No, actually that doesn't sound like a step in the right direction.

If one wants to write server-side code which is independent of the client, then one should not be sending out a PDS in any case. The medium of exchange should probably be XML. Arguably, if it were a .NET client one would want a dataset instead, but if one goes that route then one is not going to have client independence, is one?

More control over WRITE-XML and READ-XML would certainly be welcome. From what I see on the forums, people who try to use these for anything other than out and in to ABL run into a lot of problems.

Now, a true XML datatype would be a nice idea! And if you wanted to support automatic conversion from PDS to that datatype, that might be cool, but also somewhat redundant wrt a flexible WRITE-XML method.

A couple of observations...

First, coming from the context of the kinds of things that people have done with TT and to a lesser extent PDS in the .p world, then adding capabilities like BY-REFERENCE, passing handles, etc. seems like a clear plus compared to actual copies of the whole thing ... assuming, of course, that one is in the same session so such things are even possible.

But, let's shift mentality to an OO context where we expect data to be encapsulated, not being passed around from routine to routine. If something needs to happen to the data, it should be a method on the object containing the data, not some procedure that gets passes a copy of the data, whether it is the original instance or not. If one is in the same session, it is the object itself which is "passed" and that is inherently a handle, not a deep copy. So, really, it seems to me that these new language features might be irrelevant if one was using real OO thinking.

Also, one needs to keep clear what is and is not in the same session. If one is in the same session, then there is no need to pass the data anywhere, one just needs to give consumers a way to reach the data. If the data is encapsulated in an object, then there is no need to pass it anywhere ... just provide a way for any potential consumer to find the object.

If, however, one has multiple sessions, then the fancy ways of avoiding a copy of the TT or PDS are irrelevant anyway. One has to serialize in some fashion. If one serializes to XML, then one opens up the potential for either end being non-ABL, including Sonic sources and sinks. If one makes it some special proprietary serialization, then the consumer can only be ABL.

So, one interesting evaluation which I would think it would be fairly easy for someone at PSC to make is to take a sample of TTs and PDSs, preferably borrowed from some real world applications and test both the relative size of the proprietary serialized form compared to the XML form and the end to end speed of transmitting the proprietary form versus the XML form. Either there is a big difference or there isn't and either that difference is in favor of the proprietary form or it isn't. Measure time roundtrip for simplicity. If the difference is negligible or in favor of XML, then I have a hard time finding an argument for the proprietary form in any circumstance other than some nominal ease of coding.

In addition, it should be noted that if one is going to send a PDS over the wire, one has to send the logic in addition to the data. This appears to have some advantages because it means that one can get some or the semi-automated before image handling and the like, but it is also constricting in a way. Not only does it require an ABL client, but it also means that the consumer is tied to the concept of this object in its server-side context. What, for example, does an ABL client which is procuring a PDS from the server need with fill method code? Its already filled.

You know, while it might be too late for Exchange, we could always do a roundtable at PSDN Live 08.

As always, your mileage may vary depending on

circumstances, but overall, an ABL class definition

creates another form of .r file, and will not

automatically be faster than executing a .r file

compiled from a procedure. This is an area where the

product group continues to work on optimizations, but

where our high-level language does not (yet) compete

with 3GLs in raw load or execution speed.

Then you have a looming issue coming down the pipe as developers build-out larger OO based systems.

That said,

the ABL RUN statement is not that slow, and it

certainly should not take "an extraordinary amount of

time before the program does anything". So if that

observation is not exaggerated, there must be

something else going on.

I'd be open to suggestions - in my "managed SP" system, which I would say works in a similar fashion to how OO code'll load, in some of the more involved screens, I've got users looking at tens of seconds to go from an opening a menu to actually seeing something. During this time, all that was going on in the background was loading a bunch of SP's. I haven't used a profiler, but I have done logging on the loads, and in some cases I'm seeing hundreds of ms pass between "run x.sp persistent" and the first statement in "x.sp" being executed.

(When in doubt, blame the

ProPath... ). ProPath or other configuration

issues aside, is this for instance a persistent

procedure that has a substantial main block that does

a lot of setup before you can run its entry points?

I'm running 64bit HP-UX, 64bit client, and for the most part the SP's don't do much during the setup process except make sure all the SP's they need are loaded. The procedure manager loads the required SP's (and their dependencies), and that's about it.

One general way to attack a load-and-initialize

problem is to have some sort of simple service

manager that keeps track of running procedure

instances and allows them to be reused. (But maybe

that is part of your "managed SP" system?)

That's exactly what my procedure manager does.

The issue isn't managing the procedures themselves, it's the amount of time it's taking the AVM to load the r-code. Even with -q and -mmax set so no swapping is going on, the delays are rather painful, even on a decent sized system w/a good SAN behind it. I've tried memory-mapped procedure libraries as a for-instances, and while it helps, it's not enough to make a big difference.

I've reported this to PSC TS, they couldn't help me as improving RUN performance is on the schedule, but not for a while now, so I sent Salvador all the info I had, and the customer has to live with the issue.

Now, I expect whatever handles OO class r-code management w/in the AVM will be a lot faster than my ABL procedure manager implementation, but as these OO deployments get bigger, I expect r-code load performance to become a serious concern unless something is done to improve things.

As a data point for consideration - the system I'm working with here has some 1200 SP programs, and upwards of ~150 SP instances loaded at one time. I did a poll on the PEG for size of OO implementations, and I don't think I found anyone over 100 classes in total, much less loaded at once.

I should note that I sent in an example to PSC TS which showed a dramatic difference in load times between different procedure instances - one was a few milliseconds, while a following program - which was actually smaller - was around 75ms.

I'm not sure where the delay was coming from, but it needs to be looked into.

> You know, while it might be too late for Exchange, we could always do a roundtable at PSDN Live 08.

This is a very good suggestion and I'll pas it on.

> I.e., automatic "type" conversion between PDS and XML?

> No, actually that doesn't sound like a step in the right direction.

Well, if your application does not use ProDataSets at all or at least never on an OE client, then sending all data to service requesters as XML could make perfect sense. But many developers do and will want to take advantage of ProDataSets and temp-tables on the client. Many developers will be very interested in the new Advanced UI capabilities in 10.2. If your client requirements don't go in this direction, that's perfectly OK, of course, and one important type of use case.

I'm suggesting that a hypothetical product enhancement to allow a PDS or tt to be passed as output and received flexibly as either a PDS / tt or XML would allow people the flexibility to define the data whichever way they wish, and to receive it in any type of client that was happy with either type of format.

I do think it would be fun and educational at the same time.

I'm never going to oppose adding functionality to ABL, but recognizing that every addition is competing for limited resources, let me review the business case.

There are two contexts in which one needs to evaluate new ABL functionality -- improvements for the installed base and improvements for those developing new applications. I suppose one might consider those with existing applications who are going to engage in application transformation to separate UI from BL and add a new client as a possible third class.

With respect to the installed base, there is a mixture of character clients, ABL GUI clients, non-ABL clients, and browser clients. Those that are still ChUI clients by and large are probably not going to use this capability in any way. Those that are non-ABL or browser clients can only use the XML version except for those which are .NET and can use the more limited dataset version. So, the real candidate for this proposal is those who have ABL GUI clients.

That is probably a fairly large group. I hope so, because that group is really the only one for which the .NET enhancements in 10.2 are useful. While the Advanced GUI work was obviously quite resource consuming and very nicely done as far as I can see, its real brilliance is that it is possible to sizzle up a limited number of windows without having to redevelop the entire client side of the application. I.e., one can substantially improve the application without having to completely transform the client. If one was going to transform the client, I think a minority would choose to stick with ABL on the client and thus the 10.2 improvements would be irrelevant for them.

So, consider this in relation to your proposal. If there is an ABL client already, whether or not it is using PDS, what obstacle are you removing about making the existing support for PDS over the wire be either/or PDS/XML? If people can already send a PDS over the wire to an ABL client, the only thing they gain by the XML support is the ability to support a different kind of client, maybe. But, if they are interested in a different kind of client, then they are transforming and the PDS support becomes meaningless.

I.e., for those with existing ABL clients, it seems to me that PDS support is already there and an XML adds nothing, unless it turns out to offer higher performance. If the current client is not ABL or one is transforming to non-ABL, then XML is the obvious choice and the option of PDS support is irrelevant. No?

And, for those developing new applications or putting a new client on an old application, I suggest that a minority of those are likely to use an ABL client in this era. Some might, like that car rental software AP, but they will be the exception. Even if they do decide to use an ABL client with PDS, why would they not just send the PDS over the wire? And, if they were not using an ABL client and especially if they wanted to have the flexibility to change clients later without touching the server code, why wouldn't they use XML?

I.e., I don't see the use case where switching back and forth is meaningful.

But, let's shift mentality to an OO context where we expect data to be encapsulated, not being passed around from routine to routine. If something needs to happen to the data, it should be a method on the object containing the data, not some procedure that gets passes a copy of the data, whether it is the original instance or not. If one is in the same session, it is the object itself which is "passed" and that is inherently a handle, not a deep copy. So, really, it seems to me that these new language features might be irrelevant if one was using real OO thinking.

I agree, and that's what I was trying to suggest in one of my responses. If a data instance is encapsulated as an object, and has the right set of methods to provide controlled and indirect access to the data it holds, then other objects in that session need only the object reference to be able to access. it. Only the class that holds the data itself (and possibly subclasses that specialize its behavior and can inherit the data definitions) should be aware of whatever the data definition is.

In addition, it should be noted that if one is going to send a PDS over the wire, one has to send the logic in addition to the data. This appears to have some advantages because it means that one can get some or the semi-automated before image handling and the like, but it is also constricting in a way. Not only does it require an ABL client, but it also means that the consumer is tied to the concept of this object in its server-side context. What, for example, does an ABL client which is procuring a PDS from the server need with fill method code? Its already filled.

You note several basic issues here. It is true that at present you cannot serialize an entire object (that is, data and r-code in our case) and pass it across the wire as a parameter. You have to extract the data to be passed, and pass that (as a PDS or whatever else) as a data parameter into another object on the other side, which might or might not be an instance of the But as you note, the notion of passing the entire object across the wire is problematic in any case. The totality of the code needed on the server side certainly isn't relevant to a client. And if the client isn't OE, you can't pass the object anyway. Even if it's a different product version or for some other reason the r-code isn't fully compatible, you couldn't even reliably pass it even to OE. So the notion of passing the fully encapsulated object has severe limitations anyway (which aren't in any way distinctive to OE and ABL). So at some level there has to be an understanding of separating the data from the totality of all the logic that may affect the data in any case.

Tim, I think there is something interesting here, but I would like to suggest a branch in the discussion in order to avoid hijacking John's discussion. It seems to me that there are a couple of issues here that could get separated out.

The first question, relative to John's thread, is whether the use of a more pure OO approach has implications for the number of run-time components relative to other architectures. I think the answer is yes, depending. That "depending" comes from two sources. One has to do with whether one follows the 3GL OO model of having only entity objects and collections where it is obviously possible to get a very large number of entity objects instantiated. The approach of using a set object containing a PDS or temp-table avoids that proliferation and is, I think, one of the ways that a 4GL OO differs from 3GL. The other issue we have seen in a thread on the PEG is the notion of not only using individual entity objects for every instance, but building that single row entity object around a temp-table. Clearly, there are performance implications for having 10,000 temp-tables instantiated at the same time.

So, relative to the topic of John's thread, either the mixed relational object model he has proposed in whitepapers or the set oriented object model I have proposed significantly reduce the number of simultaneously instantiated objects compared with the pure OO approach and thus tend to avoid the issues you raise.

The second question, which I think is separate from this topic and might deserve its own thread, is whether an OO approach inherently results in a significantly larger number of simultaneously instantiated runtime components. I suspect the answer is "sort of", i.e., no more so than a superprocedure-rich non-OO model.

The third question is your observed performance issues. I certainly think that needs exploration, but I definitely think it deserves its own thread since it doesn't necessarily have anything to do with OO. In particular, I think we need to accumulate some experience in order to determine where those performance issues arise since the cost of instantiating a single runtime component is normally quite small and certainly doesn't produce the delays you are experiencing. So, the question is, what conditions produce it? Is it the number of simultaneously instantiated run time components? Is it memory usage or some other tuning issue? Is it something to do with the nature of those run-time units, e.g., like the large number of temp-tables issue reported on the PEG? Or is it something that is specific to the use of superprocedures and might thus actually be alleviated by a move to OO? All very interesting and worth exploration.

I would

suggest that one of the advantages of ABL coming to the OO table

late is that we are coming to the table after the broad development

of messaging backbones. The use of an ESB as the backbone for

distributed applications creates a rather different environment

than something like CORBA because of the strong orientation toward

technology neutral messages and the notion that it is up to the

consumer to decide what it wants to do with the message. This is

the sort of model I am advocating for OERA applications going

forward, i.e., anytime one is decoupling two elements enough to be

sending something across a wire, then the message should be

technology neutral like XML.

Tim, I think there is something interesting here, but

I would like to suggest a branch in the discussion in

order to avoid hijacking John's discussion. It seems

to me that there are a couple of issues here that

could get separated out.

The issue is wildly different load times for r-code, which, when loading a small number of small files - isn't an issue, but can be for larger number of bigger r-code files.

I was able to demonstrate this to PSC TS w/an example which showed ~75x different in load times between two pieces of r-code. For small sets of r-code, this isn't an issue, but as an OO system grows organically, there'll be an increasing amount of r-code loaded at a time, and I firmly believe these load times will turn into a serious issue.

I agree though - this is a separate issue from what this thread originally started with. John and I are doing some off-line discussions so he can get his head around my situation here, and we'll see where that goes.

Just a quick word to acknowledge that we got the information that Tim refers to regarding performance and have been working on it. We try to include ABL performance enhancements on each release and we are currently planning a number of them for a yet unnamed release, possibly in 2008.

Does this imply that you have identified some conditions which result in performance issues so that you have something targeted to improve? If so, it would be interesting to get some hints about what those conditions might be ... again, better on a new thread.

Just a quick word to acknowledge that we got the

information that Tim refers to regarding performance

and have been working on it.

Excellent.

We try to include ABL

performance enhancements on each release and we are

currently planning a number of them for a yet

unnamed release, possibly in 2008.

So will my client see their load performance improve sometime soon?

Working on it. No date yet.

My last posting to this thread re performance, to get it back on track.

Attempting to decompose your discussion, it does seem to me that there are both some things which are invariant across different UI models and some things which take on a very different character depending on the UI model. Thus, it can be very useful to consider different UI models in order to illuminate how each piece should be working in relationship to any other.

I suppose one extreme is the self-service character client ... although there are probably not a lot of new instances of those being created today. However, it is easy to see why applications rooted in this past had limited or now layering in the application because the whole enchilada was running on one box and, there wasn't and still isn't much in the way of tools for factoring an application into cooperating sessions.

Next up are the heavy client applications, notably those with ABL clients. There are probably a fair number of those out there from the days of client/server, where the separation forced some division of logic, but with ABL on both sides the exchange was dominated by proprietary forms of connection. But, here we had state maintained in the client and a general sense that the server side should be stateless, although there are various aspects of wanting to preserve state between calls, just not within a specific session.

Moving to non-ABL clients there is a push for more technology neutral forms of connection and a cleaner sense of operation by contract with neither side making assumptions about how the other will implement functionality. Note that this implies that there are a number of features in recent versions of ABL which are not useful across the client/server split, not only because non-ABL technologies don't support ABL constructs, but since things like the before-image capabilities of ProDataSets are very technology specific and are very close to assuming specific implementation on the other side of the wire. Non-ABL clients force us to think of a more generic interface and of utilizing these ABL-specific capabilities only on the server side.

Browser clients take us yet farther ... but might also bring us back in a way. Simple page-refresh type browser clients force all of the logic back onto the server. An AJAX client might provide more sophisticated behavior in the client and a more piecemeal dialog with the server, but ultimately we are still clear that that processing should be entirely about UI and nothing else. Depending on the application, we either keep state in the browser, which limits us significantly in the complexity of the application, or we need to have a server side representation of state in a way that is separate from any business logic ... i.e., almost back to all "processing" happening on one box, but with a clear separation of responsibilities into layers.

Now, in thinking about how to use OO and various ABL capabilities in an OERA context we are talking about either a new application from scratch or a substantial transformation of a non-OO, non-OERA legacy application with varying degrees of modernity. For the new application, it seems like we should be designing for independence from the UI, regardless of what initial technology is chosen for the UI. That, I think pushes us toward a very technology neutral, message-like interface between the UI and everything else. For a new application, I am also dubious that there will be a lot of cases of ABL clients, so the proprietary route isn't really available. We could lock into a sort of alternate proprietary mode by relying on .NET specifics, but that doesn't seem particularly smart and far-sighted to me.

For an application being transformed, we are more likely to have ABL clients in the picture to start. Certainly, the capabilities of 10.2 offers a stimulating way to dress up such applications without having to completely re-invent them. But, I think we should be clear that this is what it enables, i.e., tarting up without real transformation. For some, this will be an economically attractive option and will be "good enough", but it shouldn't be what we are building OERA models around. Thus, even if the first step in a transformation process might be to continue to use existing ABL clients because it is something that can be achieved more quickly and easily than completely replacing the UI layer, we should be clear that this is a stepping stone, not an end-point and thus I find it questionable to focus the transformation around using proprietary interfaces, even when this is possible.

I think this echoes the technology-neutral, message orientation one sees in applications distributed with Sonic. The clear message here is that there are clusters of functionality which belong together as a single service and those can be tightly coupled within themselves, but that interfaces between such services need to be loose, technologically neutral, and message-oriented.

Great summary.

So one of the challanges is about "state":
- how much state does a REQUEST need for it to be processed
- where does the state live
- should the service trust incoming "state"

The "state" problem can be illustrated when you have an interactive (or "chatty") user interface and for instance an order entry screen. When a sales clerk enters a product quantity, the sales prices calculation might need the customer, order total, currency, shipping address, etc to calculate an accurate sales price. So the scope to calculate a price could be the current state of the in memory order including the sales clerk id entering the order and his timezone. So there is probably some logic in the client that determines the boundary of the request.

Now when we have a calculated price in our in memory representation of the order (ProDataSet/XML), can the service trust this information when the business logic receives a "save order REQUEST" from the caller (UI)?

It depends how your service is implemented. If it simply accepts a ProDataSet and processes rows that are marked as "changed" it probably "assumes" that the prices have been validated earlier and are OK. This approach tends to expose the database schema through all layers.

But in the "transactions and accounts" pattern, proposed here http://c2.com/cgi/wiki?TransactionsAndAccounts, you don't exchange ProDataSets, but you process transactions. Your service would be called like this: "add 10 pieces of product X for customer Y". Notice the difference. In the first case the ProDataSet tells it all and in the 2nd case you only pass the data the service needs/trusts. This leads us to the 2nd dilemma of todays application model: how to write efficient service contracts? We tend to go for generalizated contracts (ProDataSets) to allowing generic plumbing. Since it's easier to write a framework that processes ProDataSets than a framework that has to process unique requests.

I think the ABL should focus on mapping classes to database tables and hide all the database persistence for developers, the same way it did in the old days when we executed "FIND customer. DISPLAY customer.". There should be no need to access database buffers in the ABL. On top of that the ABL should introduce a message mapping mechanism that allows developers to easily compose message contracts. Map classes to messages and hide the data transformation for developers (class to XML/PDS and vice versa). To emphasize ABL as a business tier language it should implement a transparant mapping to other databases like Oracle/MySql/MsSqlServer. That way ABL could stand the test of time and you would have platform neutral business logic and an advantage over the competition.

I don't think business logic should ever trust UI. In some browser apps, the server might be stateless and the browser might maintain state during the user interaction and thus be sending that state to the server at the conclusion, but the server should never trust the browser to have followed all business rules.

And, of course, the question of how much state is needed during UI and where that state lives is a classic of design and entirely a question of "it depends". One of the things it depends on is how big a problem one has if it turns out that the BL validation doesn't accept what the UI has already accepted. If this is something that happens only 0.01% of the time and there are no serious consequences of notifying the user to try again, then no big deal.

I would question this. Object-Relational mapping is not a simple, automatic conversion, even if one has complete control over the schema, which one often doesn't. Logic is needed in this mapping process and that is the reason it is appropriate for us to be writing our own data access layer. And, frankly, this is one of the easier development problems around and one quite suitable for dramatic productivity benefit through the use of generators.

But, again, I'm afraid that we have hijacked and diluted John's thread again. We need at least three threads here. One on the issue of how to combine OO and relational data handling in ABL, one on Tim's performance issues with large numbers of compile units, and one on your architectural concerns.

John, perhaps you should start a fresh thread with a fresh summary pointing back to this thread and we can try again.