R-Code and CRC Checking: A Whole New ERA for Compiled ABL Co

I am not sure if you are referring to the DataServer specific behavior, which has been relaxed in recent releases of OE 11.

However, the CRC check for the ABL against OE databases has been relaxed since OE 10.1C. There was a release note that was made available at that time but the documentation has since then been updated to include information on when the CRC mismatch may be OK. It is to allow r-code to execute when you add new fields to a table, for supporting online schema changes.

In the "Managing ABL Applications" book, under the "Appendix B: R-code Features and Functions" chapter, under the section "Database CRC changes that do not prevent r-code execution", there are two notes that explain the scenario under which the AVM will still execute the r-code:

- You add new field to a table with a _Field-rpos value greater than any other field in the table. In other words, the new field cannot be inserted into an existing _Field-rpos sequence.

- You make changes to existing fields that do not affect the values of _Field-name, _dtype, _Field-rpos, _Extent.

So as long as you maintain the field name, data type, field rpos and extent of all the fields that existed when you compiled the r-code, and any new fields added have their field rpos higher than the last field rpos value at the time of compilation, the AVM will allow that r-code to execute.

If you find knowledge base articles (or documentation) with outdated information, I suggest you contact Technical Support and bring it to their attention.

I don't think the changes I've noticed in CRC checks are DataServer-specific, although I think DataServer applications may benefit the most from the changes.

The quote I had provided is from a generic article about how/what schema info contributes to the "CRC" checks.  The KB article is P104149.

I think the documentation you pointed me to is still wrong/understated where "_field_rpos" is concerned.  It appears to me, based on experimentation, that "_field_rpos" differences are no longer a factor that will prevent r-code execution.  Ie. based on the subtle footnote in the KB article, Progress runs an "extra runtime check" and determines that schema is compatible anyway, despite any "_field_rpos" differences.

So it does NOT appear necessary to maintain the field rpos that existed when you compiled the r-code. The AVM will allow that r-code to execute anyway. This became evident when we noticed that we could use a DF schema dump of a production database to compile source code against (ie. after creating a schema-only database).  OE now allows us to release that compiled  r-code to run on the actual production database.  Previously this was prevented because of very obscure issues in "_field_rpos"

That is not true. We still rely on the field rpos and will/should not allow the r-code to run if the field rpos values do not match. We need to validate the field rpos otherwise that could lead to corruption of data, where the data could end up in the wrong field.

When you dump the .df of a table, the .df carries the field rpos value for the fields with it (unless you chose not to keep the POSITION values) so it is expected that you load the .df into a db, compile the r-code and are able to run the same r-code.

It would be interesting to see exactly the situation you are referring to, to see if it doesn't fall into the case described in the documentation or it is something else. if you find a case where the field rpos on an existing field actually changed and you were allowed to run the existing r-code, then you should log a call with Technical Support because that is not supposed to happen.

With regards to DataServers ... it's CRC check was always more rigorous than applications compiled against an OE database.  This began to get dismantled in OE 11.2 when we added "position-independent'" r-code, meaning that the physical position of a field in the server database could change (as long as you re-execute a schema pull) to realign _Field-rpos with the new physical position on the server.  

Starting in OE 11.5, DataServers have removed all the old restrictions on CRC such that the CRC requirements are now the same as OpenEdge.  You can make any changes to existing fields that you want as long as you don't affect the values of _Field-name, _dtype, _Field-rpos, _Extent, just like OE.  You can also add fields without triggering a CRC mismatch or you can delete a field as long as there are not references to it in your r-code.  

The DataServer documentation describes these features in a section called, "Compiling OpenEdge procedures".

<p>Again, I don't think we are talking about DataServers only.  My repro is for an OE 11.3 database and OE 11.3 ABL.</p>

<p>Apparently I'm not the only one with confusion.  What can we do to request more documentation on the changes from Progress?  We actually have a software deployment model that relies on the changes in behavior and it is extremely worrisome that it is totally undocumented.</p>

<p>Here is a repro that used to generate CRC errors and does not anymore.</p>

<p> /* Prog: */

DEFINE VARIABLE v_Number AS CHARACTER NO-UNDO.

MESSAGE "Enter a Number: " UPDATE v_Number FORMAT "X(10)".

CREATE TestTable.

TestTable.TestField1 = v_Number.

TestTable.TestField3 = INTEGER(v_Number).

</p>

<p> /* DB #2 */

CREATE TABLE PUB."TestTable"(

"TestField1" VARCHAR (20) DEFAULT '' PRO_ORDER 10 PRO_CASE_SENSITIVE 'N' PRO_FORMAT 'x(8)'  )

AREA "ddn_log" PRO_HIDDEN 'N' PRO_LABEL 'TestTable' PRO_DUMP_NAME 'TestTable'

ALTER TABLE PUB."TestTable" ADD COLUMN ( "TestField2" BIT DEFAULT 'no' PRO_FORMAT 'yes/no'  )

ALTER TABLE PUB."TestTable" ADD COLUMN ( "TestField3" INTEGER DEFAULT '0' PRO_FORMAT '->>,>>9'  )

ALTER TABLE PUB."TestTable" DROP COLUMN  "TestField2"

</P>

<p> /* DB #1 */

CREATE TABLE PUB."TestTable"(

"TestField1" VARCHAR (20) DEFAULT '' PRO_ORDER 10 PRO_CASE_SENSITIVE 'N' PRO_FORMAT 'x(8)'  )

AREA "ddn_log" PRO_HIDDEN 'N' PRO_LABEL 'TestTable' PRO_DUMP_NAME 'TestTable'

ALTER TABLE PUB."TestTable" ADD COLUMN ( "TestField3" INTEGER DEFAULT '0' PRO_FORMAT '->>,>>9'  )

</P>

<P> /* Different R-pos : */

for each _field no-lock where _field._field-name begins "Test":

display _field._field-name _field._field-rpos _field._file.

end.

┌──────────────────────────────────────────────────────┐

│Field-Name                       field-rpos File-recid│

│──────────────────────────────── ────────── ──────────│

│TestField1                                2      15209│

│TestField3                                4      15209│

┌──────────────────────────────────────────────────────┐

│Field-Name                       field-rpos File-recid│

│──────────────────────────────── ────────── ──────────│

│TestField1                                2      15209│

│TestField3                                3      15209│

</P>

<

Just to review, the "TestTable" is created via simple SQL deltas.  In the case of DB2, an intermediate TestField2 is dropped.  In the case of DB1 no TestField2 was ever created in the first place.

DB2 leaves an rpos schema "hole", as they used to be called (based upon what little I know about such things).

Compiled r-code against one can be executed on the other and vice-versa without CRC errors. This is exactly what the KB article on the topic had said (P104149).  Can someone please independently verify this repro, since Fernando Souza and I aren't on the same page yet?  Can someone also please read the KB article (P104149) - which is predominantly NOT about DataServers?  Please let me know if you read the footnote in the same way that I do - ie. that CRC's alone will not prevent code execution, but will trigger an "extra runtime check".

My biggest concern is relying on a behavior whose only documentation is (at this point) just a subtle footnote in a KB article.

A secondary concern about this ABL change is that, while it is a very welcome one, it is not without its issues.  While CRC checks are not preventing code execution, they are allowing yet another problem to occur.  If you run one of the rcodes you get an error (the one compiled with larger field rpos positions).  This won't run on the DB #1 with the smaller field rpos.

I get the following:

SYSTEM ERROR: bffld: nxtfld: scan past last field. (16)

This seems like an error you'd rather not encounter if it can be identified earlier.

[/collapse]

So what is the next step?  Am I correct to say that this confusion needs to be addressed with either a KB or new documentation?  I would have opened a support case by now but somehow Progress decided to disable my SupportLink based on my customer serial number or something.  Very frustrating.

[/collapse]

Yes please resend.  

I'm told by the people here that keep track of it that we have a written contract that says we have access to Progress support.  

Nobody who has to do development work with OpenEdge products should go without support.  There are too many, shall we say, "issues" that come up.  How is it that you sell OE development licenses to your so called "end users" and then tell them they aren't supported?

[/collapse]

"There are too many, shall we say, "issues" that come up".

There are no "issues" in any other development environment....

I'm looking for constructive ways of moving forward with scary and misleading error messages that seem to say we are experiencing database corruption (although that is not the case).

Would someone with access to Progress Support please see what we can do to get the ball rolling?  To start with it would be nice to see a KB or documenation to expand on the surprising statement that "A MISMATCHING CRC IS NO LONGER AN ERROR CONDITION ".  If we move forward then some day we will have one less issue on our hands.

[/collapse]

First, thanks for the reproducible.

Just to close the loop on this, the reproducible provided shows a bug. As I explained previously, this is *not* supposed to happen. The caveat is that the bug only happens when you create the table via SQL. If you create the same table via an ABL client, you will get the expected CRC error. Therefore, once we fix the bug that is allowing this r-code to execute, you will *not* be able to run it, and that is what is expected behavior. The error you reported (16) is one of the many issues that can happen if we do not stop you from running the outdated r-code. The documentation I pointed to (that describes the cases where we allow you to still run the r-code (i.e. new fields added)) is still accurate. You just ran into a bug (which has already been logged) that needs to be addressed.

Thanks Fernando.  I am optimistic for a resolution to that open bug.

I have an open case related to the larger issue - finding out if/when CRC's are exclusively used to disqualify programs from being executed.  I think this is still a topic of confusion.  I'll definitely follow up and let you know what comes out of that case.  According to that KB article, (000010488, What Metaschema fields are involved in table and database CRC calculations) it seems that CRC's will now play a subordinate role in determining if r-code is compatible.  CRC's no longer create an error condition or, by themselves, prevent code from running on a given database.  The "extra runtime check" is apparently what really matters (going forward).

From what I can tell, it looks like we've reached a tipping point in the old "CRC"-based strategy.  Progress has continually relaxed previous constraints which had inhibited online dictionary changes, notably those kinds of changes made dynamically via SQL DML and those not requiring exclusive schema locks.

The expectation I have is that I should be able to (1) export my schema definitions from a source database to a schema-only database, (2) compile against it and then, (3) the result should be r-code that is compatible with the source.  This is not something that was possible until the very most recent releases of OE.  My recollection is that the scenario was prevented because of some of the more unusual OE schema internals (OE _field-rpos holes etc - things which aren't exposed very well, and whose values can't be influenced easily by a developer).

There is now a KB on the bug that was reported:

Running r-code leads to SYSTEM ERROR: bffld: nxtfld: scan past last field. (16) http://knowledgebase.progress.com/articles/Article/Running-r-code-leads-to-SYSTEM-ERROR-bffld-nxtfld-scan-past-last-field-16

Also the following two articles were updated with the information gathered while working on this issue. What I gather now is that the logical field positions (_field-rpos aka POSITION) are as important as they always were (and as important as the datatype) for the purpose of detecting incompatible .r-code. Any differences in these field positions will generate a CRC mismatch AND prevent .r-code from running. There are other schema-related values that may change, generate CRC mismatch, but still allow old .r-code to remain compatible and continue to be executed. The _field-rpos does not fall into this category in OE 11.3 anymore than it did in OE 11.1.

What Metaschema fields are involved in table and database CRC calculations? http://knowledgebase.progress.com/articles/Article/P104149 Cyclic Redundancy Check Explained http://knowledgebase.progress.com/articles/Article/20245