Startup procedures & blue/green deployment

Posted by hutcj on 20-Feb-2020 21:13

We have a fat ChUI/tty application that serves a few hundred direct connect clients. Recently, we implemented a blue/green deployment model. This has had a number of benefits such as preventing issues caused by compiling into the live rcode directory and having the old version ready to switch back to in case something goes wrong with the new version.

In short, there is a "blue" and "green" directory on the production server where all the build artifacts of the application such as startup scripts and rcode are deployed. The user login script reads in the active environment from a text file and then proceeds to run the blue or green version's startup script, whichever is currently "active". Whenever deploying a new version of the application, build artifacts are copied to the currently "inactive" directory where it can be verified before releasing. When all is confirmed, an admin script is run to switch the active version listed in the environment text file.

This has worked flawlessly for new sessions. However, we are having trouble getting active sessions switched over to the new environment. It is easy enough to detect the change from the primary startup procedure and manipulate the PROPATH to point to the new environment. However, if the startup procedure itself has been changed, there is potential for run-time errors for active sessions when the switch is made. Is there any way to force a restart of the ABL startup procedure? I have done some tests, and if the application encounters an error or the user CTRL-C's, the startup procedure is re-loaded from the updated PROPATH.

But I don't have a way to restart the procedure. I considered doing something like RUN start.p followed by QUIT or something, but this would just make the old version a parent procedure and the new version a child procedure. In a bash script, you can prepend any command with "exec" and that will cause the new command to actually replace the current process - anything after that statement would not be executed. I don't think ABL has anything like that.

I know it is also feasible to just force quit the session after displaying a message that the user must re-login, but am looking for something less intrusive - hoping for a seamless, automatic switch to the new environment.

Another consideration was to just let the active sessions continue on running the previous version, but this came with other problems. What if we deployed new changes and switch environments twice while the session was active? Then we have compiling into live rcode directory problems again. What if the change involved a database refactoring? Running the old version could cause errors.

One might also argue to only switch active environments when there are no active sessions, but the tolerance for down time is always becoming stricter, and there will inevitably be times that a bug needs to be fixed ASAP with an immediate release.

Any ideas? Is forcing the session to quit the only viable option?

Posted by tim on 21-Feb-2020 16:14

That should work for interactive (non-batch mode) clients. An unhandled STOP condition in a batch process does *not* restart the application.

All Replies

Posted by gus bjorklund on 20-Feb-2020 21:28

you could send a SIGINT to the user's _progres process. that would have same effect as user typing ctrl-c

Posted by hutcj on 20-Feb-2020 21:52

When updating the new PROPATH from inside the startup procedure, the check is only performed when the user is at a neutral state, such as loading a menu. We probably wouldn't want to force a switch while they are inside a program or in the middle of editing a record. Without having such context from the operating system, is there a way we could ensure the switch isn't interrupting anything important? Check if the session has an active transaction or exclusive locks?

Posted by Peter Judge on 20-Feb-2020 22:24

Another gotcha comes if you use OOABL static members. There's no way to remove/clean/destroy/reset them automagically without restarting the session. If you have code that can do the reset, that may mitigate those risks.
 
 

Posted by hutcj on 21-Feb-2020 14:40

Hang on a sec...is there a reason I can't just STOP? I was starting to write up this convoluted procedure to get the current pid and run a kill -SIGINT <pid>, but saw in the protermcap file that CTRL-C can be referenced by STOP. Other than perhaps the issue with OOABL static members (I don't think we use any), is there any downside to this?

1. Detect environment change

2. Update PROPATH

3. STOP (by definition this reloads the startup procedure)

Maybe I was way overthinking this...

documentation.progress.com/.../index.html

Posted by tim on 21-Feb-2020 16:14

That should work for interactive (non-batch mode) clients. An unhandled STOP condition in a batch process does *not* restart the application.

Posted by hutcj on 26-Feb-2020 16:08

Ok, we may have run into a problem with this approach. Database schema was updated, and clients today are reporting CRC errors. It appears that the old sessions still hold onto the old schema cache. Is there a way from the application to have the session re-read the schema or a schema cache file?

Posted by tim on 27-Feb-2020 02:38

Your options will depend on what version of OE you're running. I'm going to assume you're at 11.x and not 12.x. If that is the case, the best I can offer is that you can compile an empty .p (assuming the clients can compile). When the compiler begins, it first checks to see if the schema has been changed and if it has, it refreshes its cache.

Posted by hutcj on 28-Feb-2020 14:44

Yes, we are on 11.7.5. I'll give this a shot. Are there better options available in 12.x?

Posted by tim on 02-Mar-2020 08:59

We're getting there, but we haven't fully arrived. New online operations in 12.2 (e.g., database triggers, rename field) do automatic notification of all connected clients so they automatically update their schema cache. Unfortunately, the automatic notification for the already existing online schema changes (add new table, add new field, etc.), which are more common, did not make it under the wire for 12.2.

So you still have to rely on the compile of an empty .p to cause the refresh of the schema cache.

In 11.7.6 (coming soon) and in 12.1, the client will also refresh its schema cache if you prepare a dynamic query.

Posted by hutcj on 02-Mar-2020 14:21

That last point sounds like a pretty simple solution. I forgot we started using the -rr parameter on our clients (run-time only, doesn't allow compiling) because every once in a while some error would occur that allowed a client to drop into the procedure editor. We have fixed most of them, but it's too big a risk to live with, so we disabled access entirely with -rr.

This thread is closed