Bulk processing (Run Bulk Loads) could be interrupted. And when restarted from
C:\Pro2\db_name\bprepl\repl_mproc
it will take off from where it was stopped at. That is good. However it takes quite long time for "mproc" program to get to the point where it will start. You can see it in the log below.
Began Mass Replication on 05/21/18 at 10:26:44
Processed 32,100,000 records so far. Time is 12:18:29.
0 of these either failed to replicate or failed verification.
0 records were locked and could not be copied.
Processed 32,200,000 records so far. Time is 12:24:56.
So we had 32 million records processed the day before. When we restarted Bulk process at 10:26 it took almost 2 hours to start processing at 12:18. After that the process was moving fast as it was moving before. Just that the first part when it was looking where it was left off took considerable time.
Valeriy is exactly correct in his description of how the new and old versions of bulk load restart logic operate.
Pro2 version 5.0.1 introduced a new feature in which ROWID cache (comma delimited list of ROWIDs) was stored for in the bulk load replcontrol record. The Bulk_Max_Cache property controls how many cache entries are stored (bigger is typically better with 25 being the default.)
WHEN using ROWID cache, the time to restart from a failure is very quick. In previous versions of Pro2 bulk load (<= 5.0.0), only a count was stored in the bulk load replcontrol record, so upon restart the load program read from the beginning of the table and ignored rows below the counter. If the load was stopped after a large number of rows had been loaded, then you would notice a significant delay associated with firing the old restart logic.
Be aware, the bulk load process is running, it's just trying to get back to the record loaded before the failure.
Hope this helps,
Terry
From Pro2v5.0.4 docs:
[quote]The Bulk_Max_Cache property in the tmpl_mreplproc_restart-auto-push template stores the last processed row-ids in the cache. In general, if the loading fails or stops during the bulk-load process and then restarted, the loading starts from the beginning. In such cases, the Bulk_Max_Cache property allows you to start the bulk-load process from the last processed row-ids stored in the cache instead of loading from the beginning. This property defaults to 25 and you can modify its value using the Properties tab in the Pro2 Admin Tool.[/quote]
Hi Dmitry,
This is because in older versions, when the program is restarted, the loader reads all the records again from scratch and skips them based on the counter.
What version of Pro2 are you using? In Pro2v5.0.4, they changed the procedure, as far as I can see. Now the Bulk_Max_Cache property is used to indicate the number of last cached ROWIDs in order to start processing from them.
This cache is stored in replcontrol at loading time. For example:
for each replcontrol where groupid = "BULKCOPY" no-lock.
displ replcontrol.codeval3 format "x(120)"
end.
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│Value3 │
│────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
│Count=400000;RowidCache=0x00000000000d372d,0x00000000000d372c,0x00000000000d372b,0x00000000000d372a,0x00000000000d3729,0│
I also use Pentaho Data Integration solutions instead of the standard Bulk-loading. This solution also appeared in Pro2v5.0.4.
With it, the primary synchronization speed is increased many times over.
For comparison, a table with 10 000 000 records via Pro2 Bulk-Loading is loaded in 13-16 minutes. But with PDI it takes only about ~500 seconds.
From Pro2v5.0.4 docs:
[quote]The Bulk_Max_Cache property in the tmpl_mreplproc_restart-auto-push template stores the last processed row-ids in the cache. In general, if the loading fails or stops during the bulk-load process and then restarted, the loading starts from the beginning. In such cases, the Bulk_Max_Cache property allows you to start the bulk-load process from the last processed row-ids stored in the cache instead of loading from the beginning. This property defaults to 25 and you can modify its value using the Properties tab in the Pro2 Admin Tool.[/quote]
Valeriy is exactly correct in his description of how the new and old versions of bulk load restart logic operate.
Pro2 version 5.0.1 introduced a new feature in which ROWID cache (comma delimited list of ROWIDs) was stored for in the bulk load replcontrol record. The Bulk_Max_Cache property controls how many cache entries are stored (bigger is typically better with 25 being the default.)
WHEN using ROWID cache, the time to restart from a failure is very quick. In previous versions of Pro2 bulk load (<= 5.0.0), only a count was stored in the bulk load replcontrol record, so upon restart the load program read from the beginning of the table and ignored rows below the counter. If the load was stopped after a large number of rows had been loaded, then you would notice a significant delay associated with firing the old restart logic.
Be aware, the bulk load process is running, it's just trying to get back to the record loaded before the failure.
Hope this helps,
Terry
Thank you. I am using 4.6.7 from Oct 2016.