Batch job for performing an import is taking a long time to

Posted by jsniemi79 on 12-Oct-2014 14:11

I have a csv file that has roughly 320,000 records with 9 fields on each record.  I have a batch process setup to pull this file from a remote ftp server and perform an update/create nightly.  Upon creation or update of the record, 3 triggers are executed.  2 to attach related records and one to update a field on the object in Rollbase.  I kicked off the process manually and it too almost 26 hours to complete the process.  Obviously, that will make it difficult to update daily.  We've had other import processes that didn't take nearly that long to complete.  Is there anything I can do to make this process speed up?  I have screen shots of what I am seeing in the document below.


All Replies

Posted by Gian Torralba on 13-Oct-2014 10:28

Did you try testing it partially and disabling the triggers just to see how much time the import is taking without running any triggers. Let me know how much time an import is taking by uploading a sample record of 50,000. Another cause of this is if there are other processes running in the system jobs section in the admin settings page.

Posted by jsniemi79 on 13-Oct-2014 10:33

This particular process normally has about 50K records.  It ran again this morning with 52K and took just under 6 hours.  That was with the triggers being fired as well. 

I can't test it right now as the customer is working in there today.  I can fire off a test this evening running in manually as bulk and see what the time comes back as.  I know we've run other much larger imports that haven't take as long.

I believe this was the only thing running at the time because it was over night.  I have other batch jobs scheduled that were in the queue, but that was all that was running at that time according to the job log file.

Posted by Gian Torralba on 13-Oct-2014 10:46

Thanks for the information you provided. I just calculated that approximately, 200 records are create every minute since the process lasted 26 hours. The 52,000 record ran slower with an estimate of 144 records per minute. What process are you using for importing those records in the batch job?

Posted by pvorobie on 13-Oct-2014 10:48

Speed of import may be reduced if lots of relationships have to be resolved in process. Records without relationships typically can be created much faster.

Posted by jsniemi79 on 13-Oct-2014 10:48

I am using the schedled ftp import.  Here is how I have it set up.

Posted by jsniemi79 on 13-Oct-2014 10:51

I understand. This only has a few relationships and is pretty basic.  We have to be able to have this many and most likely more going forward.  

Posted by on 13-Oct-2014 11:57


Our imports also take a long time to process. Especially in our main employee object which has 140 relationships and about 10 triggers. It takes about two seconds to process one record. Which means that it takes 2 hours to import 3600 records and we have an import batch job running every night on each of our customer zones. (We are using import maps with about 30 to 50 mapped fields).

We tried with disabling every trigger and it actually kinda helps (With no trigger being activated, we are able to have about 3 times faster processing of imports).

But still, we would like it to be way faster because we'll soon have too much customers which means the import batch jobs are getting delayed and are now running during the day (They are planned to be processed at 8pm but are so long to process that it goes almost all day long (20hours non stop of imports)).

Could the "bulk" option be added to import batch jobs? Right now, it only is possible when doing manual imports but could be very useful for importing large amount of data using import batch jobs.

If we aren't able to improve performance of imports, we'll soon need a second STORAGE server in our private cloud instance to divide import processing of customers into two servers, but we would like to avoid that if possible.

Kind regards,


Posted by Gian Torralba on 13-Oct-2014 13:36

I will ask the developers if adding the bulk functionality is possible in the future. However, I believe that using the bulk method will not run any triggers so you will have to manually run those triggers,

Thank you,
Posted by Gian Torralba on 21-Oct-2014 13:52

May I know how many fields you have in the object you are trying to import with? I just did a 50,000 import that only took minutes to finish. However this doesn't have any triggers and I only imported a single column value to the object. The create pick list value is also unchecked when I tested it.

Thank you,
Posted by bheavican on 21-Oct-2014 14:34

There are 9 fields, two attach related triggers and the pick list is set to create.

Posted by Gian Torralba on 22-Oct-2014 14:07

Can you uncheck the picklist create and then disable all running triggers? It should import the records rather quickly.

Let me know how it goes,

Thank you,
Posted by jsniemi79 on 22-Oct-2014 18:02

Are you just looking for a benchmark time on the performance?  We have active users in here, so it isn't something I can just do during the day.  I also need the picklist values to be created and the relationship triggers to fire nightly.

Posted by Gian Torralba on 23-Oct-2014 10:23

I see. Yes, this is just to test the performance of your import process. As I mentioned previously, I imported 50k records within minutes (Using my local machine). I used a newly created object with a single import field. If you have a test environment with the same application that would greatly help us debug why the import is so slow.

Let me know how it goes.

Thank you.

Posted by jsniemi79 on 24-Oct-2014 08:41

I tried to clone the object and then run an import to test this.  I got an error that the object ID didn't exist.  I've had this happen in the past too.  If I look at my new object, it shows that the ID exists.  I wanted to get an exact copy of this, to make sure the test was accurate.  I guess I'll have to create a new object from scratch and try that.

Posted by jsniemi79 on 24-Oct-2014 08:51

I created a new object completely from scratch and got the same error message. 

I am at a loss on what to try next.

Posted by jsniemi79 on 24-Oct-2014 10:15

I had our server bounced and was able to process this new object without the error referenced above.  That seems like  a bug to me.

Here are the results of the import run in Bulk which took 3 minutes and 50 seconds:

[2014-10-24 11:08:06,520] Job "Create new records: Progress Tests: BULK for SRS Distribution Size=2823 KB [Row: 0, Completed: 0, Batch: 0ms]" (5999) placed in the queue.

[2014-10-24 11:08:06,523] Started: "Create new records: Progress Tests: BULK for SRS Distribution Size=2823 KB [Row: 0, Completed: 0, Batch: 0ms]" (5999)

[2014-10-24 11:08:06,525] Create new records: Progress Tests: BULK for SRS Distribution Size=2823 KB [Row: 0, Completed: 0, Batch: 0ms]

[2014-10-24 11:11:56,648] Finished: "Create new records: Progress Tests: BULK for SRS Distribution Size=2823 KB [Row: 54516, Completed: 54514, Batch: 404ms]" (5999)

Please let me know what you need me to do next.

Posted by Gian Torralba on 28-Oct-2014 10:48

I will check the clone object issue. Can you create a custom relationship to the object you need to attach this from and then add the same triggers that runs when you are doing an import so that we can test what really is causing the slowness of the import process.

Let me know how it goes,

Posted by jsniemi79 on 28-Oct-2014 14:01


I am not sure what you are asking me here.

Posted by Gian Torralba on 28-Oct-2014 14:36
Sorry, please follow this steps so that we can trace what is the cause of the slowness.

1. Add relationship to the cloned object to the related objects. (You told us that you have two attach triggers and an update trigger). We need to replicate this process in the cloned object.

Object A
Object B
Object C
Object C (Cloned)

Object C (Cloned) must have the same relationship as Object C to Object A and B so that we can add the attach triggers in Object C (Cloned) and after that you can retest the import. The bulk test only ran in minutes so I want to know how long will it take if those triggers are enabled.

You can delete the custom relationship after the testing is done.

Thank you,
This thread is closed