I have a task of migrating data from a CSV to Sitefinity ecommerce. The tedious part is importing 200k products from CSV. Every single product savechanges call takes time. I tweaked the code as below to run in batches of 10. Still the performance is really sad. To import 1,000 products it takes around 10 minutes. For 1 lakh proudcts it is taking more than 10 hours. Please advise on this as this is really very slow and I am stuck with a migration task to reduce the time to 1 hour.
int skip = 0;
int take = _batchSize == 0 ? 10 : _batchSize;
int currentCount = 0;
int totalCount = productImportCsvs.Count();
while (currentCount < totalCount)
string skusNotImported = string.Empty;
IEnumerable<ProductImportCSV> chunkedProducts = productImportCsvs.Skip(skip).Take(take);
skip = skip + take;
currentCount = skip;
foreach (ProductImportCSV product in chunkedProducts)
ImportSingleProduct(dynamicModuleManager, scheduledJob, catMgr, liveProducts, masterProducts, ref successfullyImportedProducts,
ref errorImportProducts, product, deleteExistingProducts);
I had already tried it even before writing the post :-(. The performance was not satisfactory and hence writing this post. Actually some part of implementation I had taken from the code itself. As mentioned 200k records even with that code took almost 10 to 15 hours. We need something which can do the work atleast within 2 hours.
Do you have some sort of SQLMapper class. I am planning to write SQLBulkCopy and finish the job, which I am completely against :-(
Tried this code already. The methodImportSingleProduct what I mentioned does the same :-(. Still it is slow. Any other suggestions.