Sitefinity 200k products importing very slow

Hi Team,

 I have a task of migrating data from a CSV to Sitefinity ecommerce. The tedious part is importing 200k products from CSV. Every single product savechanges call takes time. I tweaked the code as below to run in batches of 10. Still the performance is really sad. To import 1,000 products it takes around 10 minutes. For 1 lakh proudcts it is taking more than 10 hours. Please advise on this as this is really very slow and I am stuck with a migration task to reduce the time to 1 hour.

 int skip = 0;
                int take = _batchSize == 0 ? 10 : _batchSize;
                int currentCount = 0;

                int totalCount = productImportCsvs.Count();
                while (currentCount < totalCount)
               
                    string skusNotImported = string.Empty;
                    try
                   

                        IEnumerable<ProductImportCSV> chunkedProducts = productImportCsvs.Skip(skip).Take(take);
                        skip = skip + take;
                        currentCount = skip;

                        foreach (ProductImportCSV product in chunkedProducts)
                       
                            if (string.IsNullOrEmpty(product.SKU))
                           
                                continue;
                           

                       
                            ImportSingleProduct(dynamicModuleManager, scheduledJob, catMgr, liveProducts, masterProducts, ref successfullyImportedProducts,
                                ref errorImportProducts, product, deleteExistingProducts);
                       

                           
                        catMgr.SaveChanges();
                   

Hi Praneeth,

You can check the Product import widget sample.

More information and installation instructions is available in that article.

Regards,
Svetoslav Manchev
Telerik

HiSvetoslav ,

 I had already tried it even before writing the post :-(. The performance was not satisfactory and hence writing this post. Actually some part of implementation I had taken from the code itself. As mentioned 200k records even with that code took almost 10 to 15 hours. We need something which can do the work atleast within 2 hours.

Do you have some sort of SQLMapper class. I am planning to write SQLBulkCopy and finish the job, which I am completely against :-(

 Thanks

Praneeth

Hello Praneeth,

In order to speedup the product creation, you can try to use manager.Lifecycle.Publish(item). Also, SaveChanges() should be called in batches for example for every 50 or 100 items created.

You can check the attached code sample. This should speedup the creation.

Regards,
Svetoslav Manchev
Telerik

Tried this code already. The methodImportSingleProduct what I mentioned does the same :-(. Still it is slow. Any other suggestions.

Thanks

Praneeth

Hello Praneeth,

What is the time you have experienced for 200k products using the latest example provided? During the tests we have 20k products have been created for about 18 min.

In addition, would you please upload your entire code sample in order to inspect and test it.

Regards,
Svetoslav Manchev
Telerik