Sitefinity 200k products importing very slow

Posted by Community Admin on 05-Aug-2018 12:48

Sitefinity 200k products importing very slow

All Replies

Posted by Community Admin on 05-Dec-2014 00:00

Hi Team,

 I have a task of migrating data from a CSV to Sitefinity ecommerce. The tedious part is importing 200k products from CSV. Every single product savechanges call takes time. I tweaked the code as below to run in batches of 10. Still the performance is really sad. To import 1,000 products it takes around 10 minutes. For 1 lakh proudcts it is taking more than 10 hours. Please advise on this as this is really very slow and I am stuck with a migration task to reduce the time to 1 hour.

 int skip = 0;
                int take = _batchSize == 0 ? 10 : _batchSize;
                int currentCount = 0;

                int totalCount = productImportCsvs.Count();
                while (currentCount < totalCount)
               
                    string skusNotImported = string.Empty;
                    try
                   

                        IEnumerable<ProductImportCSV> chunkedProducts = productImportCsvs.Skip(skip).Take(take);
                        skip = skip + take;
                        currentCount = skip;

                        foreach (ProductImportCSV product in chunkedProducts)
                       
                            if (string.IsNullOrEmpty(product.SKU))
                           
                                continue;
                           

                       
                            ImportSingleProduct(dynamicModuleManager, scheduledJob, catMgr, liveProducts, masterProducts, ref successfullyImportedProducts,
                                ref errorImportProducts, product, deleteExistingProducts);
                       

                           
                        catMgr.SaveChanges();
                   

Posted by Community Admin on 10-Dec-2014 00:00
Hi Praneeth,

You can check the Product import widget sample.

More information and installation instructions is available in that article.

Regards,
Svetoslav Manchev
Telerik
 
Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
 
Posted by Community Admin on 10-Dec-2014 00:00

HiSvetoslav ,

 I had already tried it even before writing the post :-(. The performance was not satisfactory and hence writing this post. Actually some part of implementation I had taken from the code itself. As mentioned 200k records even with that code took almost 10 to 15 hours. We need something which can do the work atleast within 2 hours.

 

Do you have some sort of SQLMapper class. I am planning to write SQLBulkCopy and finish the job, which I am completely against :-(

 Thanks

Praneeth

Posted by Community Admin on 15-Dec-2014 00:00
Hello Praneeth,

In order to speedup the product creation, you can try to use manager.Lifecycle.Publish(item). Also, SaveChanges() should be called in batches for example for every 50 or 100 items created.

You can check the attached code sample. This should speedup the creation.

Regards,
Svetoslav Manchev
Telerik
 
Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
 
Posted by Community Admin on 15-Dec-2014 00:00

Tried this code already. The methodImportSingleProduct what I mentioned does the same :-(. Still it is slow. Any other suggestions.

 

Thanks

Praneeth

Posted by Community Admin on 18-Dec-2014 00:00
Hello Praneeth,

What is the time you have experienced for 200k products using the latest example provided? During the tests we have 20k products have been created for about 18 min.

In addition, would you please upload your entire code sample in order to inspect and test it.

Regards,
Svetoslav Manchev
Telerik
 
Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
 
This thread is closed