CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 3 of 3

Threaded View

  1. #1
    Join Date
    Jul 2013
    Location
    USA
    Posts
    30

    Unhappy MySQL Connector Data Dump

    I wrote an application so I can manage the products for my e-commerce site. It uses MySQL 64 bit. Problem is some of my stores have in excess of 3 million products. The files that contain the data are all CSV. I parse the lines... format the data... write to table. On most of the stores which contain 10 - 60 thousand products it only takes max 3 min to write it all in. I have been running it on this one store for 6 hours now and it has barely made a dent. I know there are almost 3mil products in this one but it should be going much faster given what the speed was for the 60K stores. Is there a faster approach to handling a massive dump like this? I should also add that there are quite a few indexes. These indexes are there for ordering and fulltext search once the products have been activated on the site. I cannot create the table without the indexes and add them later since there is already preexisting data that is using the indexes. I know this is most likely what is slowing it down but I am at a total loss what to do next.
    Last edited by RobNHood; July 31st, 2013 at 11:43 PM.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured