July 31st, 2013, 11:39 PM
MySQL Connector Data Dump
I wrote an application so I can manage the products for my e-commerce site. It uses MySQL 64 bit. Problem is some of my stores have in excess of 3 million products. The files that contain the data are all CSV. I parse the lines... format the data... write to table. On most of the stores which contain 10 - 60 thousand products it only takes max 3 min to write it all in. I have been running it on this one store for 6 hours now and it has barely made a dent. I know there are almost 3mil products in this one but it should be going much faster given what the speed was for the 60K stores. Is there a faster approach to handling a massive dump like this? I should also add that there are quite a few indexes. These indexes are there for ordering and fulltext search once the products have been activated on the site. I cannot create the table without the indexes and add them later since there is already preexisting data that is using the indexes. I know this is most likely what is slowing it down but I am at a total loss what to do next.
Last edited by RobNHood; July 31st, 2013 at 11:43 PM.
Tags for this Thread
Click Here to Expand Forum to Full Width
This is a Codeguru.com survey!