CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 3 of 3
  1. #1
    Join Date
    Jul 2013
    Location
    USA
    Posts
    30

    Unhappy MySQL Connector Data Dump

    I wrote an application so I can manage the products for my e-commerce site. It uses MySQL 64 bit. Problem is some of my stores have in excess of 3 million products. The files that contain the data are all CSV. I parse the lines... format the data... write to table. On most of the stores which contain 10 - 60 thousand products it only takes max 3 min to write it all in. I have been running it on this one store for 6 hours now and it has barely made a dent. I know there are almost 3mil products in this one but it should be going much faster given what the speed was for the 60K stores. Is there a faster approach to handling a massive dump like this? I should also add that there are quite a few indexes. These indexes are there for ordering and fulltext search once the products have been activated on the site. I cannot create the table without the indexes and add them later since there is already preexisting data that is using the indexes. I know this is most likely what is slowing it down but I am at a total loss what to do next.
    Last edited by RobNHood; July 31st, 2013 at 11:43 PM.

  2. #2
    VictorN's Avatar
    VictorN is offline Super Moderator Power Poster
    Join Date
    Jan 2003
    Location
    Hanover Germany
    Posts
    20,396

    Re: MySQL Connector Data Dump

    Victor Nijegorodov

  3. #3
    Join Date
    Jul 2013
    Location
    USA
    Posts
    30

    Re: MySQL Connector Data Dump

    Each row contains 35 fields with roughly 9 indexes including a compound fulltext index. Since the data is being parsed from a CSV file it's being read in 1 line at a time. I have only been working with MySQL 5.6 for about a month now. I decided to change the FullText Tables to InnoDB it just seems it was much faster with MyISAM. Before this I was running it in a php script driven by JQuery but the server kept timing out. Even so it just seemed faster then. I don't have 4 days to take on 1 store every minute is costing me money now

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured