CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 3 of 3
  1. #1
    Join Date
    Jun 2020
    Posts
    1

    Question PHP to Scrape large amount of data but database question....

    Extracting a huge amount of data from the likes of rightmove.co.uk and zoopla.co.uk. We will need to use this information to show changes on a membership website as well as inside each of the websites (Rightmove, zoopla etc)in questions (shown in attached screenshots)

    What would the best way to store this information so it can retrieved quickly and easily please?

    I do not think a csv file will do the trick as for example, if we start tracking the number of images, having the URLS will not help and we will need to store the actual images too in our database - right?

    Any guidance and tips you can give would be deeply appreciated



    Last edited by 2kaud; June 7th, 2020 at 08:07 AM.

  2. #2
    Arjay's Avatar
    Arjay is offline Moderator / EX MS MVP Power Poster
    Join Date
    Aug 2004
    Posts
    13,490

    Re: PHP to Scrape large amount of data but database question....

    Do these sites offer an api to help you get the data directly?

    I ask because scraping is very fragile - one tiny ui change by the site developers and your scraping can break. It can turn into a maintenance nightmare for sites that get updated frequently.

  3. #3
    Join Date
    Nov 2020
    Location
    Earth
    Posts
    4

    Re: PHP to Scrape large amount of data but database question....

    The easiest way to scrap data from webpages is to use regular expressions. For new users best&quick way - use some program services like a https://mydataprovider.com/solutions/web-scraping/.
    Last edited by Baileybob; December 2nd, 2020 at 05:25 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured