This project was built to help people and I did not earn money from my work. But you can still support my work
The key to scrape infinite scrolling pages is to use network panel in your browser to figure out the url of next page.
Sometimes you also need to take care of the http headers to make your code work.
In this exercise, try to crawl all product info.
Web scraping using XPath or CSS expression
Load JSON string and extract data
Not only crawl products but also handle pagination
Inspect Ajax requests and mimic them
Learn to inspect the fields of HTTP request
Scraping Infinite Scrolling Pages (Ajax)
Learn to scrape infinite scrolling pages
Make your spider can work with the cookie
Scrape data behind login form
Learn to scrape data behind a captcha