Inspect HTTP request | ScrapingClub

This project was built to help people and I did not earn money from my work. But you can still support my work

Some website would check HTTP headers such as Referer to block some abnormal request.

Try to extract the product detail such as title, desc and price.

Exercise List

Web scraping using XPath or CSS expression

Load JSON string and extract data

Not only crawl products but also handle pagination

Inspect Ajax requests and mimic them

Learn to inspect the fields of HTTP request

Learn to scrape infinite scrolling pages

Make your spider can work with the cookie

Scrape data behind login form

Learn to scrape data behind a captcha

Learn how to analyze minimized or compressed javascript