Right now there are mainly four typical methods for you to extract data.
Custom methodsshipped with some package such as
You can choose the one you like to extract the info, in this exercise, try to extract this product detail such as title, desc and price.
CONSCIOUS. Fitted, long-sleeved top in stretch jersey made from organic cotton with a round neckline. 92% cotton, 3% spandex, 3% rayon, 2% polyester.
Web scraping using XPath or CSS expression
Load JSON string and extract data
Not only crawl products but also handle pagination
Inspect Ajax requests and mimic them
Learn to inspect the fields of HTTP request
Learn to scrape infinite scrolling pages
Make your spider can work with the cookie
Scrape data behind login form
Learn to scrape data behind a captcha