This documentation contains everything you need to know about Scrapy.
Having trouble? We’d like to help!
- Try the FAQ – it’s got answers to some common questions.
- Looking for specific information? Try the Index or Module Index.
- Search for information in the archives of the scrapy-users mailing list, or post a question.
- Ask a question in the #scrapy IRC channel.
- Report bugs with Scrapy in our ticket tracker.
- Command line tool
- Learn about the command-line tool used to manage your Scrapy project.
- Define the data you want to scrape.
- Write the rules to crawl your websites.
- XPath Selectors
- Extract the data from web pages.
- Scrapy shell
- Test your extraction code in an interactive environment.
- Item Loaders
- Populate your items with the extracted data.
- Item Pipeline
- Post-process and store your scraped data.
- Feed exports
- Output your scraped data using different formats and storages.
- Understand the simple logging facility provided by Scrapy.
- Stats Collection
- Collect statistics about your scraping crawler.
- Sending e-mail
- Send email notifications when certain events occur.
- Telnet Console
- Inspect a running crawler using a built-in Python console.
- Web Service
- Monitor and control a crawler using a web service.
Solving specific problems¶
- Frequently Asked Questions
- Get answers to most frequently asked questions.
- Using Firefox for scraping
- Learn how to scrape with Firefox and some useful add-ons.
- Using Firebug for scraping
- Learn how to scrape efficiently using Firebug.
- Debugging memory leaks
- Learn how to find and get rid of memory leaks in your crawler.
- Downloading Item Images
- Download static images associated with your scraped items.
- Ubuntu packages
- Install latest Scrapy packages easily on Ubuntu
- Scrapy Service (scrapyd)
- Deploying your Scrapy project in production.
- Command line tool
- Learn about the command-line tool and see all available commands.
- Requests and Responses
- Understand the classes used to represent HTTP requests and responses.
- Learn how to configure Scrapy and see all available settings.
- See all available signals and how to work with them.
- See all available exceptions and their meaning.
- Item Exporters
- Quickly export your scraped items to a file (XML, CSV, etc).