Frequently Asked Questions¶
How does Scrapy compare to BeautifulSoul or lxml?¶
Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code.
Does Scrapy work with Python 3.0?¶
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment Scrapy works with Python 2.5 or 2.6.
Did Scrapy “steal” X from Django?¶
Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy.
We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We’d be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Does Scrapy work with HTTP proxies?¶
Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP
Proxy downloader middleware. See
Scrapy crashes with: ImportError: No module named win32api¶
How can I simulate a user login in my spider?¶
Can I crawl in depth-first order instead of breadth-first order?¶
Yes, there’s a setting for that:
Why does Scrapy download pages in English instead of my native language?¶
Where can I find some example code using Scrapy?¶
Scrapy comes with a built-in, fully functional project to scrape the Google
Directory. You can find it in the
examples/googledir directory of the
Also, there is a public repository of spiders called Community Spiders.
Finally, you can find some example code for performing not-so-trivial tasks in the Scrapy Recipes page.
Can I run a spider without creating a project?¶
Yes. You can use the
runspider command. For example, if you have a spider
written in a
my_spider.py file you can run it with:
scrapy-ctl.py runspider my_spider.py
I get “Filtered offsite request” messages. How can I fix them?¶
Those messages (logged with
DEBUG level) don’t necessarily mean there is a
problem, so you may not need to fix them.
Those message are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.
For more info see:
How can I make Scrapy consume less memory?¶
There’s a whole documentation section about this subject, please see: Debugging memory leaks.
Also, Python has a builtin memory leak issue which is described in Leaks without leaks.
What is the recommended way to deploy a Scrapy crawler in production?¶
If you’re running Ubuntu 9.10 or above, you can just use the official APT
repos for installing Scrapy in your servers. The Debian package automatically
deploys the service in the right place and leaves it ready to be started,
running as the
scrapy user. It also provides an upstart script for
controlling the service.
For a practical example see this guide on how to deploy a Scrapy crawler on Amazon EC2.