Extensions¶
The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy.
Extensions are just regular classes that are instantiated at Scrapy startup, when extensions are initialized.
Extension settings¶
Extensions use the Scrapy settings to manage their settings, just like any other Scrapy code.
It is customary for extensions to prefix their settings with their own name, to avoid collision with existing (and future) extensions. For example, a hypothetic extension to handle Google Sitemaps would use settings like GOOGLESITEMAP_ENABLED, GOOGLESITEMAP_DEPTH, and so on.
Loading & activating extensions¶
Extensions are loaded and activated at startup by instantiating a single
instance of the extension class. Therefore, all the extension initialization
code must be performed in the class constructor (__init__
method).
To make an extension available, add it to the EXTENSIONS
setting in
your Scrapy settings. In EXTENSIONS
, each extension is represented
by a string: the full Python path to the extension’s class name. For example:
EXTENSIONS = {
'scrapy.extensions.corestats.CoreStats': 500,
'scrapy.telnet.TelnetConsole': 500,
}
As you can see, the EXTENSIONS
setting is a dict where the keys are
the extension paths, and their values are the orders, which define the
extension loading order. Extensions orders are not as important as middleware
orders though, and they are typically irrelevant, ie. it doesn’t matter in
which order the extensions are loaded because they don’t depend on each other
[1].
However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded.
[1] This is is why the EXTENSIONS_BASE
setting in Scrapy (which
contains all built-in extensions enabled by default) defines all the extensions
with the same order (500
).
Available, enabled and disabled extensions¶
Not all available extensions will be enabled. Some of them usually depend on a
particular setting. For example, the HTTP Cache extension is available by default
but disabled unless the HTTPCACHE_ENABLED
setting is set.
Disabling an extension¶
In order to disable an extension that comes enabled by default (ie. those
included in the EXTENSIONS_BASE
setting) you must set its order to
None
. For example:
EXTENSIONS = {
'scrapy.extensions.corestats.CoreStats': None,
}
Writing your own extension¶
Each extension is a Python class. The main entry point for a Scrapy extension
(this also includes middlewares and pipelines) is the from_crawler
class method which receives a Crawler
instance. Through the Crawler object
you can access settings, signals, stats, and also control the crawling behaviour.
Typically, extensions connect to signals and perform tasks triggered by them.
Finally, if the from_crawler
method raises the
NotConfigured
exception, the extension will be
disabled. Otherwise, the extension will be enabled.
Sample extension¶
Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time:
- a spider is opened
- a spider is closed
- a specific number of items are scraped
The extension will be enabled through the MYEXT_ENABLED
setting and the
number of items will be specified through the MYEXT_ITEMCOUNT
setting.
Here is the code of such extension:
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured
logger = logging.getLogger(__name__)
class SpiderOpenCloseLogging(object):
def __init__(self, item_count):
self.item_count = item_count
self.items_scraped = 0
@classmethod
def from_crawler(cls, crawler):
# first check if the extension should be enabled and raise
# NotConfigured otherwise
if not crawler.settings.getbool('MYEXT_ENABLED'):
raise NotConfigured
# get the number of items from settings
item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000)
# instantiate the extension object
ext = cls(item_count)
# connect the extension object to signals
crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)
# return the extension object
return ext
def spider_opened(self, spider):
logger.info("opened spider %s", spider.name)
def spider_closed(self, spider):
logger.info("closed spider %s", spider.name)
def item_scraped(self, item, spider):
self.items_scraped += 1
if self.items_scraped % self.item_count == 0:
logger.info("scraped %d items", self.items_scraped)
Built-in extensions reference¶
General purpose extensions¶
Log Stats extension¶
-
class
scrapy.extensions.logstats.
LogStats
¶
Log basic stats like crawled pages and scraped items.
Core Stats extension¶
-
class
scrapy.extensions.corestats.
CoreStats
¶
Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection).
Telnet console extension¶
-
class
scrapy.telnet.
TelnetConsole
¶
Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging.
The telnet console must be enabled by the TELNETCONSOLE_ENABLED
setting, and the server will listen in the port specified in
TELNETCONSOLE_PORT
.
Memory usage extension¶
-
class
scrapy.extensions.memusage.
MemoryUsage
¶
Note
This extension does not work in Windows.
Monitors the memory used by the Scrapy process that runs the spider and:
- sends a notification e-mail when it exceeds a certain value
- closes the spider when it exceeds a certain value
The notification e-mails can be triggered when a certain warning value is
reached (MEMUSAGE_WARNING_MB
) and when the maximum value is reached
(MEMUSAGE_LIMIT_MB
) which will also cause the spider to be closed
and the Scrapy process to be terminated.
This extension is enabled by the MEMUSAGE_ENABLED
setting and
can be configured with the following settings:
Memory debugger extension¶
-
class
scrapy.extensions.memdebug.
MemoryDebugger
¶
An extension for debugging memory usage. It collects information about:
- objects uncollected by the Python garbage collector
- objects left alive that shouldn’t. For more info, see Debugging memory leaks with trackref
To enable this extension, turn on the MEMDEBUG_ENABLED
setting. The
info will be stored in the stats.
Close spider extension¶
-
class
scrapy.extensions.closespider.
CloseSpider
¶
Closes a spider automatically when some conditions are met, using a specific closing reason for each condition.
The conditions for closing a spider can be configured through the following settings:
CLOSESPIDER_TIMEOUT¶
Default: 0
An integer which specifies a number of seconds. If the spider remains open for
more than that number of second, it will be automatically closed with the
reason closespider_timeout
. If zero (or non set), spiders won’t be closed by
timeout.
CLOSESPIDER_ITEMCOUNT¶
Default: 0
An integer which specifies a number of items. If the spider scrapes more than
that amount if items and those items are passed by the item pipeline, the
spider will be closed with the reason closespider_itemcount
. If zero (or
non set), spiders won’t be closed by number of passed items.
CLOSESPIDER_PAGECOUNT¶
New in version 0.11.
Default: 0
An integer which specifies the maximum number of responses to crawl. If the spider
crawls more than that, the spider will be closed with the reason
closespider_pagecount
. If zero (or non set), spiders won’t be closed by
number of crawled responses.
CLOSESPIDER_ERRORCOUNT¶
New in version 0.11.
Default: 0
An integer which specifies the maximum number of errors to receive before
closing the spider. If the spider generates more than that number of errors,
it will be closed with the reason closespider_errorcount
. If zero (or non
set), spiders won’t be closed by number of errors.
StatsMailer extension¶
-
class
scrapy.extensions.statsmailer.
StatsMailer
¶
This simple extension can be used to send a notification e-mail every time a
domain has finished scraping, including the Scrapy stats collected. The email
will be sent to all recipients specified in the STATSMAILER_RCPTS
setting.
Debugging extensions¶
Stack trace dump extension¶
-
class
scrapy.extensions.debug.
StackTraceDump
¶
Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following:
- engine status (using
scrapy.utils.engine.get_engine_status()
) - live references (see Debugging memory leaks with trackref)
- stack trace of all threads
After the stack trace and engine status is dumped, the Scrapy process continues running normally.
This extension only works on POSIX-compliant platforms (ie. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows.
There are at least two ways to send Scrapy the SIGQUIT signal:
By pressing Ctrl-while a Scrapy process is running (Linux only?)
By running this command (assuming
<pid>
is the process id of the Scrapy process):kill -QUIT <pid>
Debugger extension¶
-
class
scrapy.extensions.debug.
Debugger
¶
Invokes a Python debugger inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally.
For more info see Debugging in Python.
This extension only works on POSIX-compliant platforms (ie. not Windows).