Installation guide
Supported Python versions
Scrapy requires Python 3.11+.
Installing Scrapy
Install Scrapy and its dependencies from PyPI with:
pip install scrapy
We strongly recommend that you install Scrapy in a dedicated virtual environment, to avoid conflicting with your system packages.
Alternatively:
From a uv project, e.g. one created with
uv init.uvmanages the virtual environment for you:uv add scrapy
From the conda-forge channel, with
conda:conda install -c conda-forge scrapy
If the installation fails while building one of the dependencies, see Installation fails while building a dependency.
Things that are good to know
Scrapy is written in pure Python and depends on a few key Python packages (among others):
lxml, an efficient XML and HTML parser
parsel, an HTML/XML data extraction library written on top of lxml
w3lib, a multi-purpose helper for dealing with URLs and web page encodings
twisted, an asynchronous networking framework
cryptography and pyOpenSSL, to deal with various network-level security needs
Some of these packages include compiled code. They provide binary wheels for common platforms, so that installing them requires no compiler; see Installation fails while building a dependency if no wheel matches your platform.
Using a virtual environment (recommended)
TL;DR: We recommend installing Scrapy inside a virtual environment on all platforms.
Python packages can be installed either globally (a.k.a. system-wide), or in user-space. We do not recommend installing Scrapy system-wide.
Instead, we recommend that you install Scrapy within a so-called “virtual
environment” (venv). Virtual environments allow you to avoid conflicts
with already-installed Python system packages (which could break some of your
system tools and scripts), and still install packages normally with pip
(without sudo or similar tools).
See Virtual Environments and Packages on how to create your virtual environment.
Once you have created a virtual environment, you can install Scrapy inside it with pip,
just like any other Python package.
Optional extras
Scrapy provides optional extras that install additional dependencies to enable specific features. To install Scrapy with one or more extras, list them in square brackets:
pip install scrapy[s3,images]
The following extras are available:
Extra |
Provides |
|---|---|
|
|
|
Reading the system clipboard with |
|
|
|
|
|
HttpxDownloadHandler, including its HTTP/2 and SOCKS proxy support |
|
|
|
|
|
Memory usage extension on Windows |
|
|
|
|
|
Amazon S3 storage for feed exports, media pipelines, and S3 downloads |
|
SOCKS proxy support in AioftpDownloadHandler |
|
|
|
uvloop event loop |
Platform-specific installation notes
Windows
Install Scrapy as described above. Dependencies that include compiled code provide wheels for 64-bit x86 Windows, so no compiler is needed.
On 32-bit or ARM64 Windows some of those wheels are missing, see
Installation fails while building a dependency. An additional option there is WSL, which lets you
follow the Linux instructions instead. Note that your
code then runs on Linux, so Windows paths do not work in settings like
FEEDS.
Linux
Install Scrapy as described above. Dependencies
that include compiled code provide manylinux and musllinux wheels, so
no compiler is needed.
Don’t use the Scrapy package provided by your distribution, e.g.
python-scrapy on Debian and Ubuntu, they are typically too old and slow to
catch up with the latest Scrapy release.
macOS
Install Scrapy as described above. Dependencies that include compiled code provide wheels for both Intel and Apple silicon, so no compiler is needed.
We recommend against using the Python interpreter that comes with macOS. Install a separate one instead, e.g. with homebrew:
brew install python
Troubleshooting
Installation fails while building a dependency
When pip finds no wheel matching your Python version and platform, it
builds the dependency from its source distribution instead, which requires a
build toolchain and the libraries that the dependency wraps. To check whether
that is what fails for you, ask pip for wheels only:
pip install --only-binary :all: scrapy
With uv, use uv add --no-build scrapy instead.
If that fails, no wheel is available, and you have the following options, easiest first:
Use an older Python version. Wheels for a new Python version may take weeks to become available after its release.
Use
conda, e.g. through Miniforge, and install Scrapy from the conda-forge channel.Build the dependency from source, following its own installation instructions, e.g. lxml installation or cryptography installation. Beyond a C compiler, this may require a Rust compiler, development files of libraries like libxml2 or OpenSSL, or both.
AttributeError: ‘module’ object has no attribute ‘OP_NO_TLSv1_1’
After you install or upgrade Scrapy, Twisted or pyOpenSSL, you may get an exception with the following traceback:
[…]
File "[…]/site-packages/twisted/protocols/tls.py", line 63, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in <module>
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
The reason you get this exception is that your system or virtual environment has a version of pyOpenSSL that your version of Twisted does not support.
To install a version of pyOpenSSL that your version of Twisted supports,
reinstall Twisted with the tls extra option:
pip install twisted[tls]
For details, see Issue #2473.