Installation guide

Supported Python versions

Scrapy requires Python 3.11+.

Installing Scrapy

Install Scrapy and its dependencies from PyPI with:

pip install scrapy

We strongly recommend that you install Scrapy in a dedicated virtual environment, to avoid conflicting with your system packages.

Alternatively:

  • From a uv project, e.g. one created with uv init. uv manages the virtual environment for you:

    uv add scrapy
    
  • From the conda-forge channel, with conda:

    conda install -c conda-forge scrapy
    

If the installation fails while building one of the dependencies, see Installation fails while building a dependency.

Things that are good to know

Scrapy is written in pure Python and depends on a few key Python packages (among others):

  • lxml, an efficient XML and HTML parser

  • parsel, an HTML/XML data extraction library written on top of lxml

  • w3lib, a multi-purpose helper for dealing with URLs and web page encodings

  • twisted, an asynchronous networking framework

  • cryptography and pyOpenSSL, to deal with various network-level security needs

Some of these packages include compiled code. They provide binary wheels for common platforms, so that installing them requires no compiler; see Installation fails while building a dependency if no wheel matches your platform.

Optional extras

Scrapy provides optional extras that install additional dependencies to enable specific features. To install Scrapy with one or more extras, list them in square brackets:

pip install scrapy[s3,images]

The following extras are available:

Extra

Provides

bpython

bpython shell

clipboard

Reading the system clipboard with genrequest

color

LOG_COLOR

gcs

Google Cloud Storage for feed exports and media pipelines

httpx

HttpxDownloadHandler, including its HTTP/2 and SOCKS proxy support

images

Images pipeline

ipython

IPython shell

memusage

Memory usage extension on Windows

ptpython

ptpython shell

robotparser

Robotexclusionrulesparser robots.txt parsing

s3

Amazon S3 storage for feed exports, media pipelines, and S3 downloads

socks

SOCKS proxy support in AioftpDownloadHandler

twisted-http2

H2DownloadHandler

uvloop

uvloop event loop

Platform-specific installation notes

Windows

Install Scrapy as described above. Dependencies that include compiled code provide wheels for 64-bit x86 Windows, so no compiler is needed.

On 32-bit or ARM64 Windows some of those wheels are missing, see Installation fails while building a dependency. An additional option there is WSL, which lets you follow the Linux instructions instead. Note that your code then runs on Linux, so Windows paths do not work in settings like FEEDS.

Linux

Install Scrapy as described above. Dependencies that include compiled code provide manylinux and musllinux wheels, so no compiler is needed.

Don’t use the Scrapy package provided by your distribution, e.g. python-scrapy on Debian and Ubuntu, they are typically too old and slow to catch up with the latest Scrapy release.

macOS

Install Scrapy as described above. Dependencies that include compiled code provide wheels for both Intel and Apple silicon, so no compiler is needed.

We recommend against using the Python interpreter that comes with macOS. Install a separate one instead, e.g. with homebrew:

brew install python

Troubleshooting

Installation fails while building a dependency

When pip finds no wheel matching your Python version and platform, it builds the dependency from its source distribution instead, which requires a build toolchain and the libraries that the dependency wraps. To check whether that is what fails for you, ask pip for wheels only:

pip install --only-binary :all: scrapy

With uv, use uv add --no-build scrapy instead.

If that fails, no wheel is available, and you have the following options, easiest first:

  • Use an older Python version. Wheels for a new Python version may take weeks to become available after its release.

  • Use conda, e.g. through Miniforge, and install Scrapy from the conda-forge channel.

  • Build the dependency from source, following its own installation instructions, e.g. lxml installation or cryptography installation. Beyond a C compiler, this may require a Rust compiler, development files of libraries like libxml2 or OpenSSL, or both.

AttributeError: ‘module’ object has no attribute ‘OP_NO_TLSv1_1’

After you install or upgrade Scrapy, Twisted or pyOpenSSL, you may get an exception with the following traceback:

[…]
  File "[…]/site-packages/twisted/protocols/tls.py", line 63, in <module>
    from twisted.internet._sslverify import _setAcceptableProtocols
  File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in <module>
    TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'

The reason you get this exception is that your system or virtual environment has a version of pyOpenSSL that your version of Twisted does not support.

To install a version of pyOpenSSL that your version of Twisted supports, reinstall Twisted with the tls extra option:

pip install twisted[tls]

For details, see Issue #2473.