Skip to main content
Version: Next

Quick start

This page gets you from installation to a running crawler in a few minutes. To build a complete scraper step by step, follow the Introduction.

Choose your crawler​

Crawlee's main crawler classes are BeautifulSoupCrawler, ParselCrawler, and PlaywrightCrawler. All crawlers share the same interface, so you can switch between them without rewriting your handlers.

Minimum Python version

Crawlee requires Python 3.10 or higher.

BeautifulSoupCrawler​

The BeautifulSoupCrawler is a plain HTTP crawler that parses HTML using the well-known BeautifulSoup library. It fetches pages with an HTTP client that mimics a browser. It's fast and light, but it can't render JavaScript.

ParselCrawler​

The ParselCrawler is similar to the BeautifulSoupCrawler but uses the Parsel library for HTML parsing. Parsel extracts data from HTML with CSS selectors and XPath, and it's the selector library Scrapy uses. Like the BeautifulSoupCrawler, the ParselCrawler can't render JavaScript.

PlaywrightCrawler​

The PlaywrightCrawler uses a headless browser controlled by the Playwright library. It can drive Chromium, Firefox, and WebKit. Choose it when the target pages render their content with JavaScript.

Installation​

Crawlee is available as the crawlee package on PyPI. The package includes the core functionality, and additional features are available as optional extras to keep dependencies small.

To install Crawlee with all extras using pip, run:

python -m pip install 'crawlee[all]'

Verify that Crawlee is successfully installed:

python -c 'import crawlee; print(crawlee.__version__)'

If you plan to use the PlaywrightCrawler, also install the Playwright browser binaries:

playwright install

For detailed installation instructions, see