Quick start
This page gets you from installation to a running crawler in a few minutes. To build a complete scraper step by step, follow the Introduction.
Choose your crawler
Crawlee's main crawler classes are BeautifulSoupCrawler, ParselCrawler, and PlaywrightCrawler. All crawlers share the same interface, so you can switch between them without rewriting your handlers.
Crawlee requires Python 3.10 or higher.
BeautifulSoupCrawler
The BeautifulSoupCrawler is a plain HTTP crawler that parses HTML using the well-known BeautifulSoup library. It fetches pages with an HTTP client that mimics a browser. It's fast and light, but it can't render JavaScript.
ParselCrawler
The ParselCrawler is similar to the BeautifulSoupCrawler but uses the Parsel library for HTML parsing. Parsel extracts data from HTML with CSS selectors and XPath, and it's the selector library Scrapy uses. Like the BeautifulSoupCrawler, the ParselCrawler can't render JavaScript.
PlaywrightCrawler
The PlaywrightCrawler uses a headless browser controlled by the Playwright library. It can drive Chromium, Firefox, and WebKit. Choose it when the target pages render their content with JavaScript.
Installation
Crawlee is available as the crawlee package on PyPI. The package includes the core functionality, and additional features are available as optional extras to keep dependencies small.
To install Crawlee with all extras using pip, run:
python -m pip install 'crawlee[all]'
Verify that Crawlee is successfully installed:
python -c 'import crawlee; print(crawlee.__version__)'
If you plan to use the PlaywrightCrawler, also install the Playwright browser binaries:
playwright install
For detailed installation instructions, see