Skip to main content

Build reliable web scrapers. Fast.

Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you.

Run on
import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, enqueueLinks, pushData, log }) {
const title = await page.title();
log.info(`Title of ${request.loadedUrl} is '${title}'`);

await pushData({ title, url: request.loadedUrl });
await enqueueLinks();
},

// Uncomment this option to see the browser window.
// headless: false,
});

await crawler.run(['https://crawlee.dev']);
Or start with a template from our CLI
$npx crawlee create my-crawler
Built with 🤍 by Apify. Forever free and open-source.

What are the benefits?

Unblock websites by default

Crawlee crawls stealthily with zero configuration, but you can customize its behavior to overcome any protection. Real-world fingerprints included.

Learn more
{
fingerprintOptions: {
fingerprintGeneratorOptions: {
browsers: ['chrome', 'firefox'],
devices: ['mobile'],
locales: ['en-US'],
},
},
},

Work with your favorite tools

Crawlee integrates BeautifulSoup, Cheerio, Puppeteer, Playwright, and other popular open-source tools. No need to learn new syntax.

Learn more
Work with your favorite toolsWork with your favorite tools

One API for headless and HTTP

Switch between HTTP and headless without big rewrites thanks to a shared API. Or even let Adaptive crawler decide if JS rendering is needed.

Learn more
const crawler = new AdaptivePlaywrightCrawler({
renderingTypeDetectionRatio: 0.1,
async requestHandler({ querySelector, enqueueLinks }) {
// The crawler detects if JS rendering is needed
// to extract this data. If not, it will use HTTP
// for follow-up requests to save time and costs.
const $prices = await querySelector('span.price')
await enqueueLinks();
},
});

What else is in Crawlee?