Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you.
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, enqueueLinks, pushData, log }) {
const title = await page.title();
log.info(`Title of ${request.loadedUrl} is '${title}'`);
await pushData({ title, url: request.loadedUrl });
await enqueueLinks();
},
// Uncomment this option to see the browser window.
// headless: false,
});
await crawler.run(['https://crawlee.dev']);
$npx crawlee create my-crawlerCrawlee crawls stealthily with zero configuration, but you can customize its behavior to overcome any protection. Real-world fingerprints included.
Learn more{
fingerprintOptions: {
fingerprintGeneratorOptions: {
browsers: ['chrome', 'firefox'],
devices: ['mobile'],
locales: ['en-US'],
},
},
},
Crawlee integrates BeautifulSoup, Cheerio, Puppeteer, Playwright, and other popular open-source tools. No need to learn new syntax.
Learn more

Switch between HTTP and headless without big rewrites thanks to a shared API. Or even let Adaptive crawler decide if JS rendering is needed.
Learn moreconst crawler = new AdaptivePlaywrightCrawler({
renderingTypeDetectionRatio: 0.1,
async requestHandler({ querySelector, enqueueLinks }) {
// The crawler detects if JS rendering is needed
// to extract this data. If not, it will use HTTP
// for follow-up requests to save time and costs.
const $prices = await querySelector('span.price')
await enqueueLinks();
},
});