Skip to main content
Version: 1.11

HttpCrawler

Specific version of generic AbstractHttpCrawler.

It uses a dummy parser that simply returns the HTTP response body as-is. Use this only if you know what you are doing. In most cases, using an HTML parser would be more beneficial. For such scenarios, consider using BeautifulSoupCrawler, ParselCrawler, or writing your own subclass of AbstractHttpCrawler.

Usage

from crawlee.crawlers import HttpCrawler, HttpCrawlingContext

crawler = HttpCrawler()

# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: HttpCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')

# Extract data from the page.
data = {
'url': context.request.url,
'response': (await context.http_response.read()).decode()[:100],
}

# Push the extracted data to the default dataset.
await context.push_data(data)

await crawler.run(['https://crawlee.dev/'])

Hierarchy

Index