Changelog
All notable changes to this project will be documented in this file.
1.11.2 (2026-11-10)โ
๐ Bug Fixesโ
- Stop truncating fractional robots.txt crawl-delay to an integer (#2392) (fa386bb) by @RaphaelFakhri, closes #2391
- Fit the default
desired_concurrencywithinmin_concurrencyandmax_concurrency(#2397) (b71d5af) by @PrashanthLande - Pass the crawler state to the rendering type detection run of
AdaptivePlaywrightCrawler(#2417) (4d3f989) by @Mantisus - Accept
Noneforproxy_configurationinBasicCrawlerOptions(#2416) (16858fd) by @vdusek - file-system: Stop KVS pagination from skipping or repeating keys (#2431) (7d271c0) by @peterolkhov
1.11.1 (2026-11-05)โ
๐ Bug Fixesโ
- Ignore a UTF-8 byte order mark in robots.txt (#2351) (3e6ee66) by @RaphaelFakhri, closes #2350
- deps: Drop the upper bound on sqlalchemy (#2356) (cadd810) by @vdusek
- sitemap: Join element text once and reject entity declarations (#2369) (242be00) by @vdusek
- browsers: Remove the persistent browser's temp directory after the browser exits (#2381) (06d331b) by @vdusek
โก Performanceโ
1.11.0 (2026-10-15)โ
๐ Featuresโ
- storages: Add direct async iteration on
DatasetandKeyValueStore(#2267) (e02e71d) by @Pijukatel, closes #1745 - crawlers: Add
extract_form_requestshelper for HTTP crawlers with HTML parsers (#2257) (62386fd) by @Mantisus - Add support for Python 3.15 (#2294) (680c7ef) by @vdusek
๐ Bug Fixesโ
- deps: Replace
jaro-winklerwithRapidFuzzin theadaptive-crawlerextra (#2282) (81ab674) by @Mantisus
โก Performanceโ
- Replace
tldextractwith a built-in Public Suffix List lookup (#2283) (a14f745) by @Mantisus, closes #2056
1.10.4 (2026-10-06)โ
๐ Bug Fixesโ
- crawlers: Parse HTML responses starting with an XML declaration as HTML in
ParselCrawler(#2262) (df1178c) by @Mantisus - crawlers: Resolve relative and invalid base URLs when extracting links (#2263) (c69f75f) by @Mantisus
- crawlers: Detect JSON served as
text/htmland handle empty responses inParselCrawler(#2266) (3a58843) by @Mantisus - Make
crawlee[all]installable withpip<26(#2270) (87561ee) by @vdusek, closes #2269 - Pause the crawl on Ctrl+C on Windows instead of crashing (#2277) (dbec1f5) by @Mantisus, closes #228
1.10.3 (2026-09-29)โ
๐ Bug Fixesโ
- Make
JsonSerializabletype alias read-only (#2249) (dd7358f) by @Pijukatel - Use container memory and CPU limits when running in a container (#2128) (40dafe5) by @Mantisus, closes #1535, #2095
- crawlers: Keep the crawler usable after a failed run (#2255) (3e73166) by @vdusek
- browsers: Retry the temp directory removal until the browser releases its files (#2239) (8f41522) by @vdusek
- throttling-manager: Defer throttled 429s without spending retries and give up on persistently rate-limited domains (#2250) (6b7db66) by @Mantisus, closes #2143
- crawlers: Decode non-UTF-8 pages correctly in
ParselCrawlerandBeautifulSoupCrawler(#2260) (209e8a1) by @Mantisus
1.10.2 (2026-09-22)โ
๐ Bug Fixesโ
- Make
extra_nameoftry_importoptional and the install hint reusable (#2247) (626f334) by @Pijukatel
1.10.1 (2026-09-16)โ
๐ Bug Fixesโ
- Name the extra to install in optional-dependency import errors (#2208) (d57c6a0) by @vdusek
- Make
enqueue_linksandexport_tooverloads consistent with their implementations (#2225) (47e4630) by @vdusek - throttling-manager: Apply per-domain backoff when
retry_on_blockedis disabled (#2158) (9c23101) by @Mantisus, closes #2126 - autoscaling: Keep
min_concurrencytasks running while the system is overloaded (#2235) (cbaefb4) by @Mantisus - Propagate autoscaled pool scheduling errors (#2229) (cb7822e) by @Kuang-xianxin
1.10.0 (2026-08-31)โ
๐ Featuresโ
- httpx: Build
HttpxHttpClientonhttpx2instead ofhttpx(#2182) (6deabad) by @Mantisus - Support timeouts in the
send_requestcontext helper (#2174) (c2013a0) by @Battleplus, closes #2138
๐ Bug Fixesโ
- http-clients: Reject a non-positive timeout instead of silently ignoring it (#2187) (b4c59c6) by @vdusek
- templates: Decouple the Camoufox Playwright version from other images (#2189) (da16f2c) by @Pijukatel, closes #2175
- throttling-manager: Correct the backoff for concurrent 429s and zero Retry-After (#2159) (142ec58) by @Mantisus, closes #2125
1.9.3 (2026-08-24)โ
๐ Bug Fixesโ
- sitemaps: Allow
Sitemap.from_xml_stringto opt into enqueue host filtering (#2162) (a3c20c1) by @aryansk, closes #2118 - playwright: Scope custom request headers to the navigation request (#2173) (6f0eff4) by @vdusek
- throttling-manager: Normalize configured domains the same way as crawled hostnames (#2157) (eb91fa7) by @Mantisus, closes #2144
- throttling-manager: Stop holding a concurrency slot while a domain is throttled (#2176) (5c1fa09) by @Mantisus, closes #2127
- throttling-manager: Open the per-domain sub-queues on first use instead of on insert (#2177) (991502f) by @Mantisus, closes #2142
- httpx: Scope cookies to the session and origin on every request (#2179) (fbecef1) by @Mantisus
1.9.2 (2026-08-17)โ
๐ Bug Fixesโ
- sitemap: Guard the unavailable
ExpatParser.flushon olderCPythonpatch releases (#2141) (7f75295) by @Mantisus, closes #2119 - dataset: Preserve
push_datacompatibility with the released Apify SDK (#2152) (091410f) by @vdusek - stagehand: Cap the
stagehanddependency below the breaking 4.x API (#2153) (263fff4) by @vdusek - sitemaps: Validate HTTP status when fetching sitemaps (#2123) (c5c1072) by @anxkhn
- browsers: Do not close a browser as inactive while a page opening is in flight (#2156) (3485535) by @vdusek
โก Performanceโ
1.9.1 (2026-08-06)โ
๐ Bug Fixesโ
- fingerprints: Enforce the user agent allow list so screen constraints are respected (#2110) (017dd1c) by @vdusek, closes #2108
- Honor
error_handlerreplacement requests onSessionErrorand preserve forefront on retries (#2105) (c731597) by @Ayush7614 - curl-impersonate: Treat a refused CONNECT tunnel as a proxy error (#2109) (54b7602) by @vdusek, closes #2111
- events: Prevent deadlock when closing or waiting for listeners from within a listener (#2088) (92ab97a) by @vdusek
- events: Correct the listener invocation, removal, and context release (#2100) (ccc2b89) by @vdusek