Open Source JavaScript Internet Software

Lightcrawler is a command-line tool designed to crawl a website and run automated audits on the discovered pages using Google Lighthouse. It works by starting from a given URL and recursively exploring linked pages to collect a set of pages that should be analyzed. Each discovered page is then evaluated using Lighthouse, which performs checks related to performance, accessibility, and web development best practices. This allows developers to audit multiple pages of a site automatically instead of manually running Lighthouse on each individual page. Lightcrawler supports configuration through a JSON configuration file, enabling users to customize how the crawler operates and which Lighthouse audits should be executed. Settings such as crawl depth and the number of concurrent browser instances can be configured to control how aggressively the crawler scans a site. It was created as a developer utility to help identify issues across an entire website more efficiently.

Methanol is a scriptable multi-purpose web crawling system with an extensible configuration system and speed-optimized architectural design. Methabot is the web crawler of Methanol.

Upon successful authentication, a cookie (named TOKEN) with the value of a signed JWT is set and the user redirected back to the originally requested path. Upon each request, Lambda@Edge checks the JWT for validity (signature, expiration date, audience and matching hosted domain) and will redirect the user to configured provider's login when their session has timed out. If your CloudFront distribution is pointed at a S3 bucket, configure origin access identity so S3 objects can be stored with private permissions. (Origin access identity requires the S3 ACL owner be the account owner. Use our s3-object-owner-monitor Lambda function if writing objects across multiple accounts.) Enable SSL/HTTPS on your CloudFront distribution; AWS Certificate Manager can be used to provision a no-cost certificate. Session duration is defined as the number of hours that the JWT is valid for. After session expiration, cloudfront-auth will redirect the user to the configured provider to re-authenticate.