Search Results for "php spider"

...The spider is a CLI tool and must be added as a cronjob. In a web based backend you can control all data and analyze your data. You can handle multiple websites in the same backend. PHP 7 or 8 + PDO (Mysql/ Sqlite)

...Tasks are scheduled by the task scheduler module in the master node, and received by the task handler module in worker nodes, which executes these tasks in task runners. Task runners are actually processes running spider or crawler programs, and can also send data through gRPC (integrated in SDK) to other data sources, e.g. MongoDB.

OpenWebSpider is an Open Source multi-threaded Web Spider (robot, crawler) and search engine with a lot of interesting features!

This is fuskit A Fusker clone it's a working clone but it needs to optimized for todays technology so please submit your optimized versions so we can make fuskit better

...Walk.exe iterates through the local files and folders of your target web application (for example a local copy of Wordpress) and generates requests via your favourite proxy (for example burp suite) against a given target url. The remote agent can be used to identify target files and folders on a live system via a PHP script on the target server(ASP/JSP coming soon). The advantage of this tool is that it allows access to files and folders (for example include or plugin folders) which are not usually seen via a spider or crawler to be security tested with traditional tools. The Py version is on its way soon.

Sitemap Creator is a PHP class which creates XML sitemaps files compatible with the standard sitemaps.org protocol supported by Google and Bing. Features Uses PHPCrawl class to crawl/spider the website and creates URLs set while all PHPCrawl methods and options are accessible through class. Ability to calculate Priority, Frequency and Last-Modified date with variety of options.