Introducing SpiderSuite: An advance web security crawler
I have always being searching for a specialized crawling tool that I can use to visualize the entire crawled surface and analyze individual page contents to help me when performing web application security auditing. After a long search and not finding one that suited my needs I decided to embark on a long journey to build one from scratch.
After spending months developing it, I’m here to officially introduce to you SpiderSuite, an advance, free and cross-platform web security crawler.
Official page: https://SpiderSuite.github.io/
The tool: https://github.com/3nock/SpiderSuite
To be more correct, SpiderSuite isn’t a tool but a suite of tools for performing web security crawling and auditing.
SpiderSuite crawls an entire target site and saves the crawled pages in a specially designed SQLite database to make it easy querying and analyze individual crawled pages.
One of SpiderSuite’s design principle is giving the user as much control as possible when crawling hence the user can configure the crawler to their liking.
Features
- Crawling entire target site surfaces fast.
- Can crawl a target site by bruteforcing pages.
- Passive crawling for links using OSINT sources such as waybackmachine.
- Performing individual manual http(s) requests and saving the results.
- Extracting and displaying important contents from the crawled pages such as scripts, styles and comments embedded in the webpage.
- Graph visualization of the entire crawled surface or a branch of the crawled surface.
- Import crawled pages from other web security crawlers and tools such as burp suite, Fiddler, Katana and Caido.
- Comparing crawled pages and comparing entire crawl projects.
- Export crawled links to different output formats such as CSV, JSON, XML, HTML and Sitemap.xml.
Alternatives
There are many web security crawling tools out there that produce somewhat similar results to SpiderSuite but didn’t quite offer the functionalities I needed. Some of the best alternatives to SpiderSuite are;
- Burp Suite — Is an integrated platform for performing security testing of web applications. It does contain a web crawler/spider but it is tightly coupled with the rest of the Burp Suite application, resource intensive, and not designed for crawling large lists of domains.
- OWASP ZAP — is an open-source web application security scanner. It does contain a powerful web crawler but just like burp suite it is also tightly coupled with the rest of ZAP.
- Katana — Is a next-generation crawling and spidering framework. It is a very powerful crawling framework but as a command line tool it is difficult to interact with the crawl results and visualizing the individual crawled pages.
Realizing the potential and capabilities that these alternative tools offer I introduced to SpiderSuite the ability to import crawl results from these tools and many other more. Hence SpiderSuite can be used to complement these tools and the other way around.
Roadmap
SpiderSuite is not yet feature complete and there is alot more to build and fix. The main goal of SpiderSuite is to become the number one go to tool for security crawling in the cyber security field.
The ambitious goals set out for SpiderSuite include:
- Polishing exisiting features.
- Increasing speed and efficient. SpiderSuite is currently very fast but if it can be even more faster and efficient why not.
- Plugins system. Support for plugins that users and the community can build and share.
- An in-built passive and active vulnerability scanner.
- Integration with existing open source vulnerability scanners such as [nuclei template engine]().
- Dynamic crawler.
- Search engine optmization (SEO) crawling.
You can take part in this development journey of SpiderSuite. See Contribution Guide
For installation and usage details, see https://github.com/3nock/SpiderSuite repository or visit https://SpiderSuite.github.io/ official site.
