hakluke/hakrawler

This is stars and forks stats for /hakluke/hakrawler repository. As of 29 Apr, 2024 this repository has 3974 stars and 457 forks.

Hakrawler Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library. Example usages Single URL: echo https://google.com | hakrawler Multiple URLs: cat urls.txt | hakrawler Timeout for each line of stdin after 5 seconds: cat urls.txt | hakrawler -timeout 5 Send all requests through a proxy: cat urls.txt | hakrawler -proxy http://localhost:8080 Include subdomains: echo https://google.com | hakrawler -subs Note:...