« Back to Index

Website Crawler

View original Gist on GitHub

Website Crawler.sh

wget --mirror \
     --convert-links \
     --adjust-extension \
     --page-requisites \
     --header="Host: www.example.com" \
     --no-parent https://beepboop.cloudfront.net