Get a listing of all the urls on a website with this simple Linux command.

This command will return a huge listing of all the visitable url`s on the http://www.google.com.au website. Give this a shot on other websites and see how you go.

[email protected]:~$ wget --spider --force-html -r -l2 http://www.google.com.au 2>&1   | grep '^--' | awk '{ print $3 }'

Very useful if someone wants to know what is actually on a certain website. This might take a bit of time to return the listing, but this does work very well when spidering a website. Below is sample output.

[email protected]:~$ wget --spider --force-html -r -l2 http://www.securitronlinux.com/linux/ 2>&1   | grep '^--' | awk '{ print $3 }'
http://www.securitronlinux.com/linux/
http://www.securitronlinux.com/linux/
http://www.securitronlinux.com/robots.txt
http://www.securitronlinux.com/linux/style.css
http://www.securitronlinux.com/linux/favicon.ico
http://www.securitronlinux.com/linux/maps/back.png
http://www.securitronlinux.com/index.php
http://www.securitronlinux.com/
http://www.securitronlinux.com/
http://www.securitronlinux.com/linux/caring_computer.php
http://www.securitronlinux.com/linux/caring_computer.php
http://www.securitronlinux.com/linux/fedora.php
http://www.securitronlinux.com/linux/fedora.php
http://www.securitronlinux.com/linux/security.php
http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/
http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/
http://www.securitronlinux.com/linux/susedvd.php
http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/
http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/
http://www.securitronlinux.com/linux/doom_files.php
http://www.securitronlinux.com/linux/doom_files.php
http://www.securitronlinux.com/linux/doom_wadfiles.php
http://www.securitronlinux.com/linux/doom_wadfiles.php
http://www.securitronlinux.com/linux/console_codes.php
http://www.securitronlinux.com/linux/console_codes.php
http://www.securitronlinux.com/linux/misc_codes.php
http://www.securitronlinux.com/linux/misc_codes.php
http://www.securitronlinux.com/linux/psx_doom.php
http://www.securitronlinux.com/linux/psx_doom.php
http://www.securitronlinux.com/linux/zdoom_acs.php
http://www.securitronlinux.com/linux/windows_3.0.php
http://www.securitronlinux.com/linux/windows_3.0.php
http://www.securitronlinux.com/linux/ubuntu-karmic.php
http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/
http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/
http://www.securitronlinux.com/linux/windows_7.php
http://www.securitronlinux.com/linux/windows_7.php
http://www.securitronlinux.com/linux/johnjohnrace.swf
http://www.securitronlinux.com/linux/doomshots/
http://www.securitronlinux.com/linux/doomshots/
http://www.securitronlinux.com/xmlrpc.php
http://www.securitronlinux.com/wp-content/themes/securitron/style.css
http://www.securitronlinux.com/page/2/
http://www.securitronlinux.com/page/2/
http://www.securitronlinux.com/feed/
http://www.securitronlinux.com/comments/feed/
http://www.securitronlinux.com/wp-includes/js/jquery/jquery.js?ver=1.12.3
http://www.securitronlinux.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.0
http://www.securitronlinux.com/wp-json/
http://www.securitronlinux.com/xmlrpc.php?rsd

No comments have been made. Use this form to start the conversation :)

Leave a Reply