Get a listing of all the urls on a website with this simple Linux command.

Posted: June 16, 2016. At: 8:14 AM. This was 1 year ago. Post ID: 9276
Page permalink.
WordPress uses cookies, or tiny pieces of information stored on your computer, to verify who you are. There are cookies for logged in users and for commenters. These cookies expire two weeks after they are set.

This command will return a huge listing of all the visitable url`s on the http://www.google.com.au website. Give this a shot on other websites and see how you go.

[email protected]:~$ wget --spider --force-html -r -l2 http://www.google.com.au 2>&1   | grep '^--' | awk '{ print $3 }'

Very useful if someone wants to know what is actually on a certain website. This might take a bit of time to return the listing, but this does work very well when spidering a website. Below is sample output.

[email protected]:~$ wget --spider --force-html -r -l2 http://www.securitronlinux.com/linux/ 2>&1   | grep '^--' | awk '{ print $3 }'
http://www.securitronlinux.com/linux/
http://www.securitronlinux.com/linux/
http://www.securitronlinux.com/robots.txt
http://www.securitronlinux.com/linux/style.css
http://www.securitronlinux.com/linux/favicon.ico
http://www.securitronlinux.com/linux/maps/back.png
http://www.securitronlinux.com/index.php
http://www.securitronlinux.com/
http://www.securitronlinux.com/
http://www.securitronlinux.com/linux/caring_computer.php
http://www.securitronlinux.com/linux/caring_computer.php
http://www.securitronlinux.com/linux/fedora.php
http://www.securitronlinux.com/linux/fedora.php
http://www.securitronlinux.com/linux/security.php
http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/
http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/
http://www.securitronlinux.com/linux/susedvd.php
http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/
http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/
http://www.securitronlinux.com/linux/doom_files.php
http://www.securitronlinux.com/linux/doom_files.php
http://www.securitronlinux.com/linux/doom_wadfiles.php
http://www.securitronlinux.com/linux/doom_wadfiles.php
http://www.securitronlinux.com/linux/console_codes.php
http://www.securitronlinux.com/linux/console_codes.php
http://www.securitronlinux.com/linux/misc_codes.php
http://www.securitronlinux.com/linux/misc_codes.php
http://www.securitronlinux.com/linux/psx_doom.php
http://www.securitronlinux.com/linux/psx_doom.php
http://www.securitronlinux.com/linux/zdoom_acs.php
http://www.securitronlinux.com/linux/windows_3.0.php
http://www.securitronlinux.com/linux/windows_3.0.php
http://www.securitronlinux.com/linux/ubuntu-karmic.php
http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/
http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/
http://www.securitronlinux.com/linux/windows_7.php
http://www.securitronlinux.com/linux/windows_7.php
http://www.securitronlinux.com/linux/johnjohnrace.swf
http://www.securitronlinux.com/linux/doomshots/
http://www.securitronlinux.com/linux/doomshots/
http://www.securitronlinux.com/xmlrpc.php
http://www.securitronlinux.com/wp-content/themes/securitron/style.css
http://www.securitronlinux.com/page/2/
http://www.securitronlinux.com/page/2/
http://www.securitronlinux.com/feed/
http://www.securitronlinux.com/comments/feed/
http://www.securitronlinux.com/wp-includes/js/jquery/jquery.js?ver=1.12.3
http://www.securitronlinux.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.0
http://www.securitronlinux.com/wp-json/
http://www.securitronlinux.com/xmlrpc.php?rsd

No comments have been made. Use this form to start the conversation :)

Leave a Reply