How to Download An Entire Website?

Sometimes, we like to have an entire site or part of it archived in our disk so that we can read it even without Internet connection. But how to do it? Here are two easy to use tools that does the trick.

1. wget:

"wget" is a simple yet powerful command line tool. As usual it is a free software that respects your freedom as well as gives you the freedom to read a site even without a working Internet connection. This tool is available by default in almost all GNU/Linux distributions. It has lot of options. You can use it to download a single file or a web page. You can also download recursively, i.e. a page and all the links in that page. You can choose to download pictures, style sheets, JavaScript files. You can choose the domains from which the files can be downloaded (by this, you can skip unwanted ads getting downloaded).

I am giving here a sample of wget script using which you can download this site.

wget --recursive --timestamping --span-hosts --page-requisites --adjust-extension --convert-links --domains=blogspot.in,bp.blogspot.com --no-parent techmusicnmore.blogspot.in

Let us see the options one by one

--recursive

This tells wget to download a page and all the links in the page recursively. Us it with caution as some sites have many levels of links and you may end up downloading lot of data. You may need to specify the depth using --level=n option to restrict downloads after a 'n' layers of depth.

--timestamping

This is a clever option that downloads a file only if a newer version is available. For example, if you have downloaded a site already or canceled it in between, then you can restart it without worrying about wasting your bandwidth to re-download all the files. This option will make wget to download only non-existing files and re-download only outdated files.

--span-hosts

This tell that wget to look in to other servers if needed. As our blog's images are stored in other domain(s), we need to give this option. But be informed that this may cause you to download the entire Internet!!! So you can restrict the domains by white-listing them using --domains option or black-listing them using --exclude-domains option.

--domains=server1,server2

As discussed in previous option, this option is used to restrict the downloads to just the domains specified in the comma separated list.

--page-requisites

This option is to download all the artifacts required to display a page properly. That is to download images, scripts, css etc.

--adjust-extension

This option cleverly converts the file extensions of pages to html if they are not so by default. Some sites will have other namings like .jsp or .gsp. In such cases too, the files will be converted to .htm(l) extension.

--convert-links

This option is used to convert all the links in the pages to point to local location. Without this, all pages will be downloaded. But the links will still point to the actual website.

--no-parent

To tell wget that it should download ONLY the links/files below the given URL. It will not allow the parents of current link to be downloaded. For example, you have a web page with links to all articles written by a particular author. If you use wget without this option, then all the pages including home page, about us etc will be downloaded. With this option, only the pages written by that author will be downloaded.

wget in action; downloading this blog ;-)

These are not the only way to do the things. You can do lot of things with wget. There are some websites which will prevent automatic downloads like this. You can cheat them by acting like a real user by hitting the server with some time intervals and masquerading that the requests are from real browser.  For that you may need to go through the manual. The link is given below.

wget manual: https://www.gnu.org/software/wget/manual/wget.html

2. Httrack

If you are not running GNU/Linux, my first advice is to switch your OS. If you can't or you need a GUI based alternate to wget even in GNU/Linux, here is Httrack. This has all the options as in wget and lot more. But it has some other options that wget doesn't. That is mainly because, wget is developed for general reasons, while Httrack is specifically developed for website archiving and crawling. So it has a slight edge over wget.

This awesome tool is available for Windows, OSX, Android and of course GNU/Linux and BSD platforms. It is a free software released under GPLv3. So have no guilt in using it. If you are a GNU/Linux user, then download the source package using wget and compile it locally for extra fun!

You can get Httrack from here: http://www.httrack.com/
Httrack for Android: https://play.google.com/store/apps/details?id=com.httrack.android
Httrack manual: http://www.httrack.com/html/index.html

5 Resons to Use a Free Operationg System

1. Freedom and Free of Cost

With Free Software, you get freedom. That is what the "Free" in Free Software means. But for our people, who are brainwashed by corporate media, free also means for no cost. The good news is, most free operating systems also come for free of cost too. Don't come with vile excuses like you have a pirated copy of Windows or you got it bundled with your machine. Former is silly and later is stupid. It is silly to steal a product when a better product is available freely and it is stupid to claim that you got Windows free of cost with your machine, while you have paid for it through "Microsoft tax". Also you can be sure that no NSA agent is offered a backdoor in your machine. If you doubt it, look in to the source for yourself!

2. Secure

Free operating systems like GNU/Linux and BSD are more secure. There are only few know viruses exist for these platforms and they are also controlled currently. Unless you are foolish enough to reveal your password or keep a password that can be easily guessed or fooled to install a malware, you are almost safe. Even though I will suggest you to have a Free Software anti-virus program like Clam-AV to prevent propagating virus unknowingly to other platforms and any new threat.

3. All in One

When you install Windows, you have to install all drivers and other software separately. And some of them are costly too. It is time consuming to keep all of the installed software up to date. You have to check the software's website regularly and download the latest version and install them. With the software repository concept, (which was there as far back as I remember) Free operating systems allow you to install and maintain all software in one place. Also you can find free software alternate for almost all tools you like. When a new version of your favorite browser or media player is released, your system update itself will update them too. Isn't that easy?

4. Choice of UI

Don't like the Aero UI? You can switch to the Classic UI. Or you can console yourself by changing the color themes. These are the few options available with Windows. With the Free operating systems, you can have wide varieties like GNOME, KDE, MATE, Xfce, LXDE, Enlightenment, Cinnamon and I can't list them all here! You have that many choices. You can install whatever you want or you can install multiple desktop environments in a single system and switch between them in login screen. You can choose not to install any UI and use in CLI mode. This will suit servers and workstation for hardcore geeks like Torvalds or Stallman.

5. No Fragmentation

If you have used Windows for a good amount of time, then you should have noticed that the system slows with time. Why? This is due to file fragmentation. That is, files are stored scattered along the disk and the hard disk head has to move back and forth to read a sequence of data, say an audio or video file. This will also reduce the life of your hard disk. Windows comes with a defragmentor tool. But that too has lot of limitation. But free operating systems come with cutting edge journaled file systems which don't fragment with time. Even if they do very rarely, you can just copy the files to some other partition and copy them back. But I have never came across such situation.

Time to Change

Windows XP's support is going to end this April. So if you are running Windows XP, it is time to go free. Free as in freedom. Here is a list of OS you can choose to get rid of the buggy, insecure XP platform.

1. aLinux

You won't feel that you have changed your OS apart from performance improvemnts, being virus free and less buggy. aLinux comes with KDE 3.5, Koffice(MS office replacement), GIMP(like Photoshop), mp3/divx/flash support, Clam-AV antivirus etc.

Get yours at http://alinux.tv

2. Lubuntu

So, this distro's look and feel is much like XP. It consumes less resources and feels great on old hardware. It comes with Abi Word, MTPaint, Xfburn (cd/dvd burning), Sylpheed (like Outlook). With 256 MB RAM, you get a perfectly working system.

More info: http://lubuntu.net

3. PC-BSD with Xfce

Ok, it may be boaring with too much Linux. Say hi to BSD with PC-BSD. Xfce gives you a classic XP like interface. AppCafe gives easy access for all packages you need and you will love its clean UI.

Check this out: http://www.pcbsd.org

4. Manjaro

Still in beta, from the house of Arch, Manjaro ives you rolling release with a wide choices of desktop environments. I suggest MATE desktop, which is both modern and traditional simultaneously. Manjaro comes with out of the box multimedia support and good hardware driver support. Only downside is, it needs bit powerful hardware.

Get your copy at: http://manjaro.org

5. Bodhi

”Enlightenment” comes here. You need 128MB RAM to run this distro. I won't say it has all tools you need, but it will meet your expectation. The lightweight UI looks so polished and you can fool your friends that yours is a latest PC with a latest OS ;-)

You can negate your bad karma here: http://www.bodhilinux.com/

Top 10 Distros

Well, I have been using Free Software based distributions from 2004 (that was when I got my first own PC). I have come across many distributions. Now, I will share 10 of the best distributions I recommend people and their target audience. The list is sort of a count-down. So let us start with 10.

10. PC-BSD

BSD is not Linux. It is a UNIX clone too. Its file system, partition management, working differs greatly from Linux based distros. But this is a very solid platform and provides a secure and robust system suitable for servers and research and development labs. If you are freaky enough, you can try to install it as a dual booting system along with a GNU/Linux system.

9. Puppy Linux

Do you have a very old PC? Or have you managed to assemble a PC from junkyard? Here is a distro that can fire it up to life. Puppy Linux requires very less memory and other requirements. Of course it lacks fancy GUI and powerful tools like office applications. But it works like charm in very old hardware. Puppy is derived from Ubuntu.

8. Lubuntu

This LXDE flavor of Ubuntu is Puppy Linux for bit newer hardware. Also this packs good driver support and adds option to install latest and greatest packages. So if you have a 10 years old PC and want it to perform fast like a new one, Lubuntu is for you.

7. Gentoo

For those who doubt pre-compiled binaries in repositories, Gentoo is the answer. If you have enough time and powerful hardware, you can do the entire compilation of your GNU/Linux system right on your machine. You may find the installation bit tough. But it is rewarding. You never need to upgrade the system. You only need to update it. This is called rolling release. Compared to Fedora in which you need to upgrade frequently, this is relatively simple. Suitable for BASHers. So install once and forget how to install. You can still complain that the stage-3 tarballs are tainted by NSA ;-)

6. Arch Linux

This is the one for those who are not afraid of command line interface. You have to set up this distro by executing commands. No GUI based installation.This is much like Gentoo minus compilation. Arch is awesome and has a very good community to support. If you search for generic Linux issues, Arch wiki is where you land.

5. Ubuntu/Linux Mint

These are derivatives of Debian. Mint comes with Mate desktop environment for those who hate GNOME 3 and Ubuntu comes with Unity interface. These distros ship with lot of proprietary drivers. The reason I recommend them is because they are suitable for lazy people who just want everything working out of the box. Especially if you have laptops with locked hardware, this will work out in most cases. But the better suggestion is to choose a more open hardware and go for a better distribution.

4. CentOS

CentOS is community supported clone of RedHat Linux. This is one of the most stable and well tested distro. But it lags a lot in terms of package versions. For example, current version of CentOS (as of 26th Feb, 2014) ships with 2.6.x kernel, while a uname -r in my machine spat out 3.13.3. But this has its own plus. All packages are well tested. Almost all bugs fixed and most drivers for bit older hardware are readily available. This makes it suitable for servers and workstations. Also this distro as of now lacks fancy desktop environments. So it uses less resources and perform well.

3. OpenSUSE

OpenSUSE is community edition of SUSE from Novell. Made in Germany. IT has good driver support and an active community to help you out. If you can switch to Tumbleweed channel, you can get a taste of rolling release like that of Arch. Yast makes the system management easy. So this is the distro suitable for home PCs with good hardware configurations. It ships with awesome desktop environments like KDE 4 and GNOME 3. Everything you need in one place!

2. Fedora

This the future version of Red Hat. What is today Fedora will become Red Hat one day. It is one of the most bleeding edge distro. This makes it a bit hard to use for people with proprietary hardware. As the kernel and other stuffs are constantly updated, it may impact the system sometimes. Mostly the stable releases won't have any problem. But don't ask me about why x or y is not working after an update. This is suitable for power users with good open hardware (without proprietary graphic cards, wi-fi chips, web cams etc).

1. BLAG/Dragora/Dynebolic/gNewSense/Musix/Parabola/Trisquel

These are all Free Software Foundation (FSF)  approved distros. They may lack some drivers, no flash,  they can't play mp3s. These are not so fancy and these are not meant for kids playing flash games or adults watching porn. But if you care about freedom, these are the distributions you should run. Especially gNewSense is the distribution that Dr. Richard Stallman himself runs. You can find more information about these distributions at http://www.gnu.org/distros/free-distros.html