Pavuk

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set. Pavuk is a multifunctional open source web grabber with slow but continous development. Its features include

• recursive downloading based on links inside HTML documents
• transformation of Gopher and FTP directories into HTML document
• supports proxy servers (HTTP, FTP, SSL, HTTP gateway for FTP, HTTP gateway for Gopher, SOCKS 4/5)
• supports authentication against HTTP servers and proxy HTTP servers
• does restart of transfer after program break, link down, timeout or some other error
• can be run on a terminal or inside an X windows window
• have Native Language Support based on GNU gettext
• FTP over SSL
• multiple round-robin used HTTP proxies