Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Scrapy - Web crawling & scraping framework for Python

Gigablast is one of the remaining four search engines in the United States that maintains its own searchable index of over a billion pages. It is scalable to thousands of servers. Has scaled to over 12 billion web pages on over 200 servers. It supports Distributed web crawler, Document conversion, Automated data corruption detection and repair,  Can cluster results from same site, Synonym search, Spell checker and lot more.  

Gigablast - Web and Enterprise search engine in C++

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set. Pavuk is a multifunctional open source web grabber with slow but continous development. Its features include
  <UL>
	<LI>recursive downloading based on links inside HTML documents</LI>
	<LI>transformation of Gopher and FTP directories into HTML document </LI>
	<LI>supports proxy servers (HTTP, FTP, SSL, HTTP gateway for FTP, HTTP gateway for Gopher, SOCKS 4/5) </LI>
	<LI>supports authentication against HTTP servers and proxy HTTP servers </LI>
	<LI>does restart of transfer after program break, link down, timeout or some other error </LI>
	<LI>can be run on a terminal or inside an X windows window</LI>
	<LI>have Native Language Support based on GNU gettext</LI>
	<LI>FTP over SSL</LI>
	<LI>multiple round-robin used HTTP proxies </LI>
 </UL>

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set.

Pavuk

Discover open source projects across all platforms

Projects

Scrapy - Web crawling & scraping framework for Python

Gigablast - Web and Enterprise search engine in C++

Pavuk

TechStack

Tagcloud

License

Suggested keywords:

Projects

Scrapy - Web crawling & scraping framework for Python

Gigablast - Web and Enterprise search engine in C++

Pavuk

TechStack

Tagcloud

License