Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.

Norconex HTTP Collector - Enterprise Web Crawler

Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc. Its main feature include
  <UL>
	<LI>Fetching, parsing and indexation in parallel and distributed</LI>
	<LI>Plugin support</LI>
	<LI>Ontology</LI>
	<LI>Clustering</LI>
	<LI>Distributed filesystem (via Hadoop)</LI>
	<LI>Link-graph database</LI>
	<LI>NTLM authentication</LI>
	<LI>MapReduce</LI>
	<LI>Many formats: plain text, HTML, XML, ZIP, OpenDocument (OpenOffice.org), Microsoft Office (Word, Excel, Powerpoint), PDF, JavaScript, RSS, RTF, MP3 (ID3 tags)</LI>
 </UL>

Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc.

Nutch - Highly extensible, highly scalable Web crawler

YaCy (read "ya see") is a free distributed search engine, built on principles of peer-to-peer (P2P) networks. It is distributed on several hundred computers so-called YaCy-peers. Each YaCy-peer independently crawls through the Internet, analyzes and indexes found web pages, and stores indexing results in a common database which is shared with other YaCy-peers using principles of P2P networks.

YaCy - Decentralized Web Search

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity. <BR><BR>It provides web interface for operator control and monitoring of crawls. It stores content to ARC or ISO WARC aggregate/transcript format.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.

Heritrix

Discover open source projects across all platforms

Projects

Norconex HTTP Collector - Enterprise Web Crawler

Nutch - Highly extensible, highly scalable Web crawler

YaCy - Decentralized Web Search

Heritrix

Tagcloud

License

Suggested keywords:

Projects

Norconex HTTP Collector - Enterprise Web Crawler

Nutch - Highly extensible, highly scalable Web crawler

YaCy - Decentralized Web Search

Heritrix

Tagcloud

License