Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.

Norconex HTTP Collector - Enterprise Web Crawler

Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc. Its main feature include
  <UL>
	<LI>Fetching, parsing and indexation in parallel and distributed</LI>
	<LI>Plugin support</LI>
	<LI>Ontology</LI>
	<LI>Clustering</LI>
	<LI>Distributed filesystem (via Hadoop)</LI>
	<LI>Link-graph database</LI>
	<LI>NTLM authentication</LI>
	<LI>MapReduce</LI>
	<LI>Many formats: plain text, HTML, XML, ZIP, OpenDocument (OpenOffice.org), Microsoft Office (Word, Excel, Powerpoint), PDF, JavaScript, RSS, RTF, MP3 (ID3 tags)</LI>
 </UL>

Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc.

Nutch - Highly extensible, highly scalable Web crawler

YaCy (read "ya see") is a free distributed search engine, built on principles of peer-to-peer (P2P) networks. It is distributed on several hundred computers so-called YaCy-peers. Each YaCy-peer independently crawls through the Internet, analyzes and indexes found web pages, and stores indexing results in a common database which is shared with other YaCy-peers using principles of P2P networks.

YaCy - Decentralized Web Search

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity. <BR><BR>It provides web interface for operator control and monitoring of crawls. It stores content to ARC or ISO WARC aggregate/transcript format.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.

Heritrix

Norconex HTTP Collector is a web spider, or crawler that aims to make Enterprise Search integrators and developers's life easier. It is Portable, Extensible, reusable, Robots.txt support, Obtain and manipulate document metadata, Resumable upon failure and lot more.

Norconex HTTP Collector - A Web Crawler in Java

Discover open source projects across all platforms

Projects

Norconex HTTP Collector - Enterprise Web Crawler

Nutch - Highly extensible, highly scalable Web crawler

YaCy - Decentralized Web Search

Heritrix

Norconex HTTP Collector - A Web Crawler in Java

Tagcloud

License

Suggested keywords:

Projects

Norconex HTTP Collector - Enterprise Web Crawler

Nutch - Highly extensible, highly scalable Web crawler

YaCy - Decentralized Web Search

Heritrix

Norconex HTTP Collector - A Web Crawler in Java

Tagcloud

License