Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity. <BR><BR>It provides web interface for operator control and monitoring of crawls. It stores content to ARC or ISO WARC aggregate/transcript format.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.

Heritrix

Bobo Browse is an information retrieval technology that provides navigational browsing into a semi-structured dataset. Beyond the result set from queries and selections, Bobo Browse also provides the facets from this point of browsing. It provides support to sort documents on fields that have multiple values. It is stable and used by LinkedIn.

Bobo - Faceted search library based on Lucene

Katta is a scalable, failure tolerant, distributed, data storage for real time access. Katta serves large, replicated, indices as shards to serve high loads and very large data sets. These indices can be of different type. Currently implementations are available for Lucene and Hadoop mapfiles. Its features include:
  <ul>
	<li>Makes serving large or high load indices easy</li>
	<li>Serves very large Lucene or Hadoop Mapfile indices as index shards on many servers</li>
	<li>Replicate shards on different servers for performance and fault-tolerance</li>
	<li>Supports pluggable network topologies</li>
	<li>Master fail-over</li>
	<li>Fast, lightweight, easy to integrate</li>
	<li>Plays well with Hadoop clusters</li>
</ul>

Katta is a scalable, failure tolerant, distributed, data storage for real time access. Katta serves large, replicated, indices as shards to serve high loads and very large data sets. These indices can be of different type. Currently implementations are available for Lucene and Hadoop mapfiles.

Katta - Lucene and more in the cloud.

Discover open source projects across all platforms

Projects

Heritrix

Bobo - Faceted search library based on Lucene

Katta - Lucene and more in the cloud.

Tagcloud

License

Suggested keywords:

Projects

Heritrix

Bobo - Faceted search library based on Lucene

Katta - Lucene and more in the cloud.

Tagcloud

License