Aperture is a Java framework for extracting and querying full-text content and metadata from various information systems. It could crawl and extract information from File system, Websites, Mail boxes and Mail servers. It supports various file formats like Office, PDF, Zip and lot more. Metadata information is extracted from image files. Aperture has a strong focus on semantics, metadata extracted could be mapped to predefined properties.

Aperture - Java framework for getting data and metadata

Norconex HTTP Collector is a web spider, or crawler that aims to make Enterprise Search integrators and developers's life easier. It is Portable, Extensible, reusable, Robots.txt support, Obtain and manipulate document metadata, Resumable upon failure and lot more.

Norconex HTTP Collector - A Web Crawler in Java

StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. StormCrawler is a library and collection of resources that developers can leverage to build their own crawlers. The good news is that doing so can be pretty straightforward. Often, all you'll have to do will be to declare StormCrawler as a Maven dependency, write your own Topology class (tip : you can extend ConfigurableTopology), reuse the components provided by the project and maybe write a couple of custom ones for your own secret sauce. 

Storm Crawler - Web crawler SDK based on Apache Storm 

Discover open source projects across all platforms

Projects

Aperture - Java framework for getting data and metadata

Norconex HTTP Collector - A Web Crawler in Java

Storm Crawler - Web crawler SDK based on Apache Storm

Tagcloud

License

Suggested keywords:

Projects

Aperture - Java framework for getting data and metadata

Norconex HTTP Collector - A Web Crawler in Java

Storm Crawler - Web crawler SDK based on Apache Storm

Tagcloud

License