Compass provides real time searchengine built on top of Lucene. It is distributed, transcational, supports Spring MVC, integrates with ORM. Simple Compass provides a simple API for working with Lucene. If you know how to use an ORM, then you will feel right at home with Compass with simple operations for save, and delete & query. Compass provides google-style search, index updates as well as more advanced concepts such as caching and index sharding (sub indexes). Compass also uses built in optimizations for concurrent commits and merges. It provides support to map	Object, JSON, XML to search engine mapping. It allows storing the index with Data Grid products such as GigaSpaces, Coherence and Terracotta.

Compass is a real time searchengine. It is built on top of lucene. It is transactional, distributed, supports Spring MVC, integrates with Hibernate.    

compass - Searchengine built on top of Lucene

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
 
Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It has run in production at Yahoo scale for over 3 years, with millions of messages per second across millions of topics. It is Horizontally scalable, Low latency, High throughput, Multi-tenancy, Geo-replication, Transparent batching of messages, Transparent handling of partitioned topics, REST API for provisioning, admin and stats and lot more. 

Apache Pulsar - Distributed pub-sub messaging platform

StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. StormCrawler is a library and collection of resources that developers can leverage to build their own crawlers. The good news is that doing so can be pretty straightforward. Often, all you'll have to do will be to declare StormCrawler as a Maven dependency, write your own Topology class (tip : you can extend ConfigurableTopology), reuse the components provided by the project and maybe write a couple of custom ones for your own secret sauce. 

Storm Crawler - Web crawler SDK based on Apache Storm 

A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC. 
Its design objectives are: 
<ol>
<li>Fast: close to the speed of direct memory access, both produding and consuming are close to O(1) memory access.</li>
<li>Persistent : all data in the system is persisted on disk and is crash resistant.</li>
<li>High-throughput: even with modest hardware Luxun can support handreds of thousands of messages per second.</li>
<li>Realtime : messages produced by producer threads should be immediately visible to consumer threads.</li>
<li>Distributed : explicit support for partitioning messages over Luxun servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.</li>
<li>Multiple clients support : use <a href="http://thrift.apache.org/">Thrift RPC</a> as communication infrastructure, easy to generate different clients for different platforms.</li>
<li>Flexible consuming semantics : supports typical consume once queue, fan out queue, even supports consume by index when needed.</li>
</ol>

A high-throughput, persistent, distributed, publish-subscribe messaging system based on memory mapped file and Thrift RPC.

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo

Discover open source projects across all platforms

Projects

compass - Searchengine built on top of Lucene

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

Apache Pulsar - Distributed pub-sub messaging platform

Storm Crawler - Web crawler SDK based on Apache Storm

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo

Tagcloud

License

Suggested keywords:

Projects

compass - Searchengine built on top of Lucene

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

Apache Pulsar - Distributed pub-sub messaging platform

Storm Crawler - Web crawler SDK based on Apache Storm

Luxun - A high-throughput, persistent, distributed, publish-subscribe messaging system based on memo

Tagcloud

License