Kudu is a storage system for tables of structured data. Kudu provides a combination of fast inserts/updates and efficient columnar scans to enable multiple real-time analytic workloads across a single storage layer. As a new complement to HDFS and Apache HBase, Kudu gives architects the flexibility to address a wider variety of use cases without exotic workarounds.

Kudu - Hadoop storage layer to enable fast analytics on fast data

ConcourseDB is a distributed self-tuning database with automatic indexing, version control and ACID transactions. ConcourseDB provides a more intuitive approach to data management that is easy to deploy, access and scale while maintaining the strong consistency of traditional database systems.

ConcourseDB  - Self-tuning database designed for both transactions and ad hoc analytics across time

Compass provides real time searchengine built on top of Lucene. It is distributed, transcational, supports Spring MVC, integrates with ORM. Simple Compass provides a simple API for working with Lucene. If you know how to use an ORM, then you will feel right at home with Compass with simple operations for save, and delete & query. Compass provides google-style search, index updates as well as more advanced concepts such as caching and index sharding (sub indexes). Compass also uses built in optimizations for concurrent commits and merges. It provides support to map	Object, JSON, XML to search engine mapping. It allows storing the index with Data Grid products such as GigaSpaces, Coherence and Terracotta.

Compass is a real time searchengine. It is built on top of lucene. It is transactional, distributed, supports Spring MVC, integrates with Hibernate.    

compass - Searchengine built on top of Lucene

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
 
Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

Apache Beam - Unified model for defining both batch and streaming data-parallel processing pipelines

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It has run in production at Yahoo scale for over 3 years, with millions of messages per second across millions of topics. It is Horizontally scalable, Low latency, High throughput, Multi-tenancy, Geo-replication, Transparent batching of messages, Transparent handling of partitioned topics, REST API for provisioning, admin and stats and lot more. 

Apache Pulsar - Distributed pub-sub messaging platform

SINGA is a distributed deep learning platform for big data analytics. It supports various deep learning models, and thus has the flexibility to allow users to customize the models that fit their business requirements. It provides a scalable architecture to train deep learning models from huge volumes of data and it makes the distributed training process transparent to users.

Apache Singa - Distributed Deep Learning Platform

CurioDB is a distributed and persistent Redis clone, built with Scala and Akka. Please note that despite the fancy logo, this is a toy project, hence the name "Curio", and any suitability as a drop-in replacement for Redis is purely incidental.

CurioDB - Distributed & Persistent Redis Clone built with Scala & Akka

NoKey is a distributed password manager that works without a master password. Instead, you can unlock your passwords by confirming from another device. E.g. if you need a password on your PC, you only have to confirm this on your phone. No need to remember any passwords. 
The passwords are encrypted using AES with a randomly generated key, here called group key. This group key is never stored anywhere directly. Instead, NoKey uses Shamir's Secret Sharing to split the group key into multiple key shares.

NoKey is a distributed password manager that works without a master password. Instead, you can unlock your passwords by confirming from another device. E.g. if you need a password on your PC, you only have to confirm this on your phone. No need to remember any passwords. 

NoKey - A distributed password manager without a master password

The MariaDB Corporation MaxScale is an intelligent proxy that allows forwarding of database statements to one or more database servers using complex rules, a semantic understanding of the database statements and the roles of the various servers within the backend cluster of databases. It extends the high availability, scalability, and security of MariaDB Server while at the same time simplifying application development by decoupling it from underlying database infrastructure.
 
MaxScale is designed to provide load balancing and high availability functionality transparently to the applications. In addition it provides a highly scalable and flexible architecture, with plugin components to support different protocols and routing decisions.
 
Its feature include Querying caching, Schema-based sharding, Data masking, Database firewall and lot more.

The MariaDB Corporation MaxScale is an intelligent proxy that allows forwarding of database statements to one or more database servers using complex rules, a semantic understanding of the database statements and the roles of the various servers within the backend cluster of databases. It extends the high availability, scalability, and security of MariaDB Server while at the same time simplifying application development by decoupling it from underlying database infrastructure.

MaxScale - An intelligent database proxy

Bedrock is a simple, modular, WAN-replicated, Blockchain-based data foundation for global-scale applications. Bedrock was built by Expensify, and is a networking and distributed transaction layer built atop SQLite, the fastest, most reliable, and most widely distributed database in the world. Bedrock is Blockchain-based. This means it uses a private blockchain to synchronize and self organize.

Bedrock - Rock solid distributed database specializing in active/active automatic failover and WAN replication

The Spread toolkit provides a high performance messaging service that is resilient to faults across local and wide area networks. Spread functions as a unified message bus for distributed applications, and provides highly tuned application-level multicast, group communication, and point to point support. Spread services range from reliable messaging to fully ordered messages with virtual synchrony delivery guarantees.

Spread toolkit - High performance messaging service that is resilient to faults across local and wide area networks

A lightweight C++11 Distributed Hash Table implementation. OpenDHT provides an easy to use distributed in-memory data store. Every node in the network can read and write values to the store. Values are distributed over the network, with redundancy.

OpenDHT - A C++11 Distributed Hash Table implementation

StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. StormCrawler is a library and collection of resources that developers can leverage to build their own crawlers. The good news is that doing so can be pretty straightforward. Often, all you'll have to do will be to declare StormCrawler as a Maven dependency, write your own Topology class (tip : you can extend ConfigurableTopology), reuse the components provided by the project and maybe write a couple of custom ones for your own secret sauce. 

Storm Crawler - Web crawler SDK based on Apache Storm 

Fully asynchronous C implementation of the Raft consensus protocol. The library has modular design: its core part implements only the core Raft algorithm logic, in a fully platform independent way. On top of that, a pluggable interface defines the I/O implementation for networking (send/receive RPC messages) and disk persistence (store log entries and snapshots). 

Raft - C implementation of the Raft consensus protocol 

Bangdb is pure vanilla key value nosql data store. The goal of bangdb is to be fast, reliable, robust, scalable and easy to use data store for various data management services required by applications. Bangdb comes in flavors like Embedded In memory, Network, Distributed data grid/ Elastic Cache. The bangdb is highly concurrent and runs parallel operations as much as possible.

BangDB - NoSQL for Real Time Performance

Discover open source projects across all platforms

Projects