Voldemort is a distributed key-value storage system. Data is automatically replicated over multiple servers. Data is automatically partitioned so each server contains only a subset of the total data. 
Server failure is handled transparently. It is used at LinkedIn for certain high-scalability storage problems where simple functional partitioning is not sufficient.

Project-voldemort - A distributed database, Clone of Amazon's Dynamo

Alibaba JStorm is an enterprise fast and stable streaming process engine. It runs program up to 4x faster than Apache Storm. It is easy to switch from record mode to mini-batch mode. It is not only a streaming process engine. It means one solution for real time requirement, whole realtime ecosystem.


jstorm - Enterprise Stream Process Engine

Ray is an open-source unified compute framework that makes it easy to scale AI and Python workloads — from reinforcement learning to deep learning to tuning, and model serving. Ray is a unified way to scale Python and AI applications from a single node to a cluster.
 
Ray AI Runtime (AIR) is an open-source toolkit for building ML applications. It provides libraries for distributed data processing, model training, tuning, reinforcement learning, model serving, and more.
 
Ray Core provides a simple and flexible API for building and running your distributed applications. You can often parallelize single machine code with little to zero code changes.
 
With a Ray cluster you can deploy your workloads on AWS, GCP, Azure or on premise. You can also use Ray cluster managers to run Ray on your existing Kubernetes, YARN, or Slurm clusters.

Ray is an open-source unified compute framework that makes it easy to scale AI and Python workloads — from reinforcement learning to deep learning to tuning, and model serving. Ray is a unified way to scale Python and AI applications from a single node to a cluster.


Ray - A unified framework for scaling AI and Python applications

The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. Cassandra is suitable for applications that can't afford to lose data. Data is automatically replicated to multiple nodes for fault-tolerance. 

 Cassandra provides support for Scale out, load balancing, cluster growth, Flexible schema, Key-oriented queries and CAP theorem (Consistency, Availability, Partition tolerance). It is in use at Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick, Cisco and more companies.

The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. Cassandra is suitable for applications that can't afford to lose data. Data is automatically replicated to multiple nodes for fault-tolerance.

Cassandra - Scalable Distributed Database

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites. Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat. Solr uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language. Solr's powerful external configuration allows it to be tailored to almost any type of application without Java coding, and it has an extensive plugin architecture when more advanced customization is required. Its feature set include Rich Document Parsing and Indexing (PDF, Word, HTML, etc) using Apache Tika, An Administration Interface, Monitorable Logging, Fast Incremental Updates and Index Replication, Highly Scalable Distributed search with sharded index across multiple hosts, HTTP interface with configurable response formats (XML/XSLT, JSON, Python, Ruby, PHP, Velocity) Solrj is embedded solr. It provides Java based API and it takes care of constructing, parsing, sending and receiving HTTP request.

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites. 

Solr - Blazing-fast, open source enterprise search platform

IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete. 
 Homepage: <A HREF="http://indextank.com/" target="_blank">http://indextank.com/</A>

IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete.

IndexTank - Search Engine powers Reddit

JanusGraph is a highly scalable graph database optimized for storing and querying large graphs with billions of vertices and edges distributed across a multi-machine cluster. JanusGraph is a transactional database that can support thousands of concurrent users, complex traversals, and analytic graph queries.
<ul>
 <li>Elastic and linear scalability for a growing data and user base.</li>
 <li>Data distribution and replication for performance and fault tolerance.</li>
 <li>Multi-datacenter high availability and hot backups.</li>
</ul>

JanusGraph is a highly scalable graph database optimized for storing and querying large graphs with billions of vertices and edges distributed across a multi-machine cluster. JanusGraph is a transactional database that can support thousands of concurrent users, complex traversals, and analytic graph queries.

JanusGraph - Distributed graph database

HBase provides support to handle BigTable - billions of rows X millions of columns. It is a scalable, distributed, versioned, column-oriented store modeled after Google's Bigtable and runs on top of HDFS (Hadoop Distributed Filesystem). It features compression, in-memory operation per-column. Data could be replicated between the nodes. HBase is used in Facebook and Twitter.

HBase -  Hadoop database

As on the ground microservice practitioners quickly realize, the majority of operational problems that arise when moving to a distributed architecture are ultimately grounded in two areas: networking and observability. It is simply an orders of magnitude larger problem to network and debug a set of intertwined distributed services versus a single monolithic application.
 
Originally built at Lyft, Envoy is a high performance C++ distributed proxy designed for single services and applications, as well as a communication bus and “universal data plane” designed for large microservice “service mesh” architectures. Built on the learnings of solutions such as NGINX, HAProxy, hardware load balancers, and cloud load balancers, Envoy runs alongside every application and abstracts the network by providing common features in a platform-agnostic manner. When all service traffic in an infrastructure flows via an Envoy mesh, it becomes easy to visualize problem areas via consistent observability, tune overall performance, and add substrate features in a single place.
 
Envoy is hosted by the Cloud Native Computing Foundation (CNCF). If you are a company that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF.

As on the ground microservice practitioners quickly realize, the majority of operational problems that arise when moving to a distributed architecture are ultimately grounded in two areas: networking and observability. It is simply an orders of magnitude larger problem to network and debug a set of intertwined distributed services versus a single monolithic application.

Envoy - C++ front/service proxy

Micro is an operating system built for the cloud. It addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provides a set of services which act as the building blocks of a platform. Micro deals with the complexity of distributed systems and provides simpler programmable abstractions to build on.
 
Micro is built as a microservices architecture and abstracts away the complexity of the underlying infrastructure. It is composed as a single logical server to the user but decompose that into the various building block primitives that can be plugged into any underlying system.
The server is composed of the following services.<ul>
 <li>API - HTTP Gateway which dynamically maps http/json requests to RPC using path based resolution</li>
 <li>Auth - Authentication and authorization out of the box using jwt tokens and rule based access control.</li>
 <li>Broker - Ephemeral pubsub messaging for asynchronous communication and distributing notifications</li>
 <li>Config - Dynamic configuration and secrets management for service level config without the need to restart</li>
 <li>Events - Event streaming with ordered messaging, replay from offsets and persistent storage</li>
 <li>Network - Inter-service networking, isolation and routing plane for all internal request traffic</li>
 <li>Proxy - An identity aware proxy used for remote access and any external grpc request traffic</li>
 <li>Runtime - Service lifecycle and process management with support for source to running auto build</li>
 <li>Registry - Centralised service discovery and API endpoint explorer with feature rich metadata</li>
 <li>Store - Key-Value storage with TTL expiry and persistent crud to keep microservices stateless</li></ul>

Micro is an operating system built for the cloud. It  addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provides a set of services which act as the building blocks of a platform. Micro deals with the complexity of distributed systems and provides simpler programmable abstractions to build on.

Micro - Distributed Cloud OS 

kumofs is a scalable and highly available distributed key-value store. 
Data is replicated over multiple servers. 
Data is partitioned over multiple servers. 
Extreme single node performance; comparable with memcached. 
Both read and write performance got improved as servers added. 
Servers can be added without stopping the system. 
Servers can be added without changing the client applications. 
The system does not stop even if one or two servers crashed. 
The system does not stop to recover crashed servers. 
Scalable from 2 to 60 servers. (more than 60 servers has not be tested yet) 
Optimized for storing a large amount of small data. 
memcached protocol support. (get, set and delete only; expiration time and flags must be 0)

kumofs is a scalable and highly available distributed key-value store.

Kumofs

Redisson - Distributed and Scalable Java data structures (Set, SortedSet, Map, ConcurrentMap, List, Queue, Deque, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, HyperLogLog) on top of Redis server. Advanced redis java client. It supports over 28+ data structures and services, Synchronous / asynchronous / reactive interfaces and lot more.

Redisson - Redis based In-Memory Data Grid for Java

Vector is an open source on-host performance monitoring framework which exposes hand picked high resolution system and application metrics to every engineer’s browser. Having the right metrics available on-demand and at a high resolution is key to understand how a system behaves and correctly troubleshoot performance issues.

Vector - Performance Monitoring Framework

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.
 
Its features include: 
<ul>
<li>Pub/Sub messaging model</li>
<li>Scheduled message delivery</li>
<li>Message retroactivity by time or offset</li>
<li>Log hub for streaming</li>
<li>Big data integration</li>
<li>Reliable FIFO and strict ordered messaging in the same queue</li>
<li>Efficient pull&amp;push consumption model</li>
<li>Million-level message accumulation capacity in a single queue</li>
<li>Multiple messaging protocols like JMS and OpenMessaging</li>
<li>Flexible distributed scale-out deployment architecture</li>
<li>Lightning-fast batch message exchange system</li>
<li>Various message filter mechanics such as SQL and Tag</li>
<li>Docker images for isolated testing and cloud isolated clusters</li>
<li>Feature-rich administrative dashboard for configuration, metrics and monitoring</li>
</ul>

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

RocketMQ - Distributed messaging and streaming data platform

Crate is an open source, highly scalable, shared-nothing distributed SQL database. Crate offers the scalability and performance of a modern No-SQL database with the power of Standard SQL. Crate’s distributed SQL query engine lets you use the same syntax that already exists in your applications or integrations, and have queries seamlessly executed across the crate cluster, including any aggregations, if needed.

Crate - The fast, scalable, easy to use SQL database with native full text search

Discover open source projects across all platforms

Projects