NATS Server is a simple, high performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. It implements a highly scalable and elegant publish-subscribe (pub/sub) distribution model. The performant nature of NATS make it an ideal base for building modern, reliable, scalable cloud native distributed systems.


gnatsd - High-Performance server for NATS, the cloud native messaging system.

Atomix is an event-driven framework for coordinating fault-tolerant distributed systems built on the Raft consensus algorithm. It provides the building blocks that solve many common distributed systems problems including group membership, leader election, distributed concurrency control, partitioning, and replication.
 
Atomix provides a collection of dead simple, asynchronous APIs for sharing mission critical state and solving a variety of common distributed systems problems. It features reliable data consistency guarantees that are maintained even when machine or network failures occur. It clusters are resilient to failure, automatically replacing failed replicas as needed without any data loss. Atomix scales along with the rest of your system, providing high read throughput while maintaining strong write consistency.

Atomix is an event-driven framework for coordinating fault-tolerant distributed systems built on the Raft consensus algorithm. It provides the building blocks that solve many common distributed systems problems including group membership, leader election, distributed concurrency control, partitioning, and replication.


Atomix - Scalable, fault-tolerant distributed systems protocols and primitives for the JVM

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".
 
Instead of managing all file metadata in a central master, the central master only manages file volumes, and it lets these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (just one disk read operation).
 
SeaweedFS started by implementing <a href="http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf">Facebook's Haystack design paper</a>. SeaweedFS is currently growing, with more features on the way.

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, SeaweedFS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

SeaweedFS - Simple and highly scalable distributed file system

Micro is an operating system built for the cloud. It addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provides a set of services which act as the building blocks of a platform. Micro deals with the complexity of distributed systems and provides simpler programmable abstractions to build on.
 
Micro is built as a microservices architecture and abstracts away the complexity of the underlying infrastructure. It is composed as a single logical server to the user but decompose that into the various building block primitives that can be plugged into any underlying system.
The server is composed of the following services.<ul>
 <li>API - HTTP Gateway which dynamically maps http/json requests to RPC using path based resolution</li>
 <li>Auth - Authentication and authorization out of the box using jwt tokens and rule based access control.</li>
 <li>Broker - Ephemeral pubsub messaging for asynchronous communication and distributing notifications</li>
 <li>Config - Dynamic configuration and secrets management for service level config without the need to restart</li>
 <li>Events - Event streaming with ordered messaging, replay from offsets and persistent storage</li>
 <li>Network - Inter-service networking, isolation and routing plane for all internal request traffic</li>
 <li>Proxy - An identity aware proxy used for remote access and any external grpc request traffic</li>
 <li>Runtime - Service lifecycle and process management with support for source to running auto build</li>
 <li>Registry - Centralised service discovery and API endpoint explorer with feature rich metadata</li>
 <li>Store - Key-Value storage with TTL expiry and persistent crud to keep microservices stateless</li></ul>

Micro is an operating system built for the cloud. It  addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provides a set of services which act as the building blocks of a platform. Micro deals with the complexity of distributed systems and provides simpler programmable abstractions to build on.

Micro - Distributed Cloud OS 

Trino is a highly parallel and distributed query engine, that is built from the ground up for efficient, low latency analytics. It is an ANSI SQL compliant query engine, that works with BI tools such as R, Tableau, Power BI, Superset and many others. It helps to natively query data in Hadoop, S3, Cassandra, MySQL, and many others, without the need for complex, slow, and error-prone processes for copying the data. 
 
Access data from multiple systems within a single query. For example, join historic log data stored in an S3 object storage with customer data stored in a MySQL relational database.

Trino is a highly parallel and distributed query engine, that is built from the ground up for efficient, low latency analytics. It is an ANSI SQL compliant query engine, that works with BI tools such as R, Tableau, Power BI, Superset and many others. It helps to natively query data in Hadoop, S3, Cassandra, MySQL, and many others, without the need for complex, slow, and error-prone processes for copying the data. 

Trino - A query engine that runs at ludicrous speed

dramatiq is a distributed task processing library for Python with a focus on simplicity, reliability and performance.

Dramatiq - Simple distributed task processing for Python 3.

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it gracefully handles leader elections, and tolerates failures of machines, including the leader. rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available.
 
if you're interested in understanding how distributed systems actually work, rqlite is a good example to study. Much thought has gone into its design and implementation, with clear separation between the various components, including storage, distributed consensus, and API. rqlite uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all.

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it gracefully handles leader elections, and tolerates failures of machines, including the leader. rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available.


rqlite - The lightweight, distributed relational database built on SQLite

SpiceDB is a database system for managing security-critical application permissions. Developers create a schema that models their permissions requirements and use a client library to apply the schema to the database, insert data into the database, and query the data to efficiently check permissions in their applications.
 
The data used to calculate permissions have the most critical correctness requirements in the entirety a software system. Despite that, developers continue to build their own ad-hoc solutions coupled to the internal code of each new project. By developing a SpiceDB schema, you can iterate far more quickly and exhaustively test designs before altering any application code. This becomes especially important as you introduce backwards-compatible changes to the schema and want to ensure that the system remains secure.

SpiceDB is a database system for managing security-critical application permissions. Developers create a schema that models their permissions requirements and use a client library to apply the schema to the database, insert data into the database, and query the data to efficiently check permissions in their applications.

SpiceDB - Inspired by Google's Zanzibar paper, SpiceDB is a database system for managing security-critical application permissions

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system. 
Data is distributed among multiple chunk servers. You can easily add and remove drives or the whole storage nodes. It is as simple as “plug and scale”. You can easily scale your LizardFS cluster up and down both vertically and horizontally by a single drive or a single node. You can have thousands of chunkservers and up to 1 Exabyte of data. Geo replication allows you to create geo stretched clusters stretching virtually any distance.

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

LizardFS - Software Defined Storage, Distributed, Parallel, Scalable, Fault-tolerant, Geo-Redundant and Highly available file system

Quickwit is a distributed search engine built from the ground up to offer cost-efficiency and high reliability. By mere mortals for mere mortals, Quickwit's architecture is as simple as possible. Quickwit is written in Rust and built on top of the mighty tantivy library. It designed to index big datasets straight from object storage like AWS S3 in a stateless manner. 
<ul>
<li>Scalable distributed search: Host an arbitrary number of indexes on Amazon S3 and answer search queries in less than a second with a small pool of stateless search instances</li>
<li>Stream indexing: Ingest TB of data from your favorite distributed event streaming service. As of today, Quickwit supports Apache Kafka natively. The next releases will bring support for more platforms</li>
<li>Fault-tolerant architecture that won't lose data: Quickwit achieves exactly-once processing for indexing and safely stores your data on highly reliable object storage services such as Amazon S3</li>
<li>Cloud-native, easy to operate: Thanks to true decoupled compute and storage, search instances are stateless, add or remove search nodes within seconds</li>
<li>Sub-second full-text search on cloud / distributed storage: Quickwit Search re-designed indexing and index data structure to open it in less than 60ms on Amazon S3</li>
<li>Time-based sharding: Quickwit shards data by time when enabled. And you can use a second dimension to shard data thanks to our <a href="../../docs/design/querying">tags feature</a>. Time-based queries only access splits (a data piece of the index) that match the time range of the query which leads to significant performance improvements</li>
<li>Painless multi-tenant search: Create indexes for each tenant without hurting query performance. Or group tenants into one index and use tagging to prune irrelevant splits for your tenant query to improve significantly performance</li>
<li>Elasticsearch query language support</li>
<li>Decoupled compute and storage, stateless indexers &amp; searchers</li>
<li>Schemaless analytics</li>
</ul>

Quickwit is a distributed search engine built from the ground up to offer cost-efficiency and high reliability. By mere mortals for mere mortals, Quickwit's architecture is as simple as possible. Quickwit is written in Rust and built on top of the mighty tantivy library. It designed to index big datasets straight from object storage like AWS S3 in a stateless manner.

Quickwit - Fast and cost-efficient distributed search engine for immutable data

Dubbo is a high-performance, java based RPC framework open-sourced by Alibaba. As in many RPC systems, dubbo is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. On the server side, the server implements this interface and runs a dubbo server to handle client calls. On the client side, the client has a stub that provides the same methods as the server.

Dubbo - High-performance, java based, open source RPC framework

BookKeeper is a replicated log service which can be used to build replicated state machines. A log contains a sequence of events which can be applied to a state machine. BookKeeper guarantees that each replica state machine will see all the same entries, in the same order. It provides back end support to distributed systems, such as messaging systems, coordination systems, filesystems, etc.

BookKeeper - Replicated Log Service

Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine - written in any programming language - and securely replicates it on many machines.
 
Tendermint is software for securely and consistently replicating an application on many machines. By securely, we mean that Tendermint works even if up to 1/3 of machines fail in arbitrary ways. By consistently, we mean that every non-faulty machine sees the same transaction log and computes the same state. Secure and consistent replication is a fundamental problem in distributed systems; it plays a critical role in the fault tolerance of a broad range of applications, from currencies, to elections, to infrastructure orchestration, and beyond.
 
Tendermint consists of two chief technical components: a blockchain consensus engine and a generic application interface. The consensus engine, called Tendermint Core, ensures that the same transactions are recorded on every machine in the same order. The application interface, called the Application BlockChain Interface (ABCI), enables the transactions to be processed in any programming language. Unlike other blockchain and consensus solutions, which come pre-packaged with built in state machines (like a fancy key-value store, or a quirky scripting language), developers can use Tendermint for BFT state machine replication of applications written in whatever programming language and development environment is right for them.

Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine - written in any programming language - and securely replicates it on many machines.

Tendermint - Tendermint Core (BFT Consensus) in Go

Copycat is a fault-tolerant state machine replication framework. Built on the Raft consensus algorithm, it handles replication and persistence and enforces strict ordering of inputs and outputs, allowing developers to focus on single-threaded application logic. Its event-driven model allows for efficient client communication with replicated state machines, from simple key-value stores to wait-free locks and leader elections. You supply the state machine and Copycat takes care of the rest, making it easy to build robust, safe distributed systems.


Copycat - A novel implementation of the Raft consensus algorithm

Heimdall is an HTTP client that helps your application make a large number of requests, at scale. With Heimdall, you can use a hystrix-like circuit breaker to control failing requests, Add synchronous in-memory retries to each request, with the option of setting your own retrier strategy, Create clients with different timeouts for every request.

Heimdall - An enhanced HTTP client for Go

Discover open source projects across all platforms

Projects