Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel by doing load balancing. It also supports to call functions between languages. It is the nervous system for how distributed processing communicates. It is fault tolerant. A Gearman powered application consists of three parts: a client, a worker, and a job server. The client is responsible for creating a job to be run and sending it to a job server. The job server will find a suitable worker that can run the job and forwards the job on. The worker performs the work requested by the client and sends a response to the client through the job server.

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel by doing load balancing. It also supports to call functions between languages. It is the nervous system for how distributed processing communicates. It is fault tolerant.

Gearman - Application Framework to farm out work to other Machines

Riemann monitors distributed systems. It aggregates events from your servers and applications with a powerful stream processing language. Send an email for every exception raised by your code. Track the latency distribution of your web app. See the top processes on any host, by memory and CPU. Combine statistics from every Riak node in your cluster and forward to Graphite.


Riemann - Monitors Distributed Systems

FlockDB is much simpler than other graph databases such as neo4j because it tries to solve fewer problems. It scales horizontally and is designed for on-line, low-latency, high throughput environments such as web-sites. Twitter uses FlockDB to store social graphs (who follows whom, who blocks whom) and secondary indices. As of April 2010, the Twitter FlockDB cluster stores 13+ billion edges and sustains peak traffic of 20k writes/second and 100k reads/second.

FlockDB - A distributed, fault-tolerant graph database from Twitter

The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Apache Accumulo is based on Google's BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift. Apache Accumulo features a few novel improvements on the BigTable design in the form of cell-based access control and a server-side programming mechanism that can modify key/value pairs at various points in the data management process.

Apache Accumulo - Key Value Store based on Google BigTable

Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. It allows querying data from relational / nosql databases. A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization. It is developed by Facebook.

Presto - Distributed SQL query engine for big data

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It is horizontally scalable (Millions of independent topics and millions of messages published per second), Strong ordering and consistency guarantees, Low latency , REST API, Geo Replication and lot more.

Pulsar - Distributed pub-sub Messaging System from Yahoo

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Flume - Log management using HDFS

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally, so that it can scale to larger data sets and higher query rates as needed.
 
Its features include: 
<ul>
<li>A column-oriented database with various compression schemes such as Run Length, Fixed Bit Length</li>
<li>Pluggable indexing technologies - Sorted Index, Bitmap Index, Inverted Index</li>
<li>Ability to optimize query/execution plan based on query and segment metadata .</li>
<li>Near real time ingestion from Kafka and batch ingestion from Hadoop</li>
<li>SQL like language that supports selection, aggregation, filtering, group by, order by, distinct queries on fact data.</li>
<li>Support for multivalued fields</li>
<li>Horizontally scalable and fault tolerant</li>
</ul>

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally, so that it can scale to larger data sets and higher query rates as needed.

Pinot - A realtime distributed OLAP datastore

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn. It supports JSON data model, Distributed joins, subqueries, aggregation, atomic updates, Hadoop-style map/reduce.
 
It is scalable, supports sharding and replication to multiple nodes. It takes care of machine failures and network interrupts.

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn. It supports JSON data model, Distributed joins, subqueries, aggregation, atomic updates, Hadoop-style map/reduce.

RethinkDB - Distributed JSON database

RhodeCode is an open source repository management platform. It provides unified security and team collaboration across Git, Subversion, and Mercurial. Pick the repository type depending on your project, not on your code management tool. Read and write in all repository types from one central point, and see commits, forks, and changeset merges at a glance. Its features include Intuitive System Management, Full text search, Access Management, Change log etc.

RHodeCode - Centralized control for distributed repositories

Performance Co-Pilot (PCP) provides a framework and services to support system-level performance monitoring and management. It presents a unifying
abstraction for all of the performance data in a system, and many tools for interrogating, retrieving and processing that data. The distributed PCP
architecture makes it especially useful for those seeking centralized monitoring of distributed processing.

Performance Co-Pilot - System Performance and Analysis Framework.

Cadence is a distributed, scalable, durable, and highly available orchestration engine we developed at Uber Engineering to execute asynchronous long-running business logic in a scalable and resilient way.
 
Business logic is modeled as workflows and activities. Workflows are the implementation of coordination logic. Its sole purpose is to orchestrate activity executions. Activities are the implementation of a particular task in the business logic. The workflow and activity implementation are hosted and executed in worker processes. These workers long-poll the Cadence server for tasks, execute the tasks by invoking either a workflow or activity implementation, and return the results of the task back to the Cadence server. Furthermore, the workers can be implemented as completely stateless services which in turn allows for unlimited horizontal scaling.

Cadence is a distributed, scalable, durable, and highly available orchestration engine we developed at Uber Engineering to execute asynchronous long-running business logic in a scalable and resilient way.

Cadence - Distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way

YaCy (read "ya see") is a free distributed search engine, built on principles of peer-to-peer (P2P) networks. It is distributed on several hundred computers so-called YaCy-peers. Each YaCy-peer independently crawls through the Internet, analyzes and indexes found web pages, and stores indexing results in a common database which is shared with other YaCy-peers using principles of P2P networks.

YaCy - Decentralized Web Search

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information. 
 
Nebula Graph is Symmetrically distributed, Separates Storage and computation, Horizontal scalability, Strong data consistency by RAFT protocol, Role-based access control for higher level security and lot more.

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information. 

Nebula Graph - A distributed, scalable, lightning-fast graph database

A simple, fast, robust job/task queue for Node.js, backed by Redis.Bee-Queue is meant to power a distributed worker pool and was built with short, real-time jobs in mind. A web server can enqueue a job, wait for a worker process to complete it, and return its results within an HTTP request. Scaling is as simple as running more workers.

Bee Queue - A simple, fast, robust job/task queue for Node.js, backed by Redis

Discover open source projects across all platforms

Projects