Voldemort is a distributed key-value storage system. Data is automatically replicated over multiple servers. Data is automatically partitioned so each server contains only a subset of the total data. 
Server failure is handled transparently. It is used at LinkedIn for certain high-scalability storage problems where simple functional partitioning is not sufficient.

Project-voldemort - A distributed database, Clone of Amazon's Dynamo

Alibaba JStorm is an enterprise fast and stable streaming process engine. It runs program up to 4x faster than Apache Storm. It is easy to switch from record mode to mini-batch mode. It is not only a streaming process engine. It means one solution for real time requirement, whole realtime ecosystem.


jstorm - Enterprise Stream Process Engine

The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. Cassandra is suitable for applications that can't afford to lose data. Data is automatically replicated to multiple nodes for fault-tolerance. 

 Cassandra provides support for Scale out, load balancing, cluster growth, Flexible schema, Key-oriented queries and CAP theorem (Consistency, Availability, Partition tolerance). It is in use at Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick, Cisco and more companies.

The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. Cassandra is suitable for applications that can't afford to lose data. Data is automatically replicated to multiple nodes for fault-tolerance.

Cassandra - Scalable Distributed Database

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites. Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat. Solr uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language. Solr's powerful external configuration allows it to be tailored to almost any type of application without Java coding, and it has an extensive plugin architecture when more advanced customization is required. Its feature set include Rich Document Parsing and Indexing (PDF, Word, HTML, etc) using Apache Tika, An Administration Interface, Monitorable Logging, Fast Incremental Updates and Index Replication, Highly Scalable Distributed search with sharded index across multiple hosts, HTTP interface with configurable response formats (XML/XSLT, JSON, Python, Ruby, PHP, Velocity) Solrj is embedded solr. It provides Java based API and it takes care of constructing, parsing, sending and receiving HTTP request.

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites. 

Solr - Blazing-fast, open source enterprise search platform

IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete. 
 Homepage: <A HREF="http://indextank.com/" target="_blank">http://indextank.com/</A>

IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete.

IndexTank - Search Engine powers Reddit

JanusGraph is a highly scalable graph database optimized for storing and querying large graphs with billions of vertices and edges distributed across a multi-machine cluster. JanusGraph is a transactional database that can support thousands of concurrent users, complex traversals, and analytic graph queries.
<ul>
 <li>Elastic and linear scalability for a growing data and user base.</li>
 <li>Data distribution and replication for performance and fault tolerance.</li>
 <li>Multi-datacenter high availability and hot backups.</li>
</ul>

JanusGraph is a highly scalable graph database optimized for storing and querying large graphs with billions of vertices and edges distributed across a multi-machine cluster. JanusGraph is a transactional database that can support thousands of concurrent users, complex traversals, and analytic graph queries.

JanusGraph - Distributed graph database

HBase provides support to handle BigTable - billions of rows X millions of columns. It is a scalable, distributed, versioned, column-oriented store modeled after Google's Bigtable and runs on top of HDFS (Hadoop Distributed Filesystem). It features compression, in-memory operation per-column. Data could be replicated between the nodes. HBase is used in Facebook and Twitter.

HBase -  Hadoop database

Redisson - Distributed and Scalable Java data structures (Set, SortedSet, Map, ConcurrentMap, List, Queue, Deque, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, HyperLogLog) on top of Redis server. Advanced redis java client. It supports over 28+ data structures and services, Synchronous / asynchronous / reactive interfaces and lot more.

Redisson - Redis based In-Memory Data Grid for Java

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.
 
Its features include: 
<ul>
<li>Pub/Sub messaging model</li>
<li>Scheduled message delivery</li>
<li>Message retroactivity by time or offset</li>
<li>Log hub for streaming</li>
<li>Big data integration</li>
<li>Reliable FIFO and strict ordered messaging in the same queue</li>
<li>Efficient pull&amp;push consumption model</li>
<li>Million-level message accumulation capacity in a single queue</li>
<li>Multiple messaging protocols like JMS and OpenMessaging</li>
<li>Flexible distributed scale-out deployment architecture</li>
<li>Lightning-fast batch message exchange system</li>
<li>Various message filter mechanics such as SQL and Tag</li>
<li>Docker images for isolated testing and cloud isolated clusters</li>
<li>Feature-rich administrative dashboard for configuration, metrics and monitoring</li>
</ul>

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

RocketMQ - Distributed messaging and streaming data platform

Crate is an open source, highly scalable, shared-nothing distributed SQL database. Crate offers the scalability and performance of a modern No-SQL database with the power of Standard SQL. Crate’s distributed SQL query engine lets you use the same syntax that already exists in your applications or integrations, and have queries seamlessly executed across the crate cluster, including any aggregations, if needed.

Crate - The fast, scalable, easy to use SQL database with native full text search

Titan is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster. Titan is a transactional database that can support thousands of concurrent users executing complex graph traversals. It is a native Blueprints enabled graph database and as such, it supports the full TinkerPop stack of technologies.

Titan - Scalable Graph Database 

The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Apache Accumulo is based on Google's BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift. Apache Accumulo features a few novel improvements on the BigTable design in the form of cell-based access control and a server-side programming mechanism that can modify key/value pairs at various points in the data management process.

Apache Accumulo - Key Value Store based on Google BigTable

Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. It allows querying data from relational / nosql databases. A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization. It is developed by Facebook.

Presto - Distributed SQL query engine for big data

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. It is horizontally scalable (Millions of independent topics and millions of messages published per second), Strong ordering and consistency guarantees, Low latency , REST API, Geo Replication and lot more.

Pulsar - Distributed pub-sub Messaging System from Yahoo

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Flume - Log management using HDFS

Discover open source projects across all platforms

Projects