Apache Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

Apache Hadoop is a framework for running applications on large clusters built of commodity hardware. Hadoop common supports other Hadoop subprojects

Hadoop Common

Hue is a Web application for interacting with Apache Hadoop. It supports a FileBrowser for accessing HDFS, JobBrowser for accessing MapReduce jobs (MR1/MR2-YARN),  Job Designer for creating MapReduce/Streaming/Java jobs, HBase Browser for exploring and modifying HBase tables and data, Oozie App for submitting and scheduling workflows and bundles, A Pig/HBase/Sqoop2 shell,  Beeswax application for executing Hive queries, Search app for querying Solr and Solr Cloud.


Hue - The open source Apache Hadoop UI

Apache Mnemonic is a non-volatile hybrid memory storage oriented library, it proposed a non-volatile/durable Java object model and durable computing service that bring several advantages to significantly improve the performance of massive real-time data processing/analytics. developers are able to use this library to design their cache-less and SerDe-less high performance applications.
<ul>
<li>In-place data storage on local non-volatile memory</li>
<li>Durable Object Model (DOM)</li>
<li>Durable Native Computing Model (DNCM)</li>
<li>Object graphs lazy loading &amp; sharing</li>
<li>Auto-reclaim memory resources and Mnemonic objects</li>
<li>Hierarchical cache pool for massive data caching</li>
<li>Extensible memory services for new device adoption and allocation optimization</li>
<li>Durable data structure collection(WIP)</li>
<li>Durable computing service</li>
<li>Minimize memory footprint of on-heap</li>
<li>Reduce GC Overheads as the following chart shown (collected from Apache Spark experiments)</li>
<li>Drop-in Hadoop MapReduce support</li>
<li>Drop-in Hadoop Spark support</li>
</ul>

Apache Mnemonic is a non-volatile hybrid memory storage oriented library, it proposed a non-volatile/durable Java object model and durable computing service that bring several advantages to significantly improve the performance of massive real-time data processing/analytics. developers are able to use this library to design their cache-less and SerDe-less high performance applications.

Apache Mnemonic -  Non-volatile hybrid memory storage oriented library

Apache Tez is an extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop. Tez improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce’s ability to scale to petabytes of data. Important Hadoop ecosystem projects like Apache Hive and Apache Pig use Apache Tez, as do a growing number of third party data access applications developed for the broader Hadoop ecosystem.
<BR><BR>
Tez can execute complex directed acyclic graphs of general data processing tasks. In many ways it can be thought of as a more flexible and powerful successor of the map-reduce framework. By allowing projects like Apache Hive and Apache Pig to run a complex DAG of tasks, Tez can be used to process data, that earlier took multiple MR jobs, now it can be done in a single Tez job. 

Apache Tez is an extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop. Tez improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce’s ability to scale to petabytes of data. Important Hadoop ecosystem projects like Apache Hive and Apache Pig use Apache Tez, as do a growing number of third party data access applications developed for the broader Hadoop ecosystem.

Apache Tez - A Framework for YARN-based, Data Processing Applications In Hadoop

Pangool is a framework on top of Hadoop that implements Tuple MapReduce. It supports secondary sorting, Built-in reduce-side joining capabilities, Built-in serialization support for Thrift and ProtoStuff and lot more.

Pangool - Tuple MapReduce for Hadoop and MapReduce made easy

Discover open source projects across all platforms

Projects

Hadoop Common

Hue - The open source Apache Hadoop UI

Apache Mnemonic - Non-volatile hybrid memory storage oriented library

Apache Tez - A Framework for YARN-based, Data Processing Applications In Hadoop

Pangool - Tuple MapReduce for Hadoop and MapReduce made easy

Tagcloud

License

Suggested keywords:

Projects

Hadoop Common

Hue - The open source Apache Hadoop UI

Apache Mnemonic - Non-volatile hybrid memory storage oriented library

Apache Tez - A Framework for YARN-based, Data Processing Applications In Hadoop

Pangool - Tuple MapReduce for Hadoop and MapReduce made easy

Tagcloud

License