Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

Kafka - A high-throughput distributed messaging system

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Flume - Log management using HDFS

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Data flow can be tracked and modified at run time.  It automates the movement of data between disparate data sources and systems, making data ingestion fast, easy and secure. The project was created by the United States National Security Agency (NSA).
<BR>
It is data source agnostic, supporting disparate and distributed sources of differing formats, schemas, protocols, speeds and sizes such as machines, geo location devices, click streams, files, social feeds, log files and videos and more.

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Data flow can be tracked and modified at run time.  It automates the movement of data between disparate data sources and systems, making data ingestion fast, easy and secure. The project was created by the United States National Security Agency (NSA).

Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data

Chainsaw is a companion application to Log4j written by members of the Log4j development community. Chainsaw can read log files formatted in Log4j's XMLLayout, receive events from remote locations, read events from a DB, it can even work with the JDK 1.4 logging events.<BR><BR>

  It has dynamic and powerful filterting which helps to locate stuff easier(e.g. "LOGGER == 'com.mycompany' &amp;&amp; LEVEL == ERROR" ).

Chainsaw is a companion application to Log4j written by members of the Log4j development community. Chainsaw can read log files formatted in Log4j's XMLLayout, receive events from remote locations, read events from a DB, it can even work with the JDK 1.4 logging events.

Chainsaw - log viewer and analysis tool

White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

White-elephant - Hadoop log aggregator and dashboard

Discover open source projects across all platforms

Projects

Kafka - A high-throughput distributed messaging system

Flume - Log management using HDFS

Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data

Chainsaw - log viewer and analysis tool

White-elephant - Hadoop log aggregator and dashboard

Tagcloud

Suggested keywords:

Projects

Kafka - A high-throughput distributed messaging system

Flume - Log management using HDFS

Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data

Chainsaw - log viewer and analysis tool

White-elephant - Hadoop log aggregator and dashboard

Tagcloud