nxlog is a modular, multi-threaded, high-performance log management solution with multi-platform support. In concept it is similar to syslog-ng or rsyslog but is not limited to unix/syslog only. It can collect logs from files in various formats, receive logs from the network remotely over UDP, TCP or TLS/SSL . It supports platform specific sources such as the Windows Eventlog, Linux kernel logs, Android logs, local syslog etc. 
 Writing and reading logs to/from databases is also supported. The collected logs can be stored into files, databases or forwarded to a remote log server using various protocols. The old BSD Syslog and the newer IETF syslog standard is fully supported by nxlog in addition to other custom formats. A key concept in nxlog is to be able to handle and preserve structured logs so there is no need to convert everything to syslog and then parse these logs again at the other side. It has powerful message filtering, log rewrite and conversion capabilities.

nxlog is a modular, multi-threaded, high-performance log management solution with multi-platform support. In concept it is similar to syslog-ng or rsyslog but is not limited to unix/syslog only. It can collect logs from files in various formats, receive logs from the network remotely over UDP, TCP or TLS/SSL . It supports platform specific sources such as the Windows Eventlog, Linux kernel logs, Android logs, local syslog etc.

nxlog - Multi platform Log management

Graylog2 is an open source log management solution that stores your logs in ElasticSearch. It consists of a server written in Java that accepts your syslog messages via TCP, UDP or AMQP and stores it in the database. The second part is a web interface that allows you to manage the log messages from your web browser. Take a look at the screenshots or the latest release info page to get a feeling of what you can do with Graylog2. 

Graylog2 - Open Source Log Management

Sentry is a realtime event logging and aggregation platform. It specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.

Sentry - Realtime Platform-Agnostic Error Logging and Aggregation platform

Quickwit is a distributed search engine built from the ground up to offer cost-efficiency and high reliability. By mere mortals for mere mortals, Quickwit's architecture is as simple as possible. Quickwit is written in Rust and built on top of the mighty tantivy library. It designed to index big datasets straight from object storage like AWS S3 in a stateless manner. 
<ul>
<li>Scalable distributed search: Host an arbitrary number of indexes on Amazon S3 and answer search queries in less than a second with a small pool of stateless search instances</li>
<li>Stream indexing: Ingest TB of data from your favorite distributed event streaming service. As of today, Quickwit supports Apache Kafka natively. The next releases will bring support for more platforms</li>
<li>Fault-tolerant architecture that won't lose data: Quickwit achieves exactly-once processing for indexing and safely stores your data on highly reliable object storage services such as Amazon S3</li>
<li>Cloud-native, easy to operate: Thanks to true decoupled compute and storage, search instances are stateless, add or remove search nodes within seconds</li>
<li>Sub-second full-text search on cloud / distributed storage: Quickwit Search re-designed indexing and index data structure to open it in less than 60ms on Amazon S3</li>
<li>Time-based sharding: Quickwit shards data by time when enabled. And you can use a second dimension to shard data thanks to our <a href="../../docs/design/querying">tags feature</a>. Time-based queries only access splits (a data piece of the index) that match the time range of the query which leads to significant performance improvements</li>
<li>Painless multi-tenant search: Create indexes for each tenant without hurting query performance. Or group tenants into one index and use tagging to prune irrelevant splits for your tenant query to improve significantly performance</li>
<li>Elasticsearch query language support</li>
<li>Decoupled compute and storage, stateless indexers &amp; searchers</li>
<li>Schemaless analytics</li>
</ul>

Quickwit is a distributed search engine built from the ground up to offer cost-efficiency and high reliability. By mere mortals for mere mortals, Quickwit's architecture is as simple as possible. Quickwit is written in Rust and built on top of the mighty tantivy library. It designed to index big datasets straight from object storage like AWS S3 in a stateless manner.

Quickwit - Fast and cost-efficient distributed search engine for immutable data

Octopussy is a Log analyzer tool. It analyzes the log, generates reports and alerts the admin. It has LDAP support to maintain users list. It exports report by Email, FTP & SCP. Scheduled reports could be generated. RRD tool to generate graphs. 

 It is capable to parse the logs generated from lots of services such as Bind, Cisco Router, Cisco Switch, DenyAll Reverse Proxy, Drbd, F5 BigIP, Fortinet FW, Ironport MailServer, Linux Kernel/System, Linux IPTables, Monit, MySQL, Nagios, NetApp NetCache, Juniper Netscreen FW, Juniper Netscreen NSM, Postfix, PostgreSQL, Samhain, Snmpd, Squid, Sshd, Syslog-ng, Windows Snare Agent, Xen etc.

Octopussy is a Log analyzer tool. It analyzes the log, generates reports and alerts the admin. It has LDAP support to maintain users list. It exports report by Email, FTP & SCP. Scheduled reports could be generated. RRD tool to generate graphs.

Octopussy - Perl/XML Logs Analyzer, Alerter & Reporter

Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to Hadoop.

Kafka - A high-throughput distributed messaging system

Security Onion is a Linux distro for intrusion detection, network security monitoring, and log management. It’s based on Ubuntu and contains Snort, Suricata, Bro, OSSEC, Sguil, Squert, ELSA, Xplico, NetworkMiner, and many other security tools. Security Onion seamlessly weaves together three core functions: full packet capture, network-based and host-based intrusion detection systems, powerful analysis tools.

Security Onion - Linux distro for intrusion detection, network security and log management

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Flume - Log management using HDFS

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Data flow can be tracked and modified at run time. It automates the movement of data between disparate data sources and systems, making data ingestion fast, easy and secure. The project was created by the United States National Security Agency (NSA).
 
It is data source agnostic, supporting disparate and distributed sources of differing formats, schemas, protocols, speeds and sizes such as machines, geo location devices, click streams, files, social feeds, log files and videos and more.

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Data flow can be tracked and modified at run time.  It automates the movement of data between disparate data sources and systems, making data ingestion fast, easy and secure. The project was created by the United States National Security Agency (NSA).

Apache NiFi - An easy to use, powerful, and reliable system to process and distribute data

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through your browser. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly. It supports nearly all web log formats (Apache, Nginx, Amazon S3, Elastic Load Balancing, CloudFront, etc) 

GoAccess - Real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. Fluentd can be used to tail access/error logs and transport them reliably to remote systems. It can "grep" for events and send out alerts. It can function as middleware to enable asynchronous, scalable logging for user action events.
 
Fluentd tries to structure data as JSON as much as possible: this allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations (Unified Logging Layer). The downstream data processing is much easier with JSON, since it has enough structure to be accessible while retaining flexible schemas.

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. Fluentd can be used to tail access/error logs and transport them reliably to remote systems. It can "grep" for events and send out alerts. It can function as middleware to enable asynchronous, scalable logging for user action events.

Fluentd - Unified Logging Layer

Chainsaw is a companion application to Log4j written by members of the Log4j development community. Chainsaw can read log files formatted in Log4j's XMLLayout, receive events from remote locations, read events from a DB, it can even work with the JDK 1.4 logging events. 

 It has dynamic and powerful filterting which helps to locate stuff easier(e.g. "LOGGER == 'com.mycompany' &amp;&amp; LEVEL == ERROR" ).

Chainsaw is a companion application to Log4j written by members of the Log4j development community. Chainsaw can read log files formatted in Log4j's XMLLayout, receive events from remote locations, read events from a DB, it can even work with the JDK 1.4 logging events.

Chainsaw - log viewer and analysis tool

Clarity is a Splunk like web interface for your server log files. It supports searching (using grep) as well as trailing log files in realtime. It has been written using the event based architecture based on EventMachine and so allows real-time search of very large log files.

Clarity - Web interface for the grep

Logsandra is a log management application written in Python and using Cassandra as back-end. It is written as demo for cassandra but it is worth to take a look. It provides support to create your own parser. Logsandra consists of two main scripts logsandra-httpd.py and logsandra-monitord.py. logsandra-httpd should run on one server and logsandra-monitord should run on all servers producing log files. Logsandra-monitord will send the data to the specified Cassandra Cluster.

Logsandra is a log management application written in Python and using Cassandra as back-end. It is written as demo for cassandra but it is worth to take a look. It provides support to create your own parser.

Logsandra - log management using Cassandra

White Elephant is a Hadoop log aggregator and dashboard which enables visualization of Hadoop cluster utilization across users. The server is a JRuby web application. In a production environment it can be deployed to tomcat and reads aggregated usage data directly from Hadoop. This data is stored in an in-memory database provided by HyperSQL. Charting is provided by Rickshaw. This project is developed by LinkedIn.

White-elephant - Hadoop log aggregator and dashboard

Discover open source projects across all platforms

Projects