Gizmo Microservice Toolkit toolkit provides packages to put together server and pubsub daemons with the following features like Standardized configuration and logging, Health check endpoints with configurable strategies, Configuration for managing pprof endpoints and log levels, Structured logging containing basic request information, Useful metrics for endpoints, Graceful shutdowns, Basic interfaces to define our expectations and vocabulary and lot more.

Gizmo - A Microservice Toolkit from The New York Times

The Oryx open source project provides infrastructure for lambda-architecture applications on top of Spark, Spark Streaming and Kafka. On this, it provides further support for real-time, large scale machine learning, and end-to-end applications of this support for common machine learning use cases, like recommendations, clustering, classification and regression.
  

Oryx 2 - Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

A tool for managing Apache Kafka. It manages multiple clusters, Inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution), Run preferred replica election, Add partitions to existing topic, Run reassignment of partition (based on generated assignments) and lot more.

Kafka Manager - A tool for managing Apache Kafka

The Alpakka project is an open source initiative to implement stream-aware, reactive, integration pipelines for Java and Scala. It is built on top of Akka Streams, and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for backpressure. Akka Streams is a Reactive Streams and JDK 9+ java.util.concurrent.Flow-compliant implementation and therefore fully interoperable with other implementations.

Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka

Plumber is a CLI devtool for inspecting, piping, massaging and redirecting data in message systems like Kafka, RabbitMQ , GCP PubSub and many more. It helps to safely view the contents of your data streams, Decode protobuf/avro/thrift/JSON data in real-time, Route data from one place to another, Replay events into a message system on your local network and lot more.


Plumber - A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems

Kafka UI is a UI for Apache Kafka to monitor and manage Apache Kafka clusters. It is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters - Brokers, Topics, Partitions, Production, and Consumption. 
Its features include: 
<ul>
<li>Multi-Cluster Management — monitor and manage all your clusters in one place</li>
<li>Performance Monitoring with Metrics Dashboard — track key Kafka metrics with a lightweight dashboard</li>
<li>View Kafka Brokers — view topic and partition assignments, controller status</li>
<li>View Kafka Topics — view partition count, replication status, and custom configuration</li>
<li>View Consumer Groups — view per-partition parked offsets, combined and per-partition lag</li>
<li>Browse Messages — browse messages with JSON, plain text, and Avro encoding</li>
<li>Dynamic Topic Configuration — create and configure new topics with dynamic configuration</li>
<li>Configurable Authentification — secure your installation with optional Github/Gitlab/Google OAuth 2.0</li>
</ul>

Kafka UI is a UI for Apache Kafka to monitor and manage Apache Kafka clusters. It is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters - Brokers, Topics, Partitions, Production, and Consumption.

Kafka UI - Open-Source Web UI for Apache Kafka Management

Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). It includes a high-level API for easily producing and consuming messages, and a low-level API for controlling bytes on the wire when the high-level API is insufficient.


 Sarama - Go library for Apache Kafka 0.8, 0.9, and 0.10.

Snowplow is an enterprise-strength marketing and product analytics platform. It identifies your users, and tracks the way they engage with your website or application. It stores your users' behavioural data in a scalable "event data warehouse" you control: in Amazon S3 and (optionally) Amazon Redshift or Postgres. Lets you leverage the biggest range of tools to analyze that data, including big data tools (e.g. Spark) via EMR or more traditional tools e.g. Looker, Mode, Superset, Re:dash to analyze that behavioural data.

Snowplow - Cloud-native web, mobile and event analytics, running on AWS and on-premise with Kafka

ThingsBoard is an open-source IoT platform for data collection, processing, visualization, and device management. It enables device connectivity via industry standard IoT protocols - MQTT, CoAP and HTTP and supports both cloud and on-premises deployments. ThingsBoard combines scalability, fault-tolerance and performance so you will never lose your data.
 
It helps to provision and manage devices and assets, Collect and visualize data, Raise alarms on incoming telemetry events, Construct your ThingsBoard cluster and get maximum scalability and fault-tolerance with microservices architecture. Construct your ThingsBoard cluster and get maximum scalability and fault-tolerance with microservices architecture.

ThingsBoard is an open-source IoT platform for data collection, processing, visualization, and device management. It enables device connectivity via industry standard IoT protocols - MQTT, CoAP and HTTP and supports both cloud and on-premises deployments. ThingsBoard combines scalability, fault-tolerance and performance so you will never lose your data.

ThingsBoard - IoT platform for data collection, processing, visualization, and device management

Redpanda is a streaming platform for mission critical workloads. It is Kafka API compatible so any client that works with Kafka should work out of the box with Redpanda. It does not have dependency on ZooKeeper, JVM. Redpanda is a new storage engine, optimized for streaming data, using a thread-per-core architecture focused on delivering stable tail-latencies. It transforms data inline with embedded WASM engine. 
 
Its features include:
<ul>
<li>Kafka API compatible&nbsp;</li>
<li>Up to 4x faster Kafka transactions</li>
<li>Automatic data and load rebalancing</li>
<li>Automatic hardware optimization</li>
<li>Unlimited partitions for extreme flexibility &nbsp; &nbsp;</li>
<li>Built-in HTTP support &nbsp; &nbsp;</li>
<li>Support for favorite Kafka tooling &nbsp; &nbsp;</li>
<li>Built-in Prometheus support &nbsp; &nbsp;</li>
<li>Native K8s operator</li>
<li>Automatic leadership rebalancing &nbsp; &nbsp;</li>
<li>TLS encryption in flight</li>
<li>Unlimited horizontal scaling &nbsp; &nbsp;</li>
</ul>

Redpanda is a streaming platform for mission critical workloads. It is Kafka API compatible so any client that works with Kafka should work out of the box with Redpanda. It does not have dependency on ZooKeeper, JVM. Redpanda is a new storage engine, optimized for streaming data, using a thread-per-core architecture focused on delivering stable tail-latencies. It transforms data inline with embedded WASM engine. 

Redpanda - A Kafka API compatible streaming platform for mission-critical workloads

Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.

Kafdrop – Kafka Web UI

Ockam provides end-to-end encryption and mutual authentication for distributed applications. Transport layer security protocols are unable to protect application messages because their protection is constrained by the length and duration of the underlying transport connection. Ockam is a suite of programming libraries and infrastructure that makes it simple for our applications to guarantee end-to-end integrity, authenticity, and confidentiality of data.
 
It helps to create and manage cryptographic keys, unique device identifiers, and verifiable credentials in your connected environments.

Ockam provides end-to-end encryption and mutual authentication for distributed applications. Transport layer security protocols are unable to protect application messages because their protection is constrained by the length and duration of the underlying transport connection. Ockam is a suite of programming libraries and infrastructure that makes it simple for our applications to guarantee end-to-end integrity, authenticity, and confidentiality of data.

Ockam - End-to-end encryption and mutual authentication for distributed applications

Debezium is a distributed platform that turns your existing databases into event streams, so applications can see and respond immediately to each row-level change in the databases. Debezium is built on top of Apache Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, from where your application consumes them. This makes it possible for your application to easily consume all of the events correctly and completely.

Debezium - Stream changes from your databases.

KafkaJS is a modern Apache Kafka client for Node.js. It is compatible with Kafka 0.10+ and offers native support for 0.11 features. It has support for Producer, Consumer groups with pause, resume, and seek, GZIP compression, Admin client.



kafkajs - A modern Apache Kafka client for node.js

Tremor is an event processing system. It was originally designed as a replacement for software such as Logstash or Telegraf. However tremor has outgrown this singular use case by supporting more complex workflows such as aggregation, rollups, an ETL language, and a query language. 
 
Tremor has been successfully used to replace logstash as a Kafka to Elastic Search ingress. In this scenario, it reduced the required compute resources by about 80% (YMMV) when decoding, classify, and rate-limiting the traffic. A secondary but perhaps more important effect was that tremors dynamic backpressure and rate-limiting allowed the ElasticSearch system to stay healthy and current despite overwhelming amounts of logs during spikes.

Tremor is an event processing system. It was originally designed as a replacement for software such as Logstash or Telegraf. However tremor has outgrown this singular use case by supporting more complex workflows such as aggregation, rollups, an ETL language, and a query language. 

Tremor - An early-stage event processing system for unstructured data with rich support for structural pattern-matching, filtering and transformation

Discover open source projects across all platforms

Projects