Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.
 
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
 
Airflow pipelines are defined in Python, allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.
 
There are other software packages that focus on lower level aspects of data processing, like Hive, Pig, or Cascading. Luigi is not a framework to replace these. Instead it helps you stitch many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python snippet, dumping a table from a database, or anything else. It's easy to build up long-running pipelines that comprise thousands of tasks and take days or weeks to complete. Luigi takes care of a lot of the workflow management so that you can focus on the tasks themselves and their dependencies.
 
Luigi internally used at <a href="https://www.spotify.com">Spotify</a> to run
thousands of tasks every day, organized in complex dependency graphs.
Most of these tasks are Hadoop jobs. Luigi provides an infrastructure
that powers all kinds of stuff including recommendations, toplists, A/B
test analysis, external reports, internal dashboards, etc

The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

Luigi - Python module that helps you build complex pipelines of batch jobs

Camunda is an open source platform for workflow and business process management. You can model and execute BPMN 2.0, CMMN 1.1 and DMN 1.1. Camunda's core is a rock-solid, freaking fast execution engine that is horizontally scalable and comes with helpful web applications.
 
camunda BPM platform is a flexible framework for workflow and process automation. It's core is a native BPMN 2.0 process engine that runs inside the Java Virtual Machine. It can be embedded inside any Java application and any Runtime Container. It integrates with Java EE 6 and is a perfect match for the Spring Framework. On top of the process engine, you can choose from a stack of tools for human workflow management, operations & monitoring.

Camunda is an open source platform for workflow and business process management. You can model and execute BPMN 2.0, CMMN 1.1 and DMN 1.1. Camunda's core is a rock-solid, freaking fast execution engine that is horizontally scalable and comes with helpful web applications.

Camunda -  Platform for Workflow and Business Process Management

Rundeck is an open source automation service with a web console, command line tools and a WebAPI. It lets you easily run automation tasks across a set of nodes.  It can schedule a job, more secure replacement of cron. It provides support for runbook automation where it can execute common set of reusable procedures.

Rundeck - Job Scheduler and Runbook Automation

Obsei is a low code AI powered automation tool. It can be used in various business flows like social listening, AI based alerting, brand image analysis, comparative study and more. 
It consist of Observer, Analyzer and Informer. Observer observes the platform like Twitter, Facebook, App Stores, Google reviews, Amazon reviews, News, Website etc and feed that information. Analyzer performs text analysis like classification, sentiment, translation, PII on the analyzed data. Informer sends it to ticketing system, data store, dataframe etc for further action and analysis.

Obsei - Low code AI powered automation tool

Shuttle is a website allowing for the automatic extraction and reintegration of localizable strings in a code base. It also provides an API where articles can be submitted for translation and retrieved. In addition, it provides a workflow for translators and reviewers optimized for the efficient processing of many strings. Finally, for project managers, it provides a dashboard allowing them to view and manage the progress of a localization effort. Shuttle can be thought of as a Continuous Integration system for translations.
 
Typical workflow: 
<ol>
<li>An engineer makes a commit to a Project, and marks that commit as requiring
localization in Shuttle.</li>
<li>Shuttle scans the commit for any localizable strings using its {Importer}s.</li>
<li>New or modified localizable strings are marked as pending translation. They
appear on the translators’ dashboard.</li>
<li>Translators translate all these strings. They then appear on the reviewers’
dashboard.</li>
<li>Reviewers review and approve translations.</li>
<li>Once all translations applying to a commit are approved for all of a
Project's required localizations, the Commit is marked as ready.</li>
<li>When the commit is deployed, Shuttle provides a manifest of translated
strings that is downloaded as part of the deploy artifact.</li>
</ol>

Shuttle is a website allowing for the automatic extraction and reintegration of localizable strings in a code base. It also provides an API where articles can be submitted for translation and retrieved. In addition, it provides a workflow for translators and reviewers optimized for the efficient processing of many strings. Finally, for project managers, it provides a dashboard allowing them to view and manage the progress of a localization effort. Shuttle can be thought of as a Continuous Integration system for translations.

Shuttle - String extraction, translation and export tools for the 21st century

Jesque is an implementation of Resque in Java. It is fully-interoperable with the Ruby and Node.js (Coffee-Resque) implementations.

Jesque - Job queue in Java

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows. Azkaban was designed primarily with usability in mind. It has been running at LinkedIn for several years, and drives many of their Hadoop and data warehouse processes.
 
Its features include: 
<ul><li><li>Compatible with any version of Hadoop</li><li>Easy to use web UI</li><li>Simple web and http workflow uploads</li><li>Project workspaces</li><li>Scheduling of workflows</li><li>Modular and pluginable</li><li>Authentication and Authorization</li><li>Tracking of user actions</li><li>Email alerts on failure and successes</li><li>SLA alerting and auto killing</li><li>Retrying of failed jobs</li></li></ul>

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows. Azkaban was designed primarily with usability in mind. It has been running at LinkedIn for several years, and drives many of their Hadoop and data warehouse processes.

Azkaban - Batch workflow job scheduler created at LinkedIn to run Hadoop jobs

Oozie is a workflow scheduler system to manage Apache Hadoop jobs. Its jobs are Directed Acyclical Graphs (DAGs) of actions and triggered by time (frequency) and data availability. Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp) as well as system specific jobs (such as Java programs and shell scripts).

Apache Oozie - Workflow Scheduler for Hadoop

Stateless4j helps to create state machines and lightweight state machine-based workflows directly in java code. It supports Hierarchical states, Entry/exit events for states, Guard clauses to support conditional transitions, Reentrant states and lot more.

Stateless4j -  Lightweight Java State Machine

Activiti is a light-weight workflow and Business Process Management (BPM) Platform targeted at business people, developers and system admins. Its core is a super-fast and rock-solid BPMN 2 process engine for Java. It's open-source and distributed under the Apache license. Activiti runs in any Java application, on a server, on a cluster or in the cloud. It integrates perfectly with Spring, it is extremely lightweight and based on simple concepts.

Activiti - BPMN 2.0 Platform

Flowable is a compact and highly efficient workflow and Business Process Management (BPM) platform for developers, system admins and business users. It is lightning fast, tried and tested BPMN 2 process engine. It can run embedded in a Java application, or as a service on a server, a cluster, and in the cloud. It integrates perfectly with Spring. With a rich Java and REST API, it is the ideal engine for orchestrating human or system activities.

Flowable - The Java BPM Engine

Discover open source projects across all platforms

Projects