The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.
 
There are other software packages that focus on lower level aspects of data processing, like Hive, Pig, or Cascading. Luigi is not a framework to replace these. Instead it helps you stitch many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python snippet, dumping a table from a database, or anything else. It's easy to build up long-running pipelines that comprise thousands of tasks and take days or weeks to complete. Luigi takes care of a lot of the workflow management so that you can focus on the tasks themselves and their dependencies.
 
Luigi internally used at <a href="https://www.spotify.com">Spotify</a> to run
thousands of tasks every day, organized in complex dependency graphs.
Most of these tasks are Hadoop jobs. Luigi provides an infrastructure
that powers all kinds of stuff including recommendations, toplists, A/B
test analysis, external reports, internal dashboards, etc

The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

Luigi - Python module that helps you build complex pipelines of batch jobs

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows. Azkaban was designed primarily with usability in mind. It has been running at LinkedIn for several years, and drives many of their Hadoop and data warehouse processes.
 
Its features include: 
<ul><li><li>Compatible with any version of Hadoop</li><li>Easy to use web UI</li><li>Simple web and http workflow uploads</li><li>Project workspaces</li><li>Scheduling of workflows</li><li>Modular and pluginable</li><li>Authentication and Authorization</li><li>Tracking of user actions</li><li>Email alerts on failure and successes</li><li>SLA alerting and auto killing</li><li>Retrying of failed jobs</li></li></ul>

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows. Azkaban was designed primarily with usability in mind. It has been running at LinkedIn for several years, and drives many of their Hadoop and data warehouse processes.

Azkaban - Batch workflow job scheduler created at LinkedIn to run Hadoop jobs

Discover open source projects across all platforms

Projects

Luigi - Python module that helps you build complex pipelines of batch jobs

Azkaban - Batch workflow job scheduler created at LinkedIn to run Hadoop jobs

TechStack

Tagcloud

Suggested keywords:

Projects

Luigi - Python module that helps you build complex pipelines of batch jobs

Azkaban - Batch workflow job scheduler created at LinkedIn to run Hadoop jobs

TechStack

Tagcloud