Azkaban documentation!

What is Azkaban

Azkaban is a distributed Workflow Manager, implemented at LinkedIn to solve the problem of Hadoop job dependencies. We had jobs that needed to run in order, from ETL jobs to data analytics products.

See the Getting Started for more details.

Features

• Distributed Multiple Executor
• MySQL Retry
• Friendly UI
• Conditional Workflow
• Data Triggers
• High Security
• Support plug-in extensions, from Web UI to job Execution
• Full Authorship management system

Contents:

• Getting Started
  • Building from Source
  • Getting started with the Solo Server
  • Getting started with the Multi Executor Server
  • Set up Azkaban Plugins
  • Property Overrides
  • Upgrading DB from 2.1
  • Upgrading DB from 2.7.0
• Configurations
  • Azkaban Web Server Configurations
  • Azkaban Executor Server Configuration
  • Plugin Configurations
• UserManager
  • XmlUserManager
  • Custom User Manager
• Creating Flows
  • Flow 2.0 Basics
  • Job Dependencies
  • Job Config
  • Flow Config
  • Embedded Flows
  • Download Examples
• Using Azkaban
  • Create Projects
  • Upload Projects
  • Flow View
  • Project Permissions
  • Executing Flow
  • Executions
  • Schedule Flow
  • Job Page
  • Job Details
• Conditional Workflow
  • What is Azkaban Conditional Workflow?
  • How to define a condition?
  • View condition from Azkaban web UI
  • Download Examples
• Flow Trigger Dependency Plugin
  • Kafka Event Based trigger
  • Getting Started with Deploying Event Trigger on the Azkaban
  • Event Based Trigger Plugin Configuration
  • Event Trigger Instance Configuration
  • Event Based Trigger Example with Azkaban UI
  • Limitation
• API Documentation
  • Authenticate
  • Fetch user Projects
  • Create a Project
  • Delete a Project
  • Upload a Project Zip
  • Fetch Flows of a Project
  • Fetch Jobs of a Flow
  • Fetch Executions of a Flow
  • Fetch Running Executions of a Flow
  • Execute a Flow
  • Cancel a Flow Execution
  • Schedule a period-based Flow (Deprecated)
  • Flexible scheduling using Cron
  • Fetch a Schedule
  • Unschedule a Flow
  • Set a SLA
  • Fetch a SLA
  • Pause a Flow Execution
  • Resume a Flow Execution
  • Fetch a Flow Execution
  • Fetch Execution Job Logs
  • Fetch Flow Execution Updates
  • Fetch Logs of a Project
  • Image Management APIs
• How Tos
  • Force execution to an executor
  • Setting flow priority in multiple executor mode
  • Enabling and Disabling Queue in multiple executor mode
  • Reloading executors in multiple executor mode
  • Logging logs to a Kafka cluster
• Plugins
  • HadoopSecurityManager
  • Azkaban HDFS Browser
  • JobType Plugins
• Jobtypes
  • Command Job Type (built-in)
  • HadoopShell Job Type
  • Java Job Type
  • hadoopJava Type
  • Pig Type
  • Hive Type
  • Common Configurations
  • VoldemortBuildandPush Type
  • Create Your Own Jobtypes
  • System Statistics
  • Reload Jobtypes
  • Examples

Community

Users and development team are in the Gitter.

Developers interested in getting more involved with Azkaban may join the mailing lists mailing lists, report bugs, and make contributions.