Oozie工作流框架使用指南.pptx

工作流框架使用指南 2013年10月 System Installation Job Design Oozie Overview WorkFlow Design Coordinator Design Reference Contents Requirements Cloudera Manager CDH Oozie Install Config Hue Install Config (2 items) Basic Operation System Installation Action Overview An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or action node. Job Designer Job Design Example Fs Action MapReduce Action Job Design Oozie A workflow scheduler system to manage Apache Hadoop jobs WorkFlow Workflow jobs are Directed Acyclical Graphs (DAGs) of actions Coordinator Coordinator jobs are recurrent workflow jobs triggered by time (frequency) and data availabilty. Integrated? MR\FS\Email\Shell\Ssh\Hive\Pig\Sqoop\Distcp\Java Scalable Reliable Extensible Oozie Overview Does not support circles Workflow Nodes Control Flow Nodes Workflow Action Nodes WorkFlow Recovery oozie.wf.rerun.failnodes oozie.wf.rerun.skip.nodes WorkFlow Design Workflow definitions can be parameterized When workflow node is executed by Oozie all the ELs are resolved into concrete values EL expressions can be used in the configuration values of action and decision nodes Workflow Job Properties (or Parameters) Expression Language Functions Parameterization of Workflows Example Start Map Reduce1 输出500行 Map Reduce2 End 是 Coordinator execute workflow jobs： Recurrent Interdependent Coordinator? Based： Time intervals Data availability Time intervals and/or data availability Coordinator Overview Coordinator Archtechture Datetime, Frequency and Time-Period Coordinator Action Parameterization of Coordinator Coordinator Design Coordinator Design 定期创建文件夹 开始时间： 结束时间： 频率：每3分钟一次 文件夹名字为标定时间格式化yyyy-MM-ddTHH-mm Example Oozie Home page Documents Hue Home page Tutorials Expression Language Tutorials Reference