Ray Workflow - Durable Ray tasks for efficient, flexible data pipelines

Описание: Ray Workflow - Durable Ray tasks for efficient, flexible data pipelines

Traditionally, a workflow consists of a pipeline of tasks, executed and automated according to a set of procedural rules. Workflows enable coordinating and monitoring among distributed people, organizations, and tasks with strong durability, observability, and repeatability.

Recently there is a growing trend of workflow-as-code for applications pipelines in favor of these properties, represented by workflow systems like AirFlow, Prefect, Temporal, and so on. However, many workflows today are data workflows: they are application pipelines that may pass and process large amounts of data between steps. Examples include ETL workloads and ML pipelines. These aforementioned workflow systems are less efficient and flexible for data processing, while Ray offers both efficiency and flexibility for data-intensive workloads. Combining the advantages of both Ray and a workflow system, we show that efficiency, durability, and flexibility can be achieved simultaneously on data pipelines with durable Ray tasks via Ray Workflow.

This talk gives an introduction to Ray Workflow, how you can use Ray Workflow as durable Ray tasks, and how to program data pipelines with Ray Workflow. Ray Workflow will be available as alpha in Ray 2.0.

See all Ray Summit content @ http://anyscale.com/ray-summit-2022