Skip to content

gsoc26: Declarative Workflow Engine #79

Description

@DhanashreePetare

Feature Request

Describe the feature you'd like

  • A YAML-based pipeline system that chains download, deploy, and delete steps into a single sequential workflow, run via databusclient workflow run pipeline.yml. Each step's output can be referenced by later steps using ${steps.step_name.output_files} syntax, resolved at runtime.
  • Each step declares its own on_error behavior: fail, continue, or retry (with configurable max_attempts/delay_seconds).
  • Environment variables (${VAR_NAME}) are substituted at parse time, with a clear early error if a referenced variable is missing.

Why is this feature important?
Today, running a multi-step Databus operation (e.g. download → deploy → delete old version) requires manually chaining CLI commands, with no structured record of what was run and no way to reuse a pipeline definition. This closes that gap, building on the existing stateless download()/deploy()/delete() functions without modifying their internals.

Additional context
Depends on the manifest system (#64) and replay (#65) already being in place, since a workflow run reuses the same underlying API functions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions