Skip to content

fix: bump DataFusion to 54.1.0 and adapt to RecursiveQuery API change#1645

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:bump-datafusion-54.1.0
Open

fix: bump DataFusion to 54.1.0 and adapt to RecursiveQuery API change#1645
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:bump-datafusion-54.1.0

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #1644.

Rationale for this change

The workspace dependencies were declared as version = "54", so Cargo was free
to resolve them to DataFusion 54.1.0 — but the crate no longer compiled when it
did. DataFusion 54.1.0 added a schema: DFSchemaRef field to
datafusion_expr::logical_plan::RecursiveQuery, and PyRecursiveQuery::new()
built that struct with a literal initializer:

error[E0063]: missing field `schema` in initializer of `datafusion::logical_expr::RecursiveQuery`
  --> crates/core/src/expr/recursive_query.rs:72:20

What changes are included in this PR?

  • Bump the workspace DataFusion requirements from "54" to "54.1" and update
    Cargo.lock to 54.1.0. Pinning the minor version keeps the build from silently
    resolving back to 54.0.0, which no longer satisfies the code below.
  • Build RecursiveQuery through the new RecursiveQuery::try_new() constructor
    instead of a struct literal. try_new() computes the output schema by
    reconciling the static and recursive terms, so it is fallible;
    PyRecursiveQuery::new() now returns PyDataFusionResult<Self>.
  • Extend test_recursive_query to round-trip the node back through the Python
    constructor, covering the new schema-reconciliation path.

Are there any user-facing changes?

datafusion.expr.RecursiveQuery(...) can now raise when the static and
recursive terms do not have the same number of columns. Previously such
arguments produced a RecursiveQuery node that was invalid downstream, so this
turns a late failure into an immediate one. No other public API changes.

DataFusion 54.1.0 added a `schema` field to `RecursiveQuery`, breaking the
struct literal in `PyRecursiveQuery::new()`. Build the node through the new
fallible `RecursiveQuery::try_new()` constructor, which reconciles the schema
of the static and recursive terms.

Pin the workspace requirements to `54.1` so the build cannot resolve back to
54.0.0, which no longer satisfies this code.
@andygrove

Copy link
Copy Markdown
Member Author

@timsaucer for context, Ballista Rust crates upgraded to 54.1.0 to bring in a needed correctness fix, but the Ballista Python module could not upgrade due to the breaking API change in DF 54.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build fails against DataFusion 54.1.0 due to new schema field on RecursiveQuery

1 participant