datafusion: support REST format table partition commands

[sundapeng]

Summary

Add Spark-style partition administration to DataFusion for catalog-managed internal Format Tables loaded from REST Catalog. REST partition registrations become the authoritative partition set used by both SQL commands and scans.

Changes

• Add REST Catalog APIs for paginated partition listing and create/drop registration, including retry-safe 1000-entry batching for idempotent requests.
• Load eligible internal Format Tables with immutable REST partition metadata and scan only registered partitions.
• Add DataFusion support for:
  • SHOW PARTITIONS table [PARTITION (...)]
  • ALTER TABLE table ADD [IF NOT EXISTS] PARTITION (...)
  • ALTER TABLE table DROP [IF EXISTS] PARTITION (...)
  • MSCK REPAIR TABLE table [{ADD|DROP|SYNC} PARTITIONS]
• Normalize typed partition literals, default partition values, and Java-compatible partition paths.
• Keep repair metadata-only and fail closed before destructive DROP/SYNC reconciliation.

Scope and limitations

• Applies only to partitioned internal REST Format Tables with metastore.partitioned-table=true and a non-engine implementation.
• DROP PARTITION currently accepts one complete partition specification per statement.
• Only MSCK REPAIR TABLE is supported; bare REPAIR TABLE is rejected.
• Public API changes are additive; the storage format is unchanged.

Testing

• cargo fmt --all -- --check
• git diff --check
• cargo check --locked -p paimon -p paimon-datafusion
• cargo test --locked -p paimon --test format_partition_test
• cargo test --locked -p paimon --test rest_api_test
• cargo test --locked -p paimon --test rest_catalog_test
• cargo test --locked -p paimon api_request::tests::test_partition_request_flags_default_to_true
• cargo test --locked -p paimon-datafusion --lib
• cargo test --locked -p paimon-datafusion --test rest_format_partition_sql
• cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D warnings

[JingsongLi]

1. High: DROP PARTITION is inconsistent with Java semantics

   • Rust only supports single, complete partitions: sql_context.rs:1064 rejects multiple specs, and sql_context.rs:1734 uses require_complete=true.
   • Java supports dropping multiple partitions at once and also supports arbitrary partial specifications such as dt=... and hh=..., which are expanded to include all matching leaf partitions.
   • Impact: Commands executable in Java, such as DROP PARTITION (dt=‘20260715’) and batch DROP operations, fail outright in Rust. Recommendation: Follow Java’s approach—pre-check complete specs using list-by-names; expand partial specs after a single catalog traversal.

2. High: Catalog-managed scans lack partition pruning via the REST endpoint

   • Rust’s format_table_scan.rs:216 unconditionally uses list_partitions to retrieve all partitions in the table before executing the predicate locally.
   • Java FormatTableScan.java:230 extracts the leading equality prefix and the full predicate; CatalogFormatTablePartitionManager.java:68 pushes the pattern/predicate and page size (1000) down to REST.
   • Even when querying a single partition in a table with many partitions, all partition metadata is downloaded and parsed, which may cause significant latency and memory issues.

3. Note: Boolean partition values are incompatible with Java

   • Repair preserves and registers the original values in the catalog, but Rust format_partition.rs:216 uses str::parse::<bool>(), which only accepts lowercase true/false.
   • Java ignores case and accepts t/y/yes/1 and f/n/no/0.
   • Therefore, if a catalog entry active=TRUE is registered via Rust MSCK, subsequent scan or SHOW operations will report “invalid catalog partition metadata”; Java can read it normally. It is recommended to adopt the Java-compatible rules and perform additional cross-platform testing.