Skip to content

fix(table): validate write batch schema at the core entry#611

Open
liujiwen-up wants to merge 2 commits into
apache:mainfrom
liujiwen-up:fix/validate-write-batch-schema
Open

fix(table): validate write batch schema at the core entry#611
liujiwen-up wants to merge 2 commits into
apache:mainfrom
liujiwen-up:fix/validate-write-batch-schema

Conversation

@liujiwen-up

@liujiwen-up liujiwen-up commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #xxx

TableWrite::write_arrow_batch is the public entry point for writing Arrow data into a Paimon table, but it forwarded the caller's RecordBatch to the per-(partition, bucket) writers without validating its schema. When a caller supplied a malformed schema — e.g. a missing column — the downstream dedicated BLOB/VECTOR writer (AppendDedicatedFormatFileWriter) indexes columns positionally via batch.column(idx) and panics instead of returning a clean error (finding F5 / P2).

This change validates the incoming batch schema at the core write entry, so malformed input is rejected uniformly with Error::DataInvalid before it can reach — and crash — any downstream writer.

Brief change log

  • Add a private TableWrite::validate_write_batch_schema and call it at the very start of write_arrow_batch, before the empty-batch fast path and before writer dispatch.
  • Validation runs in order: field count → field names/order → Arrow data type (reusing the existing paimon_type_to_arrow conversion and the VALUE_KIND_FIELD_NAME constant; no new public API).
  • Legal input contract enforced per write mode:
    • Normal write: schema must match the table fields exactly (count, order, name, type).
    • changelog-producer=input: table fields, optionally followed by a single trailing _VALUE_KIND: Int8 column.
    • cross-partition / rowkind.field: only the table fields are accepted; _VALUE_KIND is generated internally, so a caller-supplied _VALUE_KIND is rejected.
  • Illegal input returns Error::DataInvalid with a message carrying the expected/actual context of the first mismatching field.
  • No change to the downstream dedicated_format_file_writer.rs indexing/write logic, and no cast / reorder-by-name / column-filling / schema coercion.

Tests

Added regression tests in the existing table_write test module (assertions key on the error type + context rather than the full message string):

  • BLOB dedicated writer with a missing field returns DataInvalid (no panic).
  • VECTOR dedicated writer with a missing field returns DataInvalid (no panic).
  • Reordered columns, wrong field name, wrong Arrow type, unexpected extra column.
  • Illegal appended _VALUE_KIND on a plain write.
  • input-changelog _VALUE_KIND in the wrong position / with the wrong type.
  • rowkind.field rejects a caller-supplied _VALUE_KIND.
  • A schema-invalid empty batch is still rejected (validation runs before the empty-batch fast path).
  • A VECTOR batch whose FixedSizeList inner field diverges from the canonical form is rejected up front.
  • Existing legal inputs (normal write, input changelog with/without _VALUE_KIND, cross-partition, rowkind.field) continue to pass.

Verified locally:

  • cargo test -p paimon --lib table_write — 71 passed
  • cargo test -p paimon --lib — 1743 passed, 0 failed
  • cargo fmt --all --check, cargo check -p paimon --all-targets,
    cargo clippy -p paimon --all-targets -- -D warnings — all clean

API and Format

No public API or storage format change. The added validation is a private method; the accepted-input contract is unchanged for all existing legal write modes, and previously-accepted batches keep working.

Documentation

No documentation changes required.

liujiwen-up and others added 2 commits July 25, 2026 23:07
Reject malformed Arrow batch schemas at the TableWrite::write_arrow_batch
public entry, before the empty-batch fast path and writer dispatch, so
downstream dedicated BLOB/VECTOR writers never receive a batch they index
positionally and panic. Validation checks field count, names/order, and
Arrow data types, allowing a trailing _VALUE_KIND: Int8 only for
changelog-producer=input. Illegal input returns Error::DataInvalid.
Allow the validated trailing _ROW_ID system column so schema checks do not break DataFusion UPDATE and MERGE paths.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant