fix(table): validate write batch schema at the core entry#611
Open
liujiwen-up wants to merge 2 commits into
Open
fix(table): validate write batch schema at the core entry#611liujiwen-up wants to merge 2 commits into
liujiwen-up wants to merge 2 commits into
Conversation
Reject malformed Arrow batch schemas at the TableWrite::write_arrow_batch public entry, before the empty-batch fast path and writer dispatch, so downstream dedicated BLOB/VECTOR writers never receive a batch they index positionally and panic. Validation checks field count, names/order, and Arrow data types, allowing a trailing _VALUE_KIND: Int8 only for changelog-producer=input. Illegal input returns Error::DataInvalid.
Allow the validated trailing _ROW_ID system column so schema checks do not break DataFusion UPDATE and MERGE paths. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #xxx
TableWrite::write_arrow_batchis the public entry point for writing Arrow data into a Paimon table, but it forwarded the caller'sRecordBatchto the per-(partition, bucket) writers without validating its schema. When a caller supplied a malformed schema — e.g. a missing column — the downstream dedicated BLOB/VECTOR writer (AppendDedicatedFormatFileWriter) indexes columns positionally viabatch.column(idx)and panics instead of returning a clean error (finding F5 / P2).This change validates the incoming batch schema at the core write entry, so malformed input is rejected uniformly with
Error::DataInvalidbefore it can reach — and crash — any downstream writer.Brief change log
TableWrite::validate_write_batch_schemaand call it at the very start ofwrite_arrow_batch, before the empty-batch fast path and before writer dispatch.paimon_type_to_arrowconversion and theVALUE_KIND_FIELD_NAMEconstant; no new public API).changelog-producer=input: table fields, optionally followed by a single trailing_VALUE_KIND: Int8column.rowkind.field: only the table fields are accepted;_VALUE_KINDis generated internally, so a caller-supplied_VALUE_KINDis rejected.Error::DataInvalidwith a message carrying the expected/actual context of the first mismatching field.dedicated_format_file_writer.rsindexing/write logic, and no cast / reorder-by-name / column-filling / schema coercion.Tests
Added regression tests in the existing
table_writetest module (assertions key on the error type + context rather than the full message string):DataInvalid(no panic).DataInvalid(no panic)._VALUE_KINDon a plain write._VALUE_KINDin the wrong position / with the wrong type.rowkind.fieldrejects a caller-supplied_VALUE_KIND._VALUE_KIND, cross-partition,rowkind.field) continue to pass.Verified locally:
cargo test -p paimon --lib table_write— 71 passedcargo test -p paimon --lib— 1743 passed, 0 failedcargo fmt --all --check,cargo check -p paimon --all-targets,cargo clippy -p paimon --all-targets -- -D warnings— all cleanAPI and Format
No public API or storage format change. The added validation is a private method; the accepted-input contract is unchanged for all existing legal write modes, and previously-accepted batches keep working.
Documentation
No documentation changes required.