Skip to content

[WIP][iceberg] Support the v3 variant type with row lineage and REST catalog#8795

Draft
plusplusjiajia wants to merge 8 commits into
apache:masterfrom
plusplusjiajia:gh-iceberg-variant
Draft

[WIP][iceberg] Support the v3 variant type with row lineage and REST catalog#8795
plusplusjiajia wants to merge 8 commits into
apache:masterfrom
plusplusjiajia:gh-iceberg-variant

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

Add Iceberg v3 variant type support to Paimon's Iceberg compatibility layer, with the row lineage and REST catalog upgrade it requires. The three are interdependent: v3 variant metadata without row lineage is spec-invalid (Iceberg 1.10+ won't parse metadata lacking next-row-id/first-row-id), and row lineage without the REST upgrade still leaves rest-catalog storage creating v2 tables that reject v3 features.

  • Variant type: IcebergDataField maps Paimon VARIANT ↔ Iceberg "variant" both ways. Before a snapshot is published, IcebergCommitCallback rejects types that cannot produce readable metadata: variant when format-version < 3 (and as a partition key, unconditionally), and nanosecond timestamps on every version (Paimon writes them as Parquet INT96, which Iceberg cannot read as timestamp_ns). Validation runs before publish and before a tag is written.
  • Row lineage: IcebergManifestFileMeta/IcebergDataFileMeta gain first_row_id; ranges are assigned when writing the manifest list, mirroring Iceberg's ManifestListWriter. Tables upgraded from v2 get their surviving manifests assigned on the first v3 commit; regeneration continues past the highest published id so ids are never reused; rewritten/merged manifests keep unchanged rows' _row_id.
  • REST upgrade: bump to Iceberg 1.10.1 (1.8.1 can't parse variant). Creation passes the configured format-version; a populated table whose version differs is dropped and recreated. Fallout: Hadoop 3.3.6 for FileSystem.openFile, jetty/commons-lang3 test-classpath alignment, RemoveSnapshotRemoveSnapshots.

@plusplusjiajia plusplusjiajia changed the title [iceberg] Support the v3 variant type with row lineage and REST catalog [WIP][iceberg] Support the v3 variant type with row lineage and REST catalog Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant