Skip to content

feat: adopt the Vortex "editions" model (frozen encoding sets + writer edition guard) #301

Description

@dfa1

Context

Upstream merged vortex-data/vortex#8871, formalizing editions: frozen, additive, per-family sets of encoding IDs, each with a min_vortex_version read-compatibility floor. Spec: docs/specs/editions.md. It is wired but not yet public; a follow-up makes it the default write gate.

An edition is a frozen, named, additive set of encoding IDs carrying a forever read guarantee:

  • Editions come in independently-versioned families: core (what the default writer emits) and unstable (opt-in, no guarantee). A writer targets at most one edition per family and may emit any encoding in their union.
  • Each edition records a min_vortex_version — the earliest reader release supporting every encoding in it. Recording that version is the act of freezing; an edition with None is a mutable draft.
  • Writer guarantee: if a write would emit an encoding outside the targeted edition, the write fails immediately — "edition violations never surface as someone else's read error later."
  • Reader UX: an unknown-encoding error points at the registry, telling you which edition an ID joined and therefore which reader version you need.

Where vortex-java stands

We already implement every encoding in the core family through the latest frozen edition (core2026.07.0). So this is not about chasing encodings — it is about adopting the editions model itself (a frozen registry + a writer fail-fast guard + reader error UX), which we have no equivalent of today. Our WriteRegistry will emit any registered encoding, and our unknown-encoding read error carries no "which edition / which version" guidance.

Frozen core sets vs. our EncodingId.WellKnown:

Edition min vortex Adds Supported?
core2025.05.0 0.36.0 23 baseline (primitive, bool, alp, fsst, dict, fastlanes.bitpacked/for, …) ✅ all
core2025.06.0 0.40.0 vortex.pco, vortex.sequence, vortex.zstd ✅ all
core2025.10.0 0.54.0 fastlanes.rle, vortex.fixed_size_list, vortex.listview, vortex.masked ✅ all
core2026.07.0 0.65.0 vortex.variant ✅ (ID present)
unstable (2025.05 → 2026.06) draft fastlanes.delta, vortex.patched, zstd_buffers, parquet.variant, tensor.*, onpair partial (delta, patched)

Scope

  1. Frozen edition registry (data + type). A small immutable model — EditionId (family/year/month/version, chronologically ordered within a family only) + Edition (minVortexVersion: Optional<String>, empty ⇒ draft) + additive membership data. Seed the four core and four unstable declarations from #8871. Membership is additive: an ID present in edition N is present in all later editions of the family. Back it with a golden test pinning each frozen edition's computed set (mirrors upstream's validate_edition harness) so a stray data edit fails CI.

  2. Writer edition guard (the core value). Let WriteOptions target an edition (default: latest frozen core, possibly lagging by design). During encode, if the tree would contain an encoding outside the targeted edition's union, fail fast with a VortexException naming the offending ID and the edition — never silently write a file a conforming reader can't read. Support one edition per family (e.g. core + an unstable opt-in), plus an explicit "no edition / custom encodings" escape hatch.

  3. Reader unknown-encoding UX. When ReadRegistry hits an unregistered ID (and allowUnknown is off), enrich the VortexException: if the ID is a known edition member, name the edition it joined and the min_vortex_version needed; if unknown to all editions, say so and point at allowUnknown. Mirrors the spec's "Resolving an unknown-encoding error" section.

Non-goals

  • No new encodings (we already cover core). The unstable tensor/variant/zstd_buffers/onpair encodings stay out of scope — separate tickets if ever wanted.
  • No ServiceLoader/auto-discovery — editions are registered explicitly on the builder, consistent with the existing registry design decision.
  • Not blocking on upstream making editions public; we track the frozen data now and re-sync when their default flips.

Acceptance criteria

  • Editions model + the 8 seeded declarations, with a golden test pinning frozen sets.
  • WriteOptions.edition(...); writing an out-of-edition encoding throws a VortexException naming ID + edition; escape hatch tested.
  • Reader unknown-encoding error names the edition + required version when applicable.
  • Docs: docs/reference.md + docs/compatibility.md + CLAUDE.md design-decisions/module-map updated in the same PR; CHANGELOG entry.
  • An adr/ note recording editions adoption and how it composes with ReadRegistry/WriteRegistry/allowUnknown.

Open questions

  • Do we persist the targeted edition into file metadata, or is it write-time-only? (Upstream currently treats it as a writer constraint, not a file field.)
  • Default target: latest frozen core vs. a deliberately-lagging one (upstream lags "by a few versions").

Upstream ref: vortex-data/vortex#8871 · editions spec

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions