Cascade FSST offset/length children (#299)#302
Merged
Conversation
FsstEncodingEncoder emitted its two per-row children — uncompressed lengths and cumulative code offsets — as raw vortex.primitive leaves (2-4 bytes/row each), never cascaded. The Rust reference bitpacks the monotonic offsets and constant-folds the lengths of fixed-width columns; leaving them raw was the single largest remaining size gap on nyc-311 (the date columns compress the bytes with FSST but paid full-width offsets). encodeCascade now exposes both children as open primitive child slots so the cascade bitpacks the offsets and picks vortex.constant for fixed-width lengths. The shared compression work is extracted into a compress() helper; encode() keeps the terminal raw-primitive layout for cascade depth 0. No wire/reader change — the reader already decodes bitpacked/constant FSST children (it reads Rust files that use them). nyc-311 re-encode: 1934.23 MB -> 1732.14 MB (1.10x -> 0.98x vortex-jni), i.e. vortex-java is now smaller than the Rust reference. Round-trip + 217 Java-writes-Rust-reads interop tests pass; full verify green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 24, 2026
Closed
dfa1
added a commit
that referenced
this pull request
Jul 25, 2026
encodeUtf8Cascade exposed the dict values pool as an open child (#299 Finding 2b) but left the codes as a raw vortex.primitive leaf — paying the full U8/U16/U32 code width per row. The Rust reference bitpacks dict codes (fastlanes.bitpacked), and the primitive dict path here already exposes its codes the same way; only the Utf8 path was missing it. Now both children are left open: codes (index 0) as a typed primitive array the cascade bitpacks, and the values pool (index 1) for the FSST/VarBin competition. No wire/reader change — the reader already decodes bitpacked dict codes (it reads Rust files that use them). nyc-311 re-encode: 1934.23 MB -> 1862.23 MB (1.10x -> 1.06x vortex-jni); the win spans every per-chunk Utf8 dict column, not only the street columns that surfaced it. Composes with #302 (they compress different children). Round-trip + 217 Java-writes-Rust-reads interop tests pass; full verify green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
FsstEncodingEncoderemitted its two per-row children — the uncompressed lengths and the cumulative code offsets — as rawvortex.primitiveleaves (2–4 bytes/row each), never cascaded. The Rust reference bitpacks the monotonic offsets and constant-folds the lengths of fixed-width columns. Leaving them raw was the single largest remaining size gap on the nyc-311 corpus.Per-chunk encoding tree, before vs after:
How
encodeCascadenow exposes both children as open primitive child slots so the cascade bitpacks the offsets and picksvortex.constantfor fixed-width lengths. The shared compression work is extracted into acompress()helper;encode()keeps the terminal raw-primitive layout for cascade depth 0.No wire or reader change — the reader already decodes bitpacked/constant FSST children (it reads Rust files that use exactly this shape;
RaincloudConformance+ interop suites cover it).Impact (real nyc-311 corpus, 18.49M rows × 42 cols)
−202 MB from this change alone — vortex-java is now smaller than the Rust reference on nyc-311. This helps every FSST-encoded column, not just the date columns that surfaced it.
Testing
encodeCascadeexposes the two offset children as open slots; falls back to terminal at depth 0../mvnw verifygreen.🤖 Generated with Claude Code