Summary
Follow-up to #299 / PR #302. On nyc-311, vortex-java is now 30 MB smaller than vortex-jni overall, but the single biggest column where java is still larger is a high-precision float coordinate — jni picks ALP-RD, java falls back to dict + alp.
Evidence — per-column bytes (java re-encode vs vortex-jni, nyc-311)
| Column |
jni MB |
java MB |
delta |
jni encoding |
java encoding |
| Latitude |
110.3 |
139.9 |
+29.6 |
vortex.alprd |
vortex.dict, vortex.alp |
| X Coordinate (State Plane) |
42.0 |
46.3 |
+4.3 |
fastlanes.for |
fastlanes.bitpacked |
| Longitude |
105.1 |
105.4 |
+0.3 |
alp, alprd |
alp |
~34 MB combined (Latitude dominates). Latitude/Longitude are full-precision F64 coordinates (~40.5–40.9, many mantissa bits). jni encodes Latitude with ALP-RD (ALP real-double: splits the double into a dictionary-coded left part + a bitpacked right part — the scheme ALP falls back to when values don't fit plain ALP's integer form). java instead global-dicts it and ALP-encodes the dict values, which is larger.
Hypothesis
java has an AlpRdEncodingEncoder (used correctly for Longitude, which shows alprd), so the algorithm works — this is an encoder-selection / cost gap for Latitude specifically:
- Latitude qualifies as a global-dict candidate (enough repeated rounded coordinates to pass the ratio gate) and the dict path is chosen before ALP-RD gets a fair cost comparison; or
- the ALP-RD sample-cost estimate loses to dict+ALP on the sample even though it wins on the full column.
Worth checking why Latitude and Longitude — near-identical columns — diverge (dict+alp vs alprd).
Where
writer/src/main/java/io/github/dfa1/vortex/writer/encode/AlpRdEncodingEncoder.java — cost/expectedRatio
VortexWriter.isDictCandidate (F64 admission) + the cascade competition order (Dict vs AlpRd for F64)
Context
Measured on the hydrated Raincloud nyc-311 corpus. Fuzzier than #303 (a selection-quality question, not a raw-child-encoding gap). Not started — analysis only.
Summary
Follow-up to #299 / PR #302. On nyc-311, vortex-java is now 30 MB smaller than vortex-jni overall, but the single biggest column where java is still larger is a high-precision float coordinate — jni picks ALP-RD, java falls back to
dict + alp.Evidence — per-column bytes (java re-encode vs vortex-jni, nyc-311)
vortex.alprdvortex.dict, vortex.alpfastlanes.forfastlanes.bitpackedalp, alprdalp~34 MB combined (Latitude dominates). Latitude/Longitude are full-precision F64 coordinates (~40.5–40.9, many mantissa bits). jni encodes Latitude with ALP-RD (ALP real-double: splits the double into a dictionary-coded left part + a bitpacked right part — the scheme ALP falls back to when values don't fit plain ALP's integer form). java instead global-dicts it and ALP-encodes the dict values, which is larger.
Hypothesis
java has an
AlpRdEncodingEncoder(used correctly for Longitude, which showsalprd), so the algorithm works — this is an encoder-selection / cost gap for Latitude specifically:Worth checking why Latitude and Longitude — near-identical columns — diverge (
dict+alpvsalprd).Where
writer/src/main/java/io/github/dfa1/vortex/writer/encode/AlpRdEncodingEncoder.java— cost/expectedRatioVortexWriter.isDictCandidate(F64 admission) + the cascade competition order (Dict vs AlpRd for F64)Context
Measured on the hydrated Raincloud nyc-311 corpus. Fuzzier than #303 (a selection-quality question, not a raw-child-encoding gap). Not started — analysis only.