Skip to content

nyc-311: coordinate F64 columns ~34MB larger than vortex-jni (ALP-RD not selected) #304

Description

@dfa1

Summary

Follow-up to #299 / PR #302. On nyc-311, vortex-java is now 30 MB smaller than vortex-jni overall, but the single biggest column where java is still larger is a high-precision float coordinate — jni picks ALP-RD, java falls back to dict + alp.

Evidence — per-column bytes (java re-encode vs vortex-jni, nyc-311)

Column jni MB java MB delta jni encoding java encoding
Latitude 110.3 139.9 +29.6 vortex.alprd vortex.dict, vortex.alp
X Coordinate (State Plane) 42.0 46.3 +4.3 fastlanes.for fastlanes.bitpacked
Longitude 105.1 105.4 +0.3 alp, alprd alp

~34 MB combined (Latitude dominates). Latitude/Longitude are full-precision F64 coordinates (~40.5–40.9, many mantissa bits). jni encodes Latitude with ALP-RD (ALP real-double: splits the double into a dictionary-coded left part + a bitpacked right part — the scheme ALP falls back to when values don't fit plain ALP's integer form). java instead global-dicts it and ALP-encodes the dict values, which is larger.

Hypothesis

java has an AlpRdEncodingEncoder (used correctly for Longitude, which shows alprd), so the algorithm works — this is an encoder-selection / cost gap for Latitude specifically:

  1. Latitude qualifies as a global-dict candidate (enough repeated rounded coordinates to pass the ratio gate) and the dict path is chosen before ALP-RD gets a fair cost comparison; or
  2. the ALP-RD sample-cost estimate loses to dict+ALP on the sample even though it wins on the full column.

Worth checking why Latitude and Longitude — near-identical columns — diverge (dict+alp vs alprd).

Where

  • writer/src/main/java/io/github/dfa1/vortex/writer/encode/AlpRdEncodingEncoder.java — cost/expectedRatio
  • VortexWriter.isDictCandidate (F64 admission) + the cascade competition order (Dict vs AlpRd for F64)

Context

Measured on the hydrated Raincloud nyc-311 corpus. Fuzzier than #303 (a selection-quality question, not a raw-child-encoding gap). Not started — analysis only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions