fix(facts): accept real EDGAR company-facts shapes during ingest#209
Merged
Conversation
A full bootstrap flooded with STORE_ERRORs that lost whole CIKs' facts
because the company-facts constraints were tighter than real EDGAR data:
- fp: EDGAR emits CY / H1 / H2 (calendar-year, half-year) beyond FY/Q1-Q4,
and an empty string "" for period-agnostic facts. Widen the FP enum and
add normalizeFp() to map "" / unknown codes to null at parse time (the
storage boundary already coalesces null to the "" PK sentinel), so one
odd fact can't fail a CIK's entire batch.
- val_unit: composite units run to ~20 chars; bump 12 -> 32.
- grouping: ifrs-full is 9 chars; bump 8/10 -> 20.
(Existing Postgres company_facts columns widened separately via ALTER.)
Also fix `update facts --retry-failed`, which crashed with "Query criteria
must not be empty" — the all-CIK read used query({}, ...); switch to
getAll({ orderBy }). This path is only reached by `update facts`, not
`bootstrap`, so it stayed hidden until a retry run.
Adds regression tests for the widened fp/val_unit/grouping schemas,
normalizeFp, and the getAll retry-lane read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ZKz3adGzMpEdy9jJ3HNXu
- xbrl_fact.context_ref overflowed varchar(128) on real filings (iXBRL context ids are long, esp. dimensional contexts). Widen schema to 512. - Add widenNarrowColumns() migration, run at the end of setupAllDatabases: CREATE TABLE IF NOT EXISTS never alters an existing table, so a Postgres DB set up before the company-facts (val_unit/grouping) and XBRL (context_ref) widenings kept the narrow columns and silently re-hit the original overflow. The migration brings existing columns up to the schema width. Postgres-only (SQLite TEXT is unbounded); guarded to skip columns already wide enough, so it is a cheap idempotent no-op on a clean DB. - Add context_ref width regression test (parity with the facts tests). Addresses the review's MAJOR finding: the maxLength bump alone left the fix incomplete on the very backend (Postgres) that exhibited the bug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ZKz3adGzMpEdy9jJ3HNXu
sroussey
force-pushed
the
fix/company-facts-ingest-constraints
branch
from
July 22, 2026 22:43
dc6091d to
11bc2c8
Compare
The test registers SEC_DRY_RUN=true in the global service registry to exercise the dry-run retry lane, but resetDependencyInjectionsForTesting() only re-registers repos — it never clears SEC_DRY_RUN. Under `bun test` all files share one process, so the leaked flag poisoned later files: on bun 1.3.14's file ordering it ran before SecFetchFileOutputCache.test.ts, whose saveOutput early-returns under dry-run, failing all 5 of its save-path assertions. Reset the flag to false in afterEach (matching dryRunRouting/runCommand). Verified: full suite 1938 pass / 0 fail on bun 1.3.14 in a Linux container (previously 5 fail). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ZKz3adGzMpEdy9jJ3HNXu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A full bootstrap flooded with STORE_ERRORs that lost whole CIKs' facts because the company-facts constraints were tighter than real EDGAR data:
Also fix
update facts --retry-failed, which crashed with "Query criteria must not be empty" — the all-CIK read used query({}, ...); switch to getAll({ orderBy }). This path is only reached byupdate facts, notbootstrap, so it stayed hidden until a retry run.Adds regression tests for the widened fp/val_unit/grouping schemas, normalizeFp, and the getAll retry-lane read.
Claude-Session: https://claude.ai/code/session_019ZKz3adGzMpEdy9jJ3HNXu