Harden YAML engine and conformance accounting

[Marius Storhaug (MariusStorhaug)]

Summary

• harden the repository-owned YAML parser around YAML whitespace, BOM/document-prefix state, flow folding, implicit keys, tag grammar, directives, and representation equality
• prevent lossy or unbounded PowerShell projection/serialization, including ETS collisions, nested enumerable buffering, Base64 allocation, long keys, numeric equality, signed zero, and timestamp boundaries
• strengthen yaml-test-suite canonicalization and keep official emit.yaml fixtures independent from module self-round-trip accounting
• add focused regressions for all original and final-review findings without adding higher-level Import/Export/Format/Merge/Remove commands

Stack

This is the third remediation layer and is intentionally stacked on yaml-conformance-fixes.

• Base: yaml-conformance-fixes
• Head: yaml-engine-hardening at 0c2fcc16e942ac282bb5fb2ecba4441243c16ea4
• Status: draft, mergeable, clean merge state

Original review findings

Parser and representation

1. YAML whitespace: grammar decisions now use explicit YAML s-white and ordinal comparisons; NBSP remains scalar content.
2. BOM boundaries: U+FEFF is consumed only in legal stream/document-prefix state, including before directives, markers, and bare documents where the grammar does not require ---; invalid token placements fail and quoted scalar BOM content is preserved.
3. Flow folding: repeated intra-line spaces are preserved and multiline flow folding retains blank-line newlines.
4. Anchor/alias colons: names stop before mapping indicators without dropping legal terminal colons.
5. Implicit keys: same-line and 1024-source-character limits apply only to productions YAML defines as implicit keys and count complete source syntax through the colon. The original request to reject every multiline flow-mapping key was not applied because ordinary flow mapping entries use the unrestricted flow-node production; released-suite and focused plain/quoted regressions verify this distinction.
6. Adjacent flow collections: {foo:[bar]}, {foo:{bar: baz}}, and [foo:[bar]] parse.
7. Unicode key length: limits count Unicode scalar values through System.Text.Rune, with emoji boundary coverage.
8. Tagged key identity: effective scalar and collection tags participate in representation fingerprints, including non-specific tag resolution.
9. Tag URI grammar: ns-uri-char, percent escapes, UTF-8 decoding, and backslash rejection are enforced.
10. Reserved directives: valid names and parameters such as %FOO-BAR baz parse while malformed productions fail.

Projection and serialization

11. Mixed ETS semantics: scalar, sequence, and dictionary values with incompatible note-property semantics fail instead of silently losing scalar data.
12. Resource budgets: node/scalar limits are enforced while consuming pipelines and enumerables; enumerators are disposed, reference identity is inspected before re-enumeration, nested reservations are shared, and Base64 allocation is prechecked.
13. Overlong rendered keys: emission switches to explicit ? keys and the output parses itself.
14. Finite float equality: Decimal/Double/Single fingerprints normalize mathematical significand and exponent.
15. Signed zero: validation and projection use one YAML representation-equality path and return classified errors rather than leaking OrderedDictionary exceptions.
16. ETS-reserved names: case-insensitive reserved PSCustomObject members produce the normal -AsHashtable representability guidance.
17. Timestamp boundaries: local DateTime Min/Max and DateTimeOffset representation failures are validated and classified stably.

Conformance runner

18. Canonical values: byte payloads, recursive complex keys, reference signatures, and unsupported values remain distinct; explicit !!omap order is carried from representation metadata rather than inferred from CLR dictionary type, and ordered mappings use an omap: discriminator distinct from ordinary map: values.
19. Emission surfaces: all 55 official emit.yaml fixtures are validated against an independent source/JSON oracle; module emission and reparse belong only to self-round-trip.
20. Policy precedence: 2JQS and X38W are classified before invalid-stream applicability, and X38W remains visible on its out.yaml surface.

Final-review follow-up

1. ? and : accept YAML tab s-white; compact-collection indentation restrictions remain intact.
2. Block alias scanning no longer swallows a mapping colon, so source: &a key\n*a: value produces the aliased key mapping while legal alias-name colons remain supported.
3. Ordinary flow-map keys no longer receive the implicit-key limit; flow-sequence pair keys count the complete source span from key start through separating trivia and colon.
4. TAG prefixes capture unseparated # as valid ns-uri-char; only whitespace-separated # starts a comment.
5. Repeated l-document-prefix* sequences and legal trailing prefixes are consumed without phantom documents.
6. Full-line, inline, flow, marker, and block-scalar-header comments reject raw U+FEFF before comment text is discarded.
7. Ordinary mapping canonicalization is unordered, explicit !!omap order comes from representation nodes, and official fixture results cannot be overwritten by self-round-trip logic.

A final code review also found quadratic repeated-BOM prefix rescanning. The parser now keeps source text immutable, advances per-line physical offsets, and validates a repeated prefix suffix once. The 1,000-prefix reproduction improved from 5.230s to 0.277s locally, and a mock-backed regression requires exactly one suffix validation.

Two final scoped findings were closed in separate commits:

• 995a6b9 allows noninitial/repeated document prefixes before bare documents whenever an explicit start is not required. # prefix\n<BOM>value and ---\none\n...\n<BOM>two match their BOM-free forms; a BOM before a second document without .../---, and a directive followed by a bare document, remain invalid.
• 0c2fcc1 encodes explicit ordered mappings as omap: while preserving entry order. A canonically ordered !!omap no longer equals an ordinary map, and focused out.yaml, official-oracle, and self-round-trip mutations prove representation-tag loss is detected/classified.

Released yaml-test-suite accounting

Pinned release: v2022-01-17, 402 inputs, zero genuine failures.

Surface	Pass	Policy	Fail	N/A / no fixture
Syntax	400	2	0	0
Events	308	0	0	94
JSON	277	2	0	123
out.yaml	241	1	0	160
Official emit.yaml	55	0	0	347
Module self-round-trip	305	3	0	94

Explicit policy differences

• 2JQS, X38W: YAML representation mapping-key uniqueness; both affect syntax and self-round-trip, and X38W also affects out.yaml.
• 565N: !!binary intentionally projects as byte[], affecting JSON projection only.
• J7PZ: legacy !!omap intentionally projects as OrderedDictionary, affecting JSON projection; self-round-trip preserves entries and order but cannot reconstruct the lost explicit !!omap representation tag.

No parser or emitter inability is classified as policy. There are no deferred findings.

Validation

Local PowerShell 7.6.4 Core:

• focused BOM prefix regression: 1 passed, 0 failed
• focused ordered-map regressions: 2 passed, 0 failed
• complete source Pester: 199 passed, 0 failed, 2 artifact-only skipped (201 total) in 32.48s
• PSScriptAnalyzer over src: 0 findings
• parser checks: 86 files, 0 errors
• complete six-surface 402-case run: exact counts above, 0 genuine failures

Process-PSModule run 30058785794 passed at the final SHA in 257 seconds. Windows, Linux, and macOS all used PowerShell 7.6.3 Core and tested the built module artifact.

Platform	Conformance	ConvertFrom	ConvertTo	Packaging / fresh import	Specification	Test-Yaml	Artifact total
Windows	19/0/0, 43.57s	65/0/0, 10.49s	42/0/0, 12.80s	10/0/0, 5.77s	48/0/0, 8.51s	17/0/0, 10.38s	201/0/0, 91.52s
Linux	19/0/0, 33.63s	65/0/0, 7.86s	42/0/0, 10.21s	10/0/0, 6.01s	48/0/0, 5.48s	17/0/0, 7.95s	201/0/0, 71.14s
macOS	19/0/0, 34.65s	65/0/0, 12.09s	42/0/0, 11.41s	10/0/0, 3.84s	48/0/0, 5.58s	17/0/0, 7.52s	201/0/0, 75.09s

Counts are passed/failed/skipped; total time is the sum of each platform's reported project Pester durations. The Process-PSModule source harness also passed 9/0/1 and the built-module harness passed 4/0/1 on every platform. Source and built-artifact analyzer suites each passed 73/0/2 on every platform. Build-Module passed in 21s, Build-Docs in 133s, Build-Site in 22s, aggregate test reporting in 25s, and code coverage in 65s. Repository lint and publish jobs were intentionally skipped for the manual draft-branch dispatch; no required check failed or remains pending.