Skip to content

Harden and extend Yaml parsing and add Yaml.stringify - #10287

Open
effect-bot wants to merge 54 commits into
v4/next-minorfrom
agent/effect-bot/eff-3232-printer-round-trip
Open

effect-bot wants to merge 54 commits into
v4/next-minorfrom
agent/effect-bot/eff-3232-printer-round-trip

Conversation

@effect-bot

@effect-bot effect-bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

The rest of the YAML expansion in one PR. #10271 (Yaml.parseAll, directives, and Result from Yaml.parse) has already merged. This PR adds the parser hardening, new grammar, core schema tags, and the printer. It replaces #10276, #10279, #10282, #10283, #10284, #10285 and #10272, which were reviewed as separate stages and are now closed. Their commits are kept here unchanged.

Parser

Input that used to be misread now fails or parses correctly

  • Tags. Before this PR, every tag was kept as part of the scalar text. Now the shorthand YAML 1.2 core tags resolve (§10.3): !!str, !!int, !!float, !!bool, !!null, !!seq and !!map.
    • A scalar tag applies to the scalar as written, so !!str 0x10 is "0x10" and !!int "42" is 42.
    • Each tag accepts what the spec's patterns allow. A node that doesn't match its tag fails, for example !!int x, !!bool yes or !!seq {a: 1}.
    • These still fail, with the tag named in the error: local tags (!custom), the non-specific tag !, verbatim tags (even for core tags), YAML 1.1 tags (!!binary, !!timestamp, !!set) and unknown !! tags.
    • A ! inside a plain scalar (x!y) is still text.
  • Explicit keys (? key, {? a : 1}, [? a]) fail with a clear error. The flow forms used to give the key "? a".
  • Alias keys. *name : value uses the anchored scalar as the key, where it used to be the literal text "*name". An alias to a collection used as a key fails. Anchors on keys are recorded.
  • Flow keys are converted the same way as block keys, so {0x10: 1} gives "16".
  • Plain scalars resolve with the YAML 1.2 core schema (§10.3.2), the same patterns the core tags use. tRuE, 1_000 and 0X1F are strings, and 012, 5. and .5e3 are numbers.
  • Flow separators. In flow collections, : ends a plain scalar. {a: b: c} and [a: 1: 2] fail instead of giving "b: c" and "a: 1: 2", and {http://x: 1} gives the key "http://x". A flow mapping key without a value that contains : ({a:1}) fails, since YAML reads it as the key "a:1" while JSON-like input means {a: 1}. An unquoted [ or { inside a flow plain scalar fails ([a[, b], {a[}), as ns-plain-safe-in requires.
  • Merges may copy at most 100,000 keys per document, since a chain of merges grows the result quadratically in the input.
  • Nesting is limited to 1000 levels, block and flow combined. Deeper input fails with an EncodingError instead of a stack overflow.
  • White space. Only spaces and tabs count as white space (§5.5). A no-break space (U+00A0), U+2028 and similar characters are kept as content.
  • Flow plain scalars starting with -. A - followed by white space or a flow indicator ([- 1], [1, -]) fails, as ns-plain-first in §7.3.3 requires.
  • key: - value fails. It used to be read as the string "- value".

New grammar

  • Compact nested sequences (- - value, §8.2.1), to any depth within the nesting limit.
  • Multiline single- and double-quoted scalars inside flow collections (§7.3). They fold like quoted block values, and the scan stays linear in input length.
  • A tab after a sequence dash counts as separation (§6.2), so -\t1 is an entry. A tab before a compact nested collection (-\t- 1) fails, because indentation can't use tabs.
  • A backslash followed by a literal tab in a double-quoted scalar is an escaped tab (§5.7).
  • Single pairs in flow sequences ([a: 1, b] gives [{a: 1}, "b"], §7.4.1) and flow mapping keys without values ({a, b: 1} gives {a: null, b: 1}).

Printer

Yaml.stringify(value: unknown, options?: { readonly indent?: number | undefined }): Result<string, EncodingError>
Yaml.stringifyAll(values: Iterable<unknown>, options?: { readonly indent?: number | undefined }): Result<string, EncodingError>
  • Supported values: null, booleans, numbers, strings, arrays and plain objects. The output is block style with keys in insertion order, {} and [] for empty collections, and a single trailing newline.
  • Strings print plain only when parse and the core schema both read them back as the same string. Otherwise they are double-quoted with YAML escapes. Keys follow the same rule. Spellings that YAML 1.1 or lenient parsers read differently are quoted as well, such as tRuE, 1_000 and ---x. Multiline strings print as |, |- or |+ literal block scalars when that is safe.
  • Special numbers: .inf, -.inf, .nan and -0.0 round-trip.
  • Like JSON.stringify: toJSON is applied, and undefined properties are omitted.
  • Unlike JSON.stringify, nothing is silently dropped. Bigints, functions, symbols, non-plain objects without toJSON, and cycles fail with an EncodingError that names the path.
  • Layout: nested sequences print in the two-line form (-, then an indented - item). Shared references print in full each time, without anchors.
  • Round-trip checks: every printer test checks the output against yaml 2.9.1, which is added as a devDependency of packages/effect for that purpose. Each test also reads the output back with Yaml.parse or Yaml.parseAll. The core schema patterns are shared between tag resolution and the printer.

Remaining limits

The module docs describe these limits:

  • %TAG and other directives, and YAML versions other than 1.2, are not supported.
  • Recursive aliases are not supported.
  • Mapping keys are always JavaScript strings.
  • Numbers use JavaScript number semantics, so integers beyond Number.MAX_SAFE_INTEGER lose precision unless they are quoted.
  • Comments and source formatting are not retained.

History

  • The parser stages and the printer were developed on separate branches. Merge commit f5d52c1735 joins them, and it is the only place their changes conflicted. The printer was added after the parser code in Yaml.ts, and the printer tests after the parser tests. The printer branch's older parse was dropped.
  • One minor changeset, .changeset/yaml-parse-stringify.md, is the release note for the whole PR. It replaces the seven per-stage changesets and describes the changes against v4/next-minor, with a migration bullet for each input that now reads differently or fails.
  • Two fixes from the full review follow. A mapping value's block sequence may now be indentless after node properties split across lines (a: !!seq / &items / - 1), including properties that start on the line after the key. The printer trims a literal block's trailing line breaks with a backward scan instead of /\n+$/, which retried from every newline of an interior run and took seconds on 64,000 newlines followed by text.
  • The second full review's findings come after that: flow pairs and keys without values, core-schema plain scalars (which also removes a float regex that backtracked quadratically on long digit runs), the merge budget, quoting of strings that start with --- or ..., and the release note entries for input that now fails.
  • Two performance commits come last: one for stringify (scalar classification, double-quoting and error paths) and one for parse and parseAll (single-pass line scans, character-code checks and plain key assignment). Neither changes behavior, tests or the changeset.

Validation

  • At 1b039eed5b: Yaml.test.ts and the McpServer tests pass, and the CLI, AI/MCP, ai-codegen and openapi-generator tests pass with 2,395 passed and 53 skipped.
  • pnpm lint-fix, pnpm lint, pnpm check, pnpm jsdocs --check and the four Yaml.ts doctests pass.
  • Each stage was reviewed separately. The diff of each stage is unchanged here, checked with matching stable patch IDs and git range-diff.

Closes EFF-3232

@changeset-bot

changeset-bot Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: deb32eb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 33 packages
Name Type
effect Minor
@effect/opentelemetry Minor
@effect/vitest Minor
@effect/ai-anthropic Minor
@effect/ai-cloudflare Minor
@effect/ai-openai-compat Minor
@effect/ai-openai Minor
@effect/ai-openrouter Minor
@effect/ai-typesafe Minor
@effect/atom-react Minor
@effect/atom-solid Minor
@effect/atom-vue Minor
@effect/platform-browser Minor
@effect/platform-bun Minor
@effect/platform-deno Minor
@effect/platform-node-shared Minor
@effect/platform-node Minor
@effect/sql-clickhouse Minor
@effect/sql-d1 Minor
@effect/sql-libsql Minor
@effect/sql-mssql Minor
@effect/sql-mysql2 Minor
@effect/sql-pg Minor
@effect/sql-pglite Minor
@effect/sql-sqlite-bun Minor
@effect/sql-sqlite-do Minor
@effect/sql-sqlite-node Minor
@effect/sql-sqlite-react-native Minor
@effect/sql-sqlite-wasm Minor
@effect/docgen Minor
@effect/doctest Minor
@effect/graphql-generator Minor
@effect/openapi-generator Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 39.51 KB 39.51 KB 0.00 KB (0.00%)
basic.ts 7.27 KB 7.27 KB 0.00 KB (0.00%)
batching.ts 11.16 KB 11.16 KB 0.00 KB (0.00%)
brand.ts 6.87 KB 6.87 KB 0.00 KB (0.00%)
cache.ts 11.63 KB 11.63 KB 0.00 KB (0.00%)
config.ts 22.93 KB 22.93 KB 0.00 KB (0.00%)
differ.ts 22.09 KB 22.09 KB 0.00 KB (0.00%)
http-client.ts 27.27 KB 27.27 KB 0.00 KB (0.00%)
http-router.ts 37.17 KB 37.17 KB 0.00 KB (0.00%)
logger.ts 12.10 KB 12.10 KB 0.00 KB (0.00%)
metric.ts 9.56 KB 9.56 KB 0.00 KB (0.00%)
optic.ts 7.10 KB 7.10 KB 0.00 KB (0.00%)
pubsub.ts 16.14 KB 16.14 KB 0.00 KB (0.00%)
queue.ts 13.01 KB 13.01 KB 0.00 KB (0.00%)
schedule.ts 11.88 KB 11.88 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 14.18 KB 14.18 KB 0.00 KB (0.00%)
schema-binary.ts 42.09 KB 42.09 KB 0.00 KB (0.00%)
schema-class.ts 21.83 KB 21.83 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 29.69 KB 29.69 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 28.09 KB 28.09 KB 0.00 KB (0.00%)
schema-string-transformation.ts 15.45 KB 15.45 KB 0.00 KB (0.00%)
schema-string.ts 12.38 KB 12.38 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.36 KB 16.36 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 39.05 KB 39.05 KB 0.00 KB (0.00%)
schema-toCodeDocument.ts 26.25 KB 26.25 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 20.99 KB 20.99 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 21.15 KB 21.15 KB 0.00 KB (0.00%)
schema-toFormatter.ts 21.25 KB 21.25 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 26.26 KB 26.26 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 21.33 KB 21.33 KB 0.00 KB (0.00%)
schema.ts 20.92 KB 20.92 KB 0.00 KB (0.00%)
stm.ts 14.30 KB 14.30 KB 0.00 KB (0.00%)
stream.ts 10.47 KB 10.47 KB 0.00 KB (0.00%)
supervisor.ts 11.41 KB 11.41 KB 0.00 KB (0.00%)

@effect-bot
effect-bot force-pushed the v4/next-minor branch 7 times, most recently from 97a1ee1 to 10d44ad Compare November 3, 2026 22:52
@effect-bot
effect-bot force-pushed the agent/effect-bot/eff-3232-printer-round-trip branch from 60cde73 to 006a562 Compare November 3, 2026 23:14
@effect-bot
effect-bot force-pushed the agent/effect-bot/eff-3232-printer-round-trip branch from 006a562 to 26e2c51 Compare November 3, 2026 23:40
@effect-bot
effect-bot changed the base branch from v4/next-minor to agent/architect-effect-bot/6a52aa9988ae November 3, 2026 23:40
@effect-bot
effect-bot force-pushed the agent/effect-bot/eff-3232-printer-round-trip branch from 26e2c51 to 58b2c97 Compare November 3, 2026 23:43
@effect-bot
effect-bot force-pushed the agent/architect-effect-bot/6a52aa9988ae branch from 7a672f9 to 3b465fb Compare November 3, 2026 23:43
@effect-bot effect-bot changed the title Check Yaml.stringify output against Yaml.parse Harden and extend Yaml parsing and add Yaml.stringify Nov 3, 2026
@effect-bot
effect-bot changed the base branch from agent/architect-effect-bot/6a52aa9988ae to v4/next-minor November 3, 2026 23:57
@effect-bot
effect-bot force-pushed the agent/effect-bot/eff-3232-printer-round-trip branch 2 times, most recently from d1233b6 to c05b323 Compare November 4, 2026 01:02
@effect-bot
effect-bot force-pushed the agent/effect-bot/eff-3232-printer-round-trip branch from 32277cb to 3e92887 Compare November 4, 2026 02:08
effect-bot and others added 28 commits November 4, 2026 04:11
The shorthand tags !!str, !!int, !!float, !!bool, !!null, !!seq and
!!map resolve against the scalar as written, a mismatched node fails,
and every other tag, including verbatim and non-specific tags, still
fails.
Tags keep the token start for structure and comment scanning, a node's
properties may continue on the next line and its anchor records the
resolved value, and a node has at most one anchor and tag, no
properties on an alias, and separation before flow content.
A scalar tag on a flow collection key fails like a tagged collection
value, and a node starting with '&' must name its anchor.
…inter

The printer's quoting rule and the parser's core tag resolution each
spelled out the YAML 1.2 core schema; one table now serves both.
Classify plain-safe strings in one pass with a prefilter before the core
schema patterns, escape double-quoted strings by slicing between escapes,
build error paths only when a failure happens, and cache indentation.
Scan lines and flow collections by character code instead of per-character
regex tests and single-character strings. Find a line's comment and its
mapping separator in one pass in makeLine, keep the separator on the line,
and reuse it for sequence entries instead of scanning each line again. Avoid
allocating trimmed copies for blank checks and untouched strings, skip the
core schema patterns for plain scalars that cannot match them, and assign
keys absent from the prototype chain directly instead of defining them.
@effect-bot
effect-bot force-pushed the agent/effect-bot/eff-3232-printer-round-trip branch from 0cf9e85 to 1b039ee Compare November 4, 2026 04:14
The printer now asks resolvePlain whether a spelling is a core scalar
instead of keeping a second set of the schema patterns, the non-printable
set has one code-unit definition shared by plain and block scalar checks,
and the string whitespace predicate is folded into its code-unit form.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants