Repository navigation
feat(db)!: make grouping, includes, and pooling opt-in - #2232
Open
KyleAMathews wants to merge 29 commits into
Open
KyleAMathews wants to merge 29 commits into
KyleAMathews wants to merge 29 commits into
Conversation
Joins take an operator value: .join(leftJoin(source, on)). The operator carries the join compiler and the optimizer's join pushdown onto the IR join clause, so apps without joins do not bundle them. Co-authored-by: Isaac <no-reply@databricks.com>
…ll shape Co-authored-by: Isaac <no-reply@databricks.com>
Aggregate values (count, sum, avg, min, max) carry the grouping compiler. A groupBy with no aggregate takes groupKeys(...); without it, compilation throws GroupingOperatorRequiredError. orderBy rewrites aggregates only when it finds one. Co-authored-by: Isaac <no-reply@databricks.com>
include(), toArray(), and materialize() carry the materializer and the Collection-facade adapter. A bare subquery in select() throws IncludeOperatorRequiredError; include(subquery) gives the Collection-valued include. Join operators carry the materializer for custom-key reduction. Co-authored-by: Isaac <no-reply@databricks.com>
Hooks pool a query only when its config passes pool: pooledLiveQueries. The value carries the pooled view and its identity shortcut, so apps that do not pool do not bundle them. React Suspense and DbClient queries pass pool: false and never pool. Co-authored-by: Isaac <no-reply@databricks.com>
Join calls use operator values, bare includes use include(), and the conformance pooling scenario passes pooledLiveQueries through each driver's config form. A pooled config with a query function builds it once. Co-authored-by: Isaac <no-reply@databricks.com>
The live-queries guide, JSDoc examples, and generated reference use join operators, include(), groupKeys(), and pooledLiveQueries. The join size scenario uses leftJoin; size:compare reports n/a for a scenario that cannot build against an older base. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: e642360 The changes in this PR will be included in the next version bump. This PR includes changesets to release 29 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
Important Review skippedToo many files! This PR contains 350 files, which is 50 over the limit of 300. To get a review, reduce the PR to 300 files or fewer by splitting it into smaller PRs or changing its base branch. Usage-priced reviews support at most 300 files. ⚙️ Run configuration
📒 Files selected for processing: 350
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
Size Change: +2.46 kB (+1.12%) Total Size: 223 kB 📦 View Changed
ℹ️ View Unchanged
|
More templates
@tanstack/angular-db
@tanstack/browser-db-sqlite-persistence
@tanstack/capacitor-db-sqlite-persistence
@tanstack/cloudflare-durable-objects-db-sqlite-persistence
@tanstack/db
@tanstack/db-ivm
@tanstack/db-sqlite-persistence-core
@tanstack/electric-db-collection
@tanstack/electron-db-sqlite-persistence
@tanstack/expo-db-sqlite-persistence
@tanstack/node-db-sqlite-persistence
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/react-native-db-sqlite-persistence
@tanstack/react-router-with-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/tauri-db-sqlite-persistence
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
… pool
A DbClient query never pools, so prepareLiveQueryValue drops pool and a
server preload hashes like a pooling hook. pool moves from
LiveQueryCollectionConfig to LiveQueryHookConfig, which the framework hooks
accept. Angular's { params, query } form takes pool. Join operators have a
named function type so their declarations stay portable. The benchmark runs
against either join API, and the minified API check lists the new errors.
Co-authored-by: Isaac <no-reply@databricks.com>
…rializer A join clause carries one runtime object (compiler, pushdown, validation, and public-key reduction), and the optimizer copies it by spreading the clause. Public-key reduction moves to its own module, so a joined query with a custom key no longer bundles the include materializer. The bundle test now tracks the materializer and the shared reduction module. Co-authored-by: Isaac <no-reply@databricks.com>
… input caseWhen branches take include() for a subquery; the select-object branch no longer accepts a query builder. include() of a non-builder throws a coded error. The compiler finds aggregates with one walker, and the fn.select error names include(). size:compare skips only a base that lacks a scenario's new export. Co-authored-by: Isaac <no-reply@databricks.com>
The shipped agent skills use join operators, include(), groupKeys(), and pooledLiveQueries. join() and groupBy() JSDoc describe the new parameters. The changeset covers Angular's params pool and hand-built IR. Oracle files keep their opening contract prose above their imports. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Incremental update benchmarkComparing Overall median write time vs base: 1.02× · cold hydrate time: 1.00× (geometric mean of per-case ratios; lower is faster). Writes: 3 regression(s), 2 improvement(s) (threshold: ±20% and >0.05ms). Cold hydrate: 3 regression(s), 0 improvement(s) (threshold: ±50% and >5ms). Per-case flags are noisy on shared runners. Read the geometric means first. Writes
Cold hydrate
Each row aggregates the 14.4 scale/index/write-mode configurations of that query; per-configuration tables below. 100 rows/collection | source indexes: none | synced writes — geomean 0.98×, cold 1.05×
100 rows/collection | source indexes: none | optimistic writes — geomean 1.06×, cold 1.03×, 1 change(s)
100 rows/collection | source indexes: manual | synced writes — geomean 0.93×, cold 0.93×
100 rows/collection | source indexes: manual | optimistic writes — geomean 0.98×, cold 0.96×
1,000 rows/collection | source indexes: none | synced writes — geomean 1.06×, cold 1.27×, 5 change(s)
1,000 rows/collection | source indexes: none | optimistic writes — geomean 1.08×, cold 0.87×
1,000 rows/collection | source indexes: manual | synced writes — geomean 1.11×, cold 1.24×
1,000 rows/collection | source indexes: manual | optimistic writes — geomean 1.02×, cold 0.87×
10,000 rows/collection | source indexes: none | synced writes — geomean 0.97×, cold 0.91×, 1 change(s)
10,000 rows/collection | source indexes: none | optimistic writes — geomean 1.08×, cold 1.11×, 1 change(s)
10,000 rows/collection | source indexes: manual | synced writes — geomean 0.97×, cold 0.94×
10,000 rows/collection | source indexes: manual | optimistic writes — geomean 1.02×, cold 0.94×
Runner: node v24.8.0, linux 6.17.0-1022-azure, AMD EPYC 7763 64-Core Processor. Timings on shared CI runners are noisy; treat small deltas as indicative only. |
A join operator infers the query's context from .join(). TypeScript skips that inference when noImplicitAny is off, which left every join condition with unknown refs and compile errors. The context now defaults to any there, so the condition is untyped like other unannotated callbacks in that mode. Strict mode keeps full typing. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Add scripts/check-doc-snippets.mjs, run in the CI lint job as `pnpm test:doc-snippets`. It typechecks each TypeScript example in docs/ and the shipped skills against the workspace sources. Examples that already failed are listed in a baseline by content hash, so a new or edited example must pass. The check found these errors: - Joins lost their types without noImplicitAny. Strict mode keeps them; the test compiles a wrong field in both modes. - A bare subquery in select type-checked, also nested, because the ref and nested-object shapes are open. Both select overloads now reject a value that holds a query builder outside include(), toArray(), or materialize(). - Guide and skill examples: a nested bare subquery, missing imports, an orderBy on $selected, and iteration over a Collection. Co-authored-by: Isaac <no-reply@databricks.com>
The autofix bot reformatted skill examples, which changed their hashes and made known failures look new. The baseline now hashes each example's tokens without commas, semicolons, parentheses, and spacing, so a reformatted example keeps its entry. An edit to its code still does not. Co-authored-by: Isaac <no-reply@databricks.com>
The conformance suite checked pooling only in the config form with pool. Two scenarios now run in every framework driver: - pool-opt-in: queries without pool keep a source subscription each in the function, config, and parameterized forms. - pooled-param-change: a pooled parameterized query follows its parameter between pool groups beside an unchanged peer, and releases its subscription on unmount. mountControllable takes an optional pool, and each driver uses its adapter's config form for it (React without deps, Angular params). Pooling every query by default, a pooled identity that ignores literal values, and an Angular params form that drops pool each fail them. Co-authored-by: Isaac <no-reply@databricks.com>
CI does not build the packages before lint. A subpath import such as `@tanstack/db-sqlite-persistence-core/broadcast-coordinator` resolved to dist/ locally and to nothing in CI, so one skill example failed only in CI. Each export in a package's manifest now maps to its source file. Co-authored-by: Isaac <no-reply@databricks.com>
- groupKeys() compiles without noImplicitAny, as join operators do. Both use one OperatorCallback type. The TypeScript-modes test now covers groupKeys and a reusable join operator, which passes its context explicitly and stays typed in both modes. - Document reusable join operators in the guide and the db-core skill. - Migrate the React Native shopping-list example to include() and the projects README to the current join API. - The changeset and skill say that the join methods are removed. They do not throw the coded error. The changeset notes the new type of resolveLiveQueryValue's pool option. - The bundle test runs every operator value alone and every combination of operator families, and checks group-by.ts exactly. - The doc example check reports typos of imported TanStack names and examples that do not parse, and keeps parentheses in its fingerprint. - Correct ARCHITECTURE.md and includes-runtime.ts on which module owns public-key reduction. - Remove dead code: containsAggregate, the pool key check in poolConfig, the joined-key parameter of materializeCompilation, and two needless exports. - Record the open load-subset replay counterexample (seed 1593300187), which fails the same way on main, in the coverage map. Co-authored-by: Isaac <no-reply@databricks.com>
- The TypeScript-modes test covers every operator and form: join and groupKeys, inline, reusable with context, and reusable without context, in both modes, with a stated result per cell. It compiles one program per mode, so it stays fast. - Document the typed reusable groupKeys<Context>() form. - Export JoinCondition and OperatorCallback, so a factory can name the condition type. - The doc example check reads examples through the TypeScript syntax tree: JSX text no longer breaks the fingerprint, every import form is found, and only syntax errors count as "does not parse". The typo check ignores case-only and prefix differences. A self-test with positive and negative cases runs before each check. - Regenerate the API reference. - Record the untraced includes-oracle failure, and the second seed of the load-subset counterexample, in the coverage map. Co-authored-by: Isaac <no-reply@databricks.com>
Joins are common, and models already know `.join(source, on)` and the `.leftJoin()` family. Making them opt-in saved about 3.6 kB in apps that never join. Grouping, includes, and pooling stay opt-in. - Restore `.join(source, on, type)` and the `leftJoin`, `rightJoin`, `innerJoin`, and `fullJoin` methods. Remove the join operator values, `JoinCondition`, the join runtime on IR join clauses, and `JoinOperatorRequiredError` (245). - Restore the static join compile path, main's optimizer, and main's joins.ts. Public-key reduction stays its own module, so a joined query with a custom key still does not bundle the include materializer. - Tests: a reverse codemod restored main's exact join call at each site. 25 files are identical to main again. - Docs and skills use the existing join API again. The doc example check matches a fence by its backtick count, so a ````ts example that holds a ``` block is read whole. - The TypeScript-modes test covers groupKeys only, and the bundle test treats join modules as core. Typical app: 102,023 to 94,419 B gzip (-7.5%). Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts: # .github/workflows/pr.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Every app bundled the code for grouping, includes, and live-query pooling, even when it used none of them. The query builder and the compiler imported that code statically. This change makes these three features opt-in. A feature's code arrives with the value that names it, so an app bundles only the features it imports. Joins stay in core and keep their API:
.join(source, on, type)and the.leftJoin(),.innerJoin(),.rightJoin(), and.fullJoin()methods do not change.count,sum,avg,min,max,groupKeysgroupBywith no aggregate takesgroupKeys(...)include,toArray,materializeissues: q.from(...)becomesissues: include(q.from(...))pooledLiveQueriespool: pooledLiveQueriesin the hook's query configHow it works:
groupKeys()carry the grouping compiler.orderByrewrites aggregates only when it finds one.includes-runtime.ts). They live ininclude-functions.ts, apart from the expression functions that core modules import.pooledLiveQueriescarries the pooled view and its identity shortcut. Hooks take it aspoolin their query config (LiveQueryHookConfig). Angular's{ params, query }form takes it too. React Suspense andDbClientqueries passpool: falseand never pool. ADbClientquery dropspoolfrom its identity, so a server preload and a client hook hash alike.GroupingOperatorRequiredError(246) for agroupBywith no aggregate and nogroupKeys()IncludeOperatorRequiredError(247) for a subquery inselect()without an include function. A bare subquery inselect, also in a nested object or acaseWhenbranch, is also a type error.include()of something that is not a query builder.How a query with a custom key reaches public-key reduction:
reducePublicKeysfrompublic-key-reduction.ts. It does not bundle the include materializer or create a facade adapter. Onmain, this query went through the include materializer.An earlier version of this PR also made joins opt-in, with
.join(leftJoin(source, on)). That saved about 3.6 kB in apps that never join. The maintainer reverted that part: joins are common, and models already know the existing API.The architecture document records the include contract and how values carry operator code.
Size
pnpm size:compareagainstorigin/main(gzip bytes):app(query collection, filtered live query, ReactuseLiveQuery)app-joinapp-group-byapp-includesdbpackage bundle, every export keptThe package-bundle figure is the cost of the indirection when an app uses every feature.
Evidence
This change moves behavior in areas that oracles own: query compilation, includes, pooling, and the adapter bindings. Each law below lists its evidence.
1. Pooling is opt-in per query
pool: pooledLiveQueries. Every other query has its own subscription. A pooled query with a parameter follows the parameter. Authority: the maintainer's design decision for this release (pooling off by default, opt-in by import).main, becausemainhas nopooloption. These mutants take the place of the old behavior:pool-opt-infails in React and Angular.pooled-param-changefails in React, and the oldereq-filter-peersscenario does not catch it.paramsform that dropspool:pooled-param-changefails.npx vitest run tests/conformancepasses in each adapter (React 85, Vue 87, Solid 44, Svelte 84, Angular 46).eqfilters on one field. The pooled live-query oracle owns the membership and order laws.2. An app bundles an operator's code only when it imports the operator
opt-in-operator-bundle.test.tsruns each of the 10 operator values alone and all 8 combinations of the three families, with rotating values (18 bundles). These are fixed cases, not generated histories, so ORC-015 does not apply.group-by.tswith a size limit under 1,000 bytes. The first mutant below passed it.group-by.tshelper from the live-query Collection builder fails 8 of 18 cases. A static import ofBucketFacadeAdapterinto the builder fails the absence checks.@tanstack/dbwith other packages external, so it does not cover db-ivm operators or framework adapters.pnpm size:comparemeasures bytes.3. A
groupKeys()callback compiles in both TypeScript modesgroupKeys()gets the query's context from.groupBy(). Inline, its callback is typed in strict mode and untyped withoutnoImplicitAny, as other unannotated callbacks are. Group keys that pass their context as a type argument,groupKeys<ContextFromSource<…>>(…), are typed in both modes. Authority: the maintainer's decisions in this PR's review.group-keys-typescript-modes.test.tscompiles a wrong field in three forms: inline, reusable with context, and reusable without context. It compiles each form in both modes and checks the stated result. A fixture with valid code must compile in both modes. These are fixed cases, not generated histories.groupKeys()fallback, the valid fixture fails withoutnoImplicitAny("'user' is possibly 'undefined'").OperatorCallbackthat is always untyped fails the cases that must report the wrong field.4. Examples in the docs and skills typecheck
docs/and in the shipped skills parses and typechecks against the workspace source, unless its fence saysnocheck.lefJoin. Others are code that does not parse, a missing default export, and a moved parenthesis. A scratch example with the app namescollectionandcounterpasses, as it must.pnpm test:doc-snippets: 0 new failures. Its self-test runs first, on every check.scripts/doc-snippet-syntax.mjs) fails when the typo check ignores case. It also fails when the fingerprint keeps the parentheses around JSX. The CI lint job once failed because a subpath import resolved to an unbuiltdist/. With everydist/hidden locally, the old script fails the same way.any. The check reports errors in the TanStack API, not in the example's own code. The typo check compares only with names that the example imports from a TanStack package.Other tests:
include()andgroupKeys(). The assertions did not change. When joins went back to core, a reverse codemod restoredmain's exact join call at each site, and 25 test files are identical tomainagain.groupKeys()grouping,include()Collections, pooling off by default, andpool: false.select, also in a nested object.Remaining limits:
poolcomputes the full IR hash of its query on each render. Measured on a two-conditionwherequery: building the query takes 1.1 µs, and building plus hashing takes 3.6 µs. The pooled identity adds almost nothing. For 1,000 mounted hooks, the hash adds about 2.5 ms for each render. Keeping a fast identity in core would bundle part of pooling in every app.load-subset-replay-refinement-oraclefails with seed1593300187, path121:1:2:3:2:2. The history is a held load,truncateMarkers, and a failed settle.mainat65d9ada50fails with the same counterexample. A full suite run on the branch found it again with seed-1005431615. The random campaign can fail CI intermittently until someone fixes the defect.includes-oraclerandom campaign failed on this branch without a recorded seed. Five dedicated runs and two full-suite runs passed later. The coverage map records it as open, with the witness it needs. This PR does not call it pre-existing.buildNestedSelect,include()checks the field name before the query.toArray()checks the query first. Both throw a coded error. Changing the order costs 8 bytes in every app, so this PR keeps it. The eight wrapper blocks remain, because the error-site oracle freezes one message and code for each site.query-db-collectionanddb-collection-e2e, and the link checker's 33 broken links, also occur onmain. The React Native example cannot typecheck in a workspace without built packages.✅ Checklist
@tanstack/db: 13,825 tests. One test timed out while other sessions loaded the machine, and it passes alone.@tanstack/react-db: 421@tanstack/vue-db: 134@tanstack/solid-db: 101@tanstack/svelte-db: 135@tanstack/angular-db: 76@tanstack/query-db-collection: 1,460@tanstack/db-sqlite-persistence-core: some sqlite3 CLI tests time out while the machine is loaded. Each one passes alone. This package differs frommainin two test files only.tscreports no new errors.eslintreports no errors.🚀 Release Impact
This is a breaking change for the first 1.0 release candidate. The changeset lists the migration for each feature.
This pull request and its description were written by Isaac.