Query Driven Sync with On-Demand Collection Loading #763
+16,691
−728
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 471fa8f The changes in this PR will be included in the next version bump. This PR includes changesets to release 13 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-ivm
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
Contributor
|
Size Change: +5.99 kB (+7.49%) 🔍 Total Size: 86 kB
ℹ️ View Unchanged
|
Contributor
|
Size Change: 0 B Total Size: 3.34 kB ℹ️ View Unchanged
|
samwillis
force-pushed
the
query-driven-sync
branch
from
4e77602 to
685af96
Compare
33 tasks
7 tasks
This was referenced Nov 10, 2025
) * Add unit tests that reproduce the problems with moveWindow on infinite limits * Handle Infinity limit in move * Changeset * Add support for orderBy and limit options in currentStateAsChanges * Unit tests for currentStateAsChanges * changeset * fix for eslint * Move helper functions to the end of the file * add predicate utils * optimise in when all primatives * optimisations * changeset * minusWherePredicates * Add unit test for OR in both subset and superset that shows bug with OR handling * Reorder OR handling to fix bug and handle AND similarly * change chageset * dedupe code * add createDeduplicatedLoadSubset function * convert deduper to a class, with reset, and dedupe inflight * Simplify matching of inflight requests * Improve minusWherePredicates such that it can handle common conditions in a conjunction * Modify DeduplicatedLoadSubset.loadSubset such that it only loads the missing data * Remove unused predicate helper functions * Fix eslint issue * Prettier * Unify handling of IN and OR * Handle subset that is false * Fix isLimitSubset to correctly handle unlimited subsets --------- Co-authored-by: Kevin De Porre <[email protected]>
* Callback that informs about deduplicated loadSubset calls * Unit tests for onDeduplicate callback + move dedupe test files to the right folder * changeset * Object params * Updated lockfile * fix lock file --------- Co-authored-by: Sam Willis <[email protected]>
* Handle pushed down predicates in query collection Co-authored-by: Kevin De Porre <[email protected]> Co-authored-by: Sam Willis <[email protected]> * Deduplicate loadSubset requests in Query collection * Unit test for deduplicating limited ordered queries in query collection * GC tanstack query when subscription is unsubscribed --------- Co-authored-by: Sam Willis <[email protected]> Co-authored-by: Kevin De Porre <[email protected]>
* Handle pushed down predicates in Electric collection Co-authored-by: Kevin De Porre <[email protected]> Co-authored-by: Sam Willis <[email protected]> * use the subsetDuduper for electric * Leave fixme * fix DeduplicatedLoadSubset call * fix tests --------- Co-authored-by: Sam Willis <[email protected]>
#762) * Add optional compareOptions to the Collection Config which act as collection-wide config and allow queries to override those. * Fix import * changeset * Ran prettier * update docs * Live queries inherit compare options from collection in the FROM clause * Explicit compare options for live queries * Rename compareOptions to defaultStringCollation in the collection config and live query collection config * Docs update --------- Co-authored-by: Sam Willis <[email protected]>
…#786) * feat: add expression helpers for parsing LoadSubsetOptions in queryFn Add reusable expression parsing utilities to help developers translate TanStack DB predicates (where, orderBy, limit) into their API's format. - Add expression-helpers.ts with generic parsing utilities - parseWhereExpression: Parse where clauses with custom handlers - parseOrderByExpression: Parse order by into simple array - extractSimpleComparisons: Extract simple AND-ed filters - parseLoadSubsetOptions: Convenience function for all options - walkExpression, extractFieldPath, extractValue: Lower-level helpers - Export helpers from query-db-collection package - Add comprehensive documentation to query-collection.md - How LoadSubsetOptions are passed via ctx.meta - Expression helper usage with REST and GraphQL examples - API reference for all helper functions - Tips and best practices This makes it much easier to implement query collections with predicate push-down without having to manually parse expression AST trees. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * test: add comprehensive tests for expression helpers - 35 test cases covering all helper functions - Tests for parseWhereExpression with various operators - Tests for parseOrderByExpression with sorting options - Tests for extractSimpleComparisons - Tests for parseLoadSubsetOptions - Tests for low-level helpers (extractFieldPath, extractValue, walkExpression) - Integration test for complex real-world query scenarios All tests passing with 93.65% coverage of expression-helpers.ts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * chore: add changeset for expression helpers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * docs: make GraphQL example more generic Remove Hasura-specific emphasis in GraphQL example section. The underscore-prefixed operators are common GraphQL conventions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * docs: clarify GraphQL format is not prescriptive Change 'the GraphQL format' to 'a GraphQL format' to avoid implying this is the only way to structure GraphQL queries. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * docs: remove Electric DB reference from tips Remove implementation-specific reference to Electric DB Collection from the tips section. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * refactor: make loadSubsetOptions always an object - Add default empty object parameter to createQueryFromOpts - Remove ?? {} pattern from all documentation - Update changeset example - Update PR description This makes the API cleaner - users can now access ctx.meta.loadSubsetOptions directly without null coalescing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * fix: add explicit type annotations to avoid implicit any - Import BasicExpression and OrderBy from IR namespace - Add type annotations to all lambda parameters - Remove unnecessary type checks that were always true - Fix variable shadowing in walkExpression visitor parameter - Fixes build type errors and eslint warnings in expression-helpers.ts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * style: fix prettier formatting in changeset 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * fix: correct TypeScript types in expression-helpers tests * feat: address Sam's review feedback - Add OperatorName type and operators constant to db package - Improve ParseWhereOptions handlers type to know IR operators - Support array indices in FieldPath (string | number) - Add locale/string collation options to ParsedOrderBy - Make extractSimpleComparisons throw on unsupported operations (or, not, etc.) - Update tests to expect throws instead of silent skipping * fix: remove invalid sensitivity/locale fields from test mocks CompareOptions uses StringCollationConfig which has stringSort, locale, and localeOptions fields, not sensitivity and locale at the top level. * fix: handle discriminated union for CompareOptions properly Use 'in' operator to check for optional fields (locale, localeOptions) that only exist when stringSort is 'locale' * move helpers to main db package --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Sam Willis <[email protected]>
samwillis
force-pushed
the
query-driven-sync
branch
from
f7b3988 to
55ea32e
Compare
* Implement end-to-end tests for sync feature Implement end-to-end tests for sync feature Implement end-to-end tests for sync feature chore: revert pnpm-lock.yaml to match query-driven-sync chore: update pnpm-lock.yaml with e2e test pg dependencies bump pnpm/action-setup version fix pnpmlockfile format only run normal tests on test command tidy and fix types fixes ci fix tidy checkpoint cleanup between each test add cleanup between tests all e2e tests pass! cleanup live queries fixes for ci add propper mutations to e2e tests fix ci timouts move the badly named accumulated data tests fix ci flake working query e2e tests fix lint fix assertions Fix race condition between test and logical replication stream Handle isNull and isUndefined in expression helpers Ran prettier. fix ci update docs add suport for not to extractSimpleComparisons and get tests to pass try and fix ci race * increase timeout
Contributor
|
🎉 This PR has been released!
Thank you for your contribution! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a catch all branch/PR for all PR's implementing RFC: On-Demand Collection Loading via loadSubset
Must be merged to main with a rebase to capture all the commits in this branch
To merge: