Use unnest for bulk operations in corrosion #385
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gorbak25
force-pushed
the
gorbak/process_changes_opt
branch
3 times, most recently
from
695b182 to
07bde81
Compare
gorbak25
force-pushed
the
gorbak/process_changes_opt
branch
from
07bde81 to
825559b
Compare
somtochiama
reviewed
Nov 3, 2025
somtochiama
approved these changes
Nov 3, 2025
| last_rowids[i].2 = last_rowids[i + 1].2; | ||
| } | ||
| last_rowids[last_rowids_len - 1].2 = sp | ||
| .prepare_cached("SELECT last_insert_rowid()")? |
Contributor
There was a problem hiding this comment.
Nice workaround with last_insert_row_id !
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR uses #383 for efficient bulk operations on the
crsql_changes,__corro_buffered_changes,__corro_membersand bookie tables. To make writing array queries easier I've added theunnest_paramwhich allows to use unnest in queries without too much boilerplate.I've benchmarked the effect unnest had on full changes:
Previously in #377 (WARNING: The baseline chart has the red line on 50, now we use 100) the throughput values were platoing at 50k changes/s. With unnest that value went up to 70k changes/s with no plato in sight(probably perf will increase for even higher batch sizes). For the batch size we use perf went up slightly.
Larger changes with regards to buffered changes will be done in an separate PR.