[wip] Add Catalog Hash System for Dependency Tracking #3006

Myranae · 2025-10-30T19:44:18Z

Summary:

Overview
Implements a catalog hash system to track changes in pnpm catalog dependencies (Wonder Blocks, React, etc.) across Perseus packages. This ensures packages are correctly rebuilt when their catalog dependencies are updated, even if the package code itself hasn't changed.

Key Features

Automatic Hash Updates

pnpm update-catalog-hashes - CLI command to update all catalog hashes
pnpm update-catalog-hashes --dry-run - Preview changes without writing
pnpm update-catalog-hashes --verbose - Show detailed dependency information

Integrated into Existing Workflows

Sync Script: utils/sync-dependencies.ts now automatically updates catalog hashes after syncing
Pre-publish Check: utils/pre-publish-check-ci.ts verifies all hashes are current before publishing

Smart Filtering

Only includes prodDeps and peerDeps catalogs (excludes devDeps)
Skips unpublished packages (root package, packages in changeset ignore list)
Only processes packages in the packages/ directory

Non-Test Files Added

utils/get-catalog-deps-hash.ts - Hash calculation logic
utils/maybe-update-catalog-hash.ts - Single package update logic
utils/update-catalog-hashes.ts - Main orchestration function
utils/update-catalog-hashes-cli.ts - CLI entry point
utils/verify-catalog-hashes.ts - Pre-publish verification

Files Modified

package.json - Added string-hash dependency and update-catalog-hashes script
README.md - Added documentation for catalog hashes
utils/sync-dependencies.ts - Auto-update hashes after syncing
utils/pre-publish-check-ci.ts - Verify hashes before publishing
All 12 packages in packages/ - Added initial catalogHash values

Dependencies

Added: [email protected] - Deterministic hash function (same as used in frontend)
Added: @types/[email protected] - TypeScript definitions

Issue: LEMS-3328

Test plan:

Confirm all checks pass
Manually run sync-deps after pulling Frontend and see if hashes are updated (only works if there are new depdencies)

Using this for consistency

github-actions · 2025-10-30T19:46:23Z

🗄️ Schema Change: No Changes ✅

github-actions · 2025-10-30T19:48:17Z

Size Change: 0 B

Total Size: 497 kB

ℹ️ View Unchanged

Filename	Size
`packages/kas/dist/es/index.js`	20.8 kB
`packages/keypad-context/dist/es/index.js`	1 kB
`packages/kmath/dist/es/index.js`	5.98 kB
`packages/math-input/dist/es/index.js`	99.2 kB
`packages/math-input/dist/es/strings.js`	1.61 kB
`packages/perseus-core/dist/es/index.item-splitting.js`	13.1 kB
`packages/perseus-core/dist/es/index.js`	22.4 kB
`packages/perseus-editor/dist/es/index.js`	96.6 kB
`packages/perseus-linter/dist/es/index.js`	7.21 kB
`packages/perseus-score/dist/es/index.js`	9.2 kB
`packages/perseus-utils/dist/es/index.js`	403 B
`packages/perseus/dist/es/index.js`	204 kB
`packages/perseus/dist/es/strings.js`	7.73 kB
`packages/pure-markdown/dist/es/index.js`	1.39 kB
`packages/simple-markdown/dist/es/index.js`	6.71 kB

_{compressed-size-action}

github-actions · 2025-10-30T19:48:18Z

🛠️ Item Splitting: No Changes ✅

… dependency tracking

github-actions · 2025-10-30T19:48:58Z

npm Snapshot: Published

Good news!! We've packaged up the latest commit from this PR (60d98d2) and published it to npm. You
can install it using the tag PR3006.

Example:

pnpm add @khanacademy/perseus@PR3006

If you are working in Khan Academy's frontend, you can run the below command.

./dev/tools/bump_perseus_version.ts -t PR3006

If you are working in Khan Academy's webapp, you can run the below command.

./dev/tools/bump_perseus_version.js -t PR3006

jeremywiebe

Thanks for this Tamara! I have a few thoughts before you land this:

we duplicate types in several places, could we centralize those so that they can't drift out of sync?
the utils/ folder has an internal/ subfolder. Would some of these files belong in there? I think this utils/ folder should only contain executable files and any supporting files could go into internal/.
The PR description is a bit verbose. I don't think there's value in the description outlining changed files as that is data that git will give us. It would also be nice if you could state the problem this PR solves at the outset. I think it's solving the problem of packages not being published when we bump catalog dependencies, right? Could you make that clearer in the description?

jeremywiebe · 2025-10-31T18:18:20Z

utils/update-catalog-hashes-cli.ts

+import {updateCatalogHashes} from "./update-catalog-hashes";
+
+function printHelp() {
+    console.log("Usage: update-catalog-hashes [options]");


Suggested change

console.log("Usage: update-catalog-hashes [options]");

console.log("Usage: update-catalog-hashes-cli.ts [options]");

jeremywiebe · 2025-10-31T18:19:25Z

utils/update-catalog-hashes-cli.ts

+    console.log("Examples:");
+    console.log("  update-catalog-hashes");
+    console.log("  update-catalog-hashes --dry-run");
+    console.log("  update-catalog-hashes --verbose");


These aren't useful unless they show more complex usages of flags. As it stands, there's only two flags and they're pretty common, so examples here seems like just extra noise.

jeremywiebe · 2025-10-31T18:24:24Z

utils/update-catalog-hashes.ts

+ */
+export function updateCatalogHashes(isDryRun: boolean, verbose = false): void {
+    // Get the set of packages that we don't publish, per the changeset config.
+    const {ignore = []} = loadChangesetConfig();


I would prefer we look at each package.json's private flag and/or the publishConfig. These are npm-ecosystem flags that document if the package should be published. Although the ignore config option is supported, they document that it is meant to be a temporary solution during dev.

THIS FEATURE IS DESIGNED FOR TEMPORARY USE TO ALLOW CHANGES TO BE MERGED WITHOUT PUBLISHING THEM - If you want to stop a package from being published at all, set private: true in its package.json.

jeremywiebe · 2025-10-31T18:28:15Z

utils/update-catalog-hashes.ts

+    const packageJsonPaths = fastGlob.sync("**/package.json", {
+        absolute: true,
+        ignore: ["**/node_modules/**", "**/dist/**"],
+    });


This works fine, but one option that would be easier to maintain (and not require maintaining an ignore list is to use git to figure out which files to work with. Only package.json files that are checked into git should be considered... so git ls-files "package.json" "**/package.json" would find all package.json files that are tracked... meaning we'd ignore everything in any node_module/ or dist/ folder automatically.

jeremywiebe · 2025-10-31T18:30:47Z

utils/update-catalog-hashes.ts

+    const unpublishedPackages = new Set<string>(ignore);
+    // The root package is always considered unpublished,
+    // but is not included in the changeset config ignore list.
+    unpublishedPackages.add(ROOT_PACKAGE_NAME);


We won't need this if we change to obey the package's private: true flag as our root package.json has this (correctly) set.

jeremywiebe · 2025-10-31T18:45:20Z

utils/pre-publish-check-ci.ts

+        console.log("\n🔍 Verifying catalog hashes...");
+        const catalogHashResult = verifyCatalogHashes();
+        if (!catalogHashResult.success) {
+            console.error("\n❌ Catalog hash verification failed:\n");


Why do we prefix these messages with an extra newline (\n)?

jeremywiebe · 2025-10-31T18:50:24Z

utils/verify-catalog-hashes.ts

+
+const ROOT_PACKAGE_NAME = "perseus";
+
+type PackageJson = {


It looks like we duplicate this type in at least one other file. Maybe we should have a types.ts file in this utils/ folder that hosts these types?

jeremywiebe · 2025-10-31T18:52:13Z

README.md

+Catalog hashes ensure packages are rebuilt when their catalog dependencies (Wonder Blocks, React, etc.) are updated. These hashes are automatically updated when running `utils/sync-dependencies.ts`. If you manually add catalog dependencies to a package.json, run `pnpm update-catalog-hashes` to update the hashes. The pre-publish check will verify all hashes are current before releasing.
+


Nice, this is a mostly clear explanation, thanks!

Just to check my understanding, are these hashes used to trigger a publish of affected packages or just rebuilding (I'm not trying to be pedantic, but I'm unclear what we mean when we say "rebuilt")

jeremywiebe · 2025-10-31T19:02:22Z

package.json

        "sloc": "^0.2.1",
        "storybook": "^9.0.4",
        "storybook-multilevel-sort": "^2.0.1",
+        "string-hash": "^1.1.3",


FWIW, we could use globalThis.crypto.subtle to do the hashing (via digest) as that's built-in to Node.

jeremywiebe · 2025-10-31T19:08:21Z

utils/__tests__/get-catalog-deps-hash.test.ts

+        // Assert - Empty string hashes to "5381" with string-hash
+        expect(result).toBe("5381");


thought: We document this "magic" value in several of these tests. I wonder if it'd be clearer if we had a known string value that we use if the input string is empty. ie. what if this assertion were: expect(result).toBe("--empty--").

What are your thoughts? :)

jeremywiebe

Thanks for this Tamara! I have a few thoughts before you land this:

we duplicate types in several places, could we centralize those so that they can't drift out of sync?
the utils/ folder has an internal/ subfolder. Would some of these files belong in there? I think this utils/ folder should only contain executable files and any supporting files could go into internal/.
The PR description is a bit verbose. I don't think there's value in the description outlining changed files as that is data that git will give us. It would also be nice if you could state the problem this PR solves at the outset. I think it's solving the problem of packages not being published when we bump catalog dependencies, right? Could you make that clearer in the description?

Myranae added 10 commits October 30, 2025 10:20

[tb/add-catalog-hashing] Add string-hash library

5dae2e5

Using this for consistency

[tb/add-catalog-hashing] Add logic to create and add catalog hashes

10d0801

[tb/add-catalog-hashing] Add the very first catalog hashes

740bba6

[tb/add-catalog-hashing] Add the script to the package.json

fc5f17e

[tb/add-catalog-hashing] Update sync script to update hashes too

9367e79

[tb/add-catalog-hashing] Lint updates

b832bd0

[tb/add-catalog-hashing] Add verify catalog hashes to pre-publish checks

cb7a40c

[tb/add-catalog-hashing] Update the README

f0ec82d

[tb/add-catalog-hashing] Update to ignore devDeps

ec05a49

[tb/add-catalog-hashing] Add tests for hash logic and updates

f0c43db

Myranae self-assigned this Oct 30, 2025

Myranae added the olc-5.0.c60e1 label Oct 30, 2025

Myranae changed the title ~~Add Catalog Hash System for Dependency Tracking~~ [wip] Add Catalog Hash System for Dependency Tracking Oct 30, 2025

github-actions bot added item-splitting-change and removed item-splitting-change labels Oct 30, 2025

[tb/add-catalog-hashing] docs(changeset): Add catalog hash system for…

de71187

… dependency tracking

[tb/add-catalog-hashing] Update linting

60d98d2

jeremywiebe approved these changes Oct 31, 2025

View reviewed changes

jeremywiebe reviewed Oct 31, 2025

View reviewed changes

	console.log("Usage: update-catalog-hashes [options]");
	console.log("Usage: update-catalog-hashes-cli.ts [options]");

		Catalog hashes ensure packages are rebuilt when their catalog dependencies (Wonder Blocks, React, etc.) are updated. These hashes are automatically updated when running `utils/sync-dependencies.ts`. If you manually add catalog dependencies to a package.json, run `pnpm update-catalog-hashes` to update the hashes. The pre-publish check will verify all hashes are current before releasing.

		// Assert - Empty string hashes to "5381" with string-hash
		expect(result).toBe("5381");

[wip] Add Catalog Hash System for Dependency Tracking #3006

Are you sure you want to change the base?

[wip] Add Catalog Hash System for Dependency Tracking #3006

Uh oh!

Conversation

Myranae commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary:

Test plan:

Uh oh!

github-actions bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🗄️ Schema Change: No Changes ✅

Uh oh!

github-actions bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛠️ Item Splitting: No Changes ✅

Uh oh!

github-actions bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

npm Snapshot: Published

Uh oh!

jeremywiebe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremywiebe left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Myranae commented Oct 30, 2025 •

edited

Loading

github-actions bot commented Oct 30, 2025 •

edited

Loading

github-actions bot commented Oct 30, 2025 •

edited

Loading

github-actions bot commented Oct 30, 2025 •

edited

Loading

github-actions bot commented Oct 30, 2025 •

edited

Loading