fix: do not download all blocks when listing dir

[SgtPooki]

Title

fix: do not download all blocks when listing dir

Description

Fixes #260
Related ipfs/helia#836
Related ipfs/js-ipfs-unixfs#439
Related ipfs/js-ipfs-unixfs#440

Notes & open questions

Two things:

1. There are multiple places where we could potentially use the extended: false option.
2. Currently there seems to be a bug or problem with using the ipfs-unix-fs exporter with the extended: false option.

Other places we may want to use extended: false

There are multiple places where we use unix-fs-exporter.. these were added and commented out in the initial commit in this PR: e4d7a15 (#261)

./packages/verified-fetch/src/utils/walk-path.ts

const entry of exporterWalk(path, blockstore, options) do we need to call it here?

./packages/verified-fetch/src/plugins/plugin-handle-dag-pb.ts

places where we call () => exporter

Bug with extended: false

Looking at these lines in plugin-handle-dag-pb:

helia-verified-fetch/packages/verified-fetch/src/plugins/plugin-handle-dag-pb.ts

Lines 96 to 101 in cd5a103

	const entry = await handleServerTiming('exporter-dir', '', async () => exporter(`/ipfs/${dirCid}/${rootFilePath}`, helia.blockstore, {
	signal: options?.signal,
	onProgress: options?.onProgress
	}), withServerTiming)
	log.trace('found root file at %c/%s with cid %c', dirCid, rootFilePath, entry.cid)

When extended: true, and added a console.log for entry,I get the following output:

  helia:verified-fetch:dag-pb-plugin:trace found directory at bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm/, looking for index.html +0ms
entry {
  type: 'file',
  name: 'index.html',
  path: 'bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm/index.html',
  cid: CID(bafybeig6e22n6aozgcfdqlh3ysctvsfwrqi7qa6u4qrafb5pzn3keke6ci),
  content: [AsyncGeneratorFunction: yieldFileContent],
  unixfs: UnixFS {
    type: 'file',
    data: undefined,
    blockSizes: [ 2n, 2n ],
    hashType: undefined,
    fanout: undefined,
    mtime: undefined,
    _mode: 420,
    _originalMode: 0
  },
  depth: 1,
  node: {
    Data: Uint8Array(8) [
       8, 2, 24, 4,
      32, 2, 32, 2
    ],
    Links: [ [Object], [Object] ]
  },
  size: 4n
}
  helia:verified-fetch:dag-pb-plugin:trace found root file at bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm/index.html with cid bafybeig6e22n6aozgcfdqlh3ysctvsfwrqi7qa6u4qrafb5pzn3keke6ci

When extended: false, I get the following:

  helia:verified-fetch:dag-pb-plugin:trace found directory at bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm/, looking for index.html +0ms
entry {
  cid: CID(bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm),
  name: 'bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm',
  path: 'bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm'
}
  helia:verified-fetch:dag-pb-plugin:trace found root file at bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm/index.html with cid bafybeifgpc3zqsh3ftf6u65rwuid2okrc75f7ojxipzyzlfkepzfxhlavm

The exporter is not reporting on the path passed to it.. instead taking the root and seeming to stop. This is tested with:

DEBUG="-mocha*,*,*:trace" npm run test -- -t node -g 'should abort a request while loading file data'

in /packages/verified-fetch

At the most, this seems like a bug.. at the least, it’s unexpected that passing extended: false would change the CID the response is giving back

In summary:

Passing extended: false on a path that exists, <CID>/index.html in this case, to the exporter returns only the CID of the directory, not of the file at the path I give it.

Change checklist

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation if necessary (this includes comments as well)
• I have added tests that prove my fix is effective or that my feature works

[SgtPooki]

self review

[SgtPooki]

we no longer have the sizes.. we will need to either fetch them with stat or decide on a better default.

[parkan]

@SgtPooki can I help resolve this with feedback from a user perspective?

[SgtPooki]

change in ipfs/js-ipfs-unixfs#440 should fix passing extended: false not returning what we expect

[SgtPooki]

@SgtPooki can I help resolve this with feedback from a user perspective?

@parkan I'm happy to hear about your user perspective, but I think we have the solution in hand.

[SgtPooki]

we chatted about this in ipfs/helia#850 and just setting extended: false in the fall to fs.ls should be sufficient

[SgtPooki]

currently running into issues with gateway conformance hanging.. i'm investigating

[SgtPooki]

tested this branch vs main (using npm run build && DEBUG="*,*:trace" node packages/gateway-conformance/dist/src/demo-server.js).

@parkan let me know if anything seems off here.. Ideally we would add some configurability to the dir-index-html plugin to support getting the sizes of children when necessary..

This PR:

Server timing (1.78s to load large dir)

Screenshot

Main:

Server timing (times out due to basic-server 10s timeout)

Screenshot

[github-actions]

🎉 This PR is included in version @helia/verified-fetch-v3.2.2 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[parkan]

beautiful! will test and report but the numbers are already looking good 👍