Add transport-filecoin-piece-http

[rvagg]

Intended for IPNI registrations: provider X has piece Y available on transport-filecoin-piece-http

[aschmahmann]

@rvagg seems reasonable, although is there a reason this is necessary beyond just the trustless gateway protocol for raw blocks? At the moment it seems like the same thing with a different endpoint.

[rvagg]

At the moment it seems like the same thing with a different endpoint

True, but we're reflecting existing realities here. Pieces are served exclusively via /piece/ which has its own protocol rules (relatively simple), inner IPLD blocks within a piece are served via /ipfs/ which have very different protocol rules. Byte range retrievals are probably the most important difference here being byte range requests for raw blocks. I suppose we could make sure that dat-scope=block supports standard http byte range requests and then we'd have ~compatibility (albeit with the additional need for a query string). But for now, you can at least use /piece/ as a standard HTTP retrieval endpoint for large blobs, so you can, for example, point a video player at it and let it do its own byte range retrievals.

Perhaps we could work toward a convergence so it doesn't matter? But at that point, transport-filecoin-piece-http could mean that the piece is available on both endpoints, whereas transport-ipfs-gateway-http just means it's available on /ipfs/.

[hsanjuan]

Is /piece retrieval ("protocol rules") specified anywhere? (I cannot readily find anything other than mentions in the boost docs).

I don't understand the byte-range support thing... does /piece retrieval allow byte ranges but then the the data cannot be verified after download even though it was requested by CID ? Is that a usecase?

[masih]

The "usecase" for the purposes of this PR is to have an entry in the CSV lookup table. That is all. Let us not try to solve everything everywhere all at once.

I don't think there is an FRC for /piece endpoint but there should be one. FWIW the protocol for piece retrieval via byte range request has existed in Boost from day one and continutes to exist in Curio. Those two implementations are by far the most popular Filecoin SP software out there.

[hsanjuan]

It seemed reasonable to ask if it is written somewhere what transport-filecoin-piece-http actually means. 🤷

[masih]

Here is the FRC

@rvagg please feel free to link it in the description of codec if it makes sense.

If there is anything else that's blocking this PR from being landed, please state explicitly. Otherwise, let's keep things moving.

[aschmahmann]

In the interest of not keeping anyone who is antsy here about how long approval will take I've approved the PR so @rvagg can make any changes and merge it at convenience.

A few things of note:

Here is the FRC

Is that the relevant spec, or is it really a modification to the IPNI metadata table which would itself reference the FRC? My understanding is that nothing other than IPNI metadata uses these codes. Someone (perhaps @masih or @rvagg ) probably should update that metadata doc to describe what the expected the parameters for this code are and then @rvagg could include a link to it and/or FRC-66 in the table.

I suppose we could make sure that dat-scope=block supports standard http byte range requests and then we'd have ~compatibility (albeit with the additional need for a query string)

TLDR: If using the safe piece CIDs then only the path change is required.

It's not dag-scope=block that you'd want since that still a CAR response (it returns the path blocks + the one at the end). It's format=block (or the application/vnd.ipld.raw Accept header), or honestly in the case of Filecoin pieces addressed using the Filecoin Piece multihash the raw codec will end up being used and no format=block is even necessary. i.e. a straight path change from /piece to /ipfs would do the job when using the safe piece CIDs.

[masih]

There's no antsy-ness here. Filecoin frc is the spec.

What this pr does is that it specifies a code to identify the transport. It so happens that IPNI is the only user of that identifier today. That code can be used anywhere inline with multicodec philosophy.

[rvagg]

safe piece CIDs

You're talking about v2 I believe.

:ack: re format=block and raw etc., it's been too long. The other two concerns that make this still useful to identify a separate protocol/endpoint is (a) byte ranges and (b) the optionality of an SP not serving /ipfs/ if they don't need to be doing so. Another future extension here is merkle proofs for arbitrary byte ranges also being served via /piece/ (somehow, TBD). I still quite like the model you mapped out for doing so with small piece ranges via /ipfs/ and maybe we end up doing that, but there's also a desire to ship something that's more transparently suitable for standard HTTP clients and tools (where merkle proofs might be optional), so /piece/ might be the best place to iterate on that. But maybe we just end up with both, or we iterate /ipfs/ to be all we need, 🤷. I don't think using this entry rules out changing its meaning down the road.