How to read a single row group batch, given only the row group bytes and the schema bytes

[keller-mark]

Use case

I want to use readParquet to read data from a single row group, without loading the full parquet file's bytes.

readParquetStream does not sufficiently address this use case because 1) I want to access the row groups randomly (e.g., only need to access the Kth row group, not all of them in order) and 2) I want to retain the low-level control over accessing the parquet bytes (e.g., they are not always coming from a URL, the parquet bytes may be coming from a local file or they may be coming via some other protocol).

I tried manually concatenating the row group bytes with the schema bytes, and then passing to readParquet or arrow-js-ffi's parseRecordBatch, but neither worked. I assume the footer bytes contain references to byte positions/offsets that no longer make sense when the bytes are manually concatenated like this.

[kylebarron]

I don't think that's possible in the Parquet spec. The metadata in the Parquet spec references byte ranges into the original file, and if you only have the bytes of a single row group, those integers will no longer be valid

1. I want to access the row groups randomly (e.g., only need to access the Kth row group, not all of them in order)

Have you tried ParquetFile? https://kylebarron.dev/parquet-wasm/classes/esm_parquet_wasm.ParquetFile.html

ParquetFile.read allows you to specify specific row group indexes.

Perhaps we need to improve the documentation to show that there are three different APIs to read data: the synchronous full-buffer API, the stream API, and the async file API

[keller-mark]

I had a try, and got something almost working in https://github.com/keller-mark/parquet-wasm/tree/keller-mark/read-individual-row-group. For tables without dictionary-encoded columns, it seems to work.

It currently breaks when a Dictionary-encoded column value is encountered (the value is returned as null by arrow's tableFromIPC, while flechette's tableFromIPC throws an error), and i am having trouble figure out exactly why. Columns to the right of the dictionary-encoded column end up corrupted, but values in columns to the left seem fine.

I suspect it has to be one of the following issues:

• an issue with byte offsets during dictionary decoding on the Rust side (e.g., the absolute byte offsets are used somewhere, rather than ChunkReader.get)
• a lack of inclusion of dictionary bytes in the row batch when the JS side was expecting them
• unexpected dictionary spilling/preservation on the Rust side when the opposite was expected on the JS side
• an issue with byte offsets during parsing the tableFromIPC on the JS side

[kylebarron]

tables without dictionary-encoded columns, it seems to work.

I was thinking dictionary-encoded columns might be stored elsewhere in the Parquet file.

That said, I really don't think this is a good public API. It's too low level and very hard to use correctly. I think it would be a better use of effort to improve the existing async APIs for loading files without loading the entire content.

[keller-mark]

I think it would be a better use of effort to improve the existing async APIs for loading files without loading the entire content.

Would you be in favor of an approach like ParquetFile.fromAsyncBuffer where the AsyncBuffer object matches that used by hyparquet?

interface AsyncBuffer {
  byteLength: number
  slice(start: number, end?: number): Awaitable<ArrayBuffer>
}

then, ParquetFile.read({ offset: rowOffset, limit: 1 }) could be used to read a single row, assuming it only requests the minimal amount of bytes via AsyncBuffer.slice (such as those of a single row group or the subset of data/dict pages).

[kylebarron]

What does this gain over the existing File / Blob support?

[keller-mark]

I have a bit of an odd use case where I want to make the range requests for bytes of a parquet file via a zarr store's store.getRange function https://github.com/manzt/zarrita.js/blob/5fd73b66e4856f4b1d001b01a0c1baa47e297b2f/packages/%40zarrita-storage/src/fetch.ts#L87 I also could envision this resolving #705 (e.g., if the user has full control over the requests, they can pass custom headers or other fetch parameters).

[kylebarron]

We can expose this trait, which is essentially what you want.

An initial implementation would expose just the range request interface, then later we could add support for cached metadata, etc, through the get_metadata function.