Support a parquet-based table access method as a Postgres table storage type

[CC-Hsu]

HI, Team,

As far as I know, parquet is a good data storage format for storing compressed data in column-oriented layout.

It would be great if pg_parquet can have a table access method to extend Postgres to have a high performance column-orient compressed table file format.

Best Regards.

[aykut-bozkurt]

pg_parquet aims at copying to/from regular Postgres tables for migration purposes. We might consider adding a primitive scan only fdw for performant querying of parquet files.

[pmetras]

https://github.com/adjust/parquet_fdw already has all the logic to query Parquet files from PostgreSQL. Would it be difficult to convert C++ code to Rust?

Querying Parquet files without copying the data is an important feature. I have multi-gigabytes Parquet files that I don't want to copy and create to multi-terabytes tables into PostgreSQL. Being able to query the Parquet files and join them with PostgreSQL tables gives me the best of both world.

[AndrewJackson2020]

You may also want to look into pg_duckdb as well. Supports querying parquet files from postgres via an in process duckdb instance.

https://github.com/duckdb/pg_duckdb

[CC-Hsu]

Hi, AndrewJackson2020,

Thanks for point out this, I'll take a look on pg_duckdb. 😊

[pmetras]

I think that pg_duckdb can't be used to join data from Parquet files with native PostgreSQL relations and that's the reason why I used paqruet_fdw. If your goal is only to read and query Parquet files, use Duckdb directly.