Feature Request - support an option to ignore invalid files

[baumand-amazon]

Trino will fail an entire query if there's a single bad file. For example if files are gzipped and there's a gzip file with an incomplete final chunk in the file.

If you have full control of the data being queried and can sanitize it to remove all such files then you can work around this issue but that is not always possible.

Trino could support this use case by allowing for an option to ignore malformed files.
Even better - an option specifically for gzipped files to search whatever portion of the file can be read.

A smaller feature than the above but one that would help significantly in these cases would be to return the specific file causing the problem in the error message - https://github.com/trinodb/trino/blob/421/plugin/trino-hive/src/main/java/io/trino/plugin/hive/GenericHiveRecordCursor.java#L236.

Today its difficult to locate the file that's causing the problem. And once located - one still can't use Trino for queries over all the files until all the invalid files are cleared or fixed in some way.

[raunaqmorarka]

You can use a predicate on $path column to filter out certain files.
The error message should certainly be improved to point out the problematic file, but I suggest you try it on a new release since the code you're pointing to has changed significantly.

[baumand-amazon]

Thanks Raunaq. Using the $path workaround is a good way to iteratively filter as you find the bad files, but it is still a really difficult situation to manage if you have numerous bad files. You have to find them one at a time and exclude them in $path and this process is slow and doesn't scale if you have many files to handle.

Not so familiar with the code but I think

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/line/LinePageSource.java

Line 78 in c2e7c3f

throw new TrinoException(HIVE_CURSOR_ERROR, format("Failed to read file at %s", filePath), e);

might be the up-to-date code and at least it seems to show the file location.

[raunaqmorarka]

There are a couple of options to filter groups of files: filters on partitioning columns, or filter on $file_modified_time.
In general, we try to avoid situations in Trino where we would silently skip over errors in the data and provide an incomplete result to the user. This has the potential to create silent correctness issues, e.g. the "bad" files might be valid data which is unreadable due to a bug in either the writing process or in Trino's reader and the user might remain oblivious to the underlying problem if they have a flag to skip over errors and set that.
If you have numerous bad files, then at that point I'd rather the underlying problem gets fixed (either in the writer or in Trino), than have a way to permanently work around it.

[baumand-amazon]

we try to avoid situations in Trino where we would silently skip over errors in the data and provide an incomplete result to the user

I agree that this is a reasonable default. The feature I'm suggesting is to have en explicit opt-in for this behaviour. EG: ignore-incomplete-tail-chunks-gzip (but choosing a better name). This effectively opts you in to the behaviour that tools like zless, zgrep and zcat use by default - processing the well-formed leading chunks.

The problem with filters on $file_modified_time or other columns is you don't know where the bad files might be so you can't construct a filter to exclude them.

Imagine the situation where you have hundreds of hosts offloading logs of a java app using log4j. You configure log4j to compress logs with gzip when they are rotated. You upload all the gzip logs from the hosts to s3. You want to query these logs with trino without having to validate every file. The problem is that if the process crashes while it's gzipping the file - you can have a partially written gzip file with an incomplete final chunk and that can be uploaded.

Now your ability to search any time-range containing that bad file is gone. You may not control the process that uploads the logs so you can't easily prevent files with incomplete trailers being uploaded.

The option I'm suggesting would make trino still usable for such a situation to query across the logs. And for anyone who wants to hard fail on an incomplete trailing chunk (in general I'd guess this would be rare - even if it will be the default) they don't have to opt in.