-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[feature](iceberg) Support Partition Evolution DDL for Iceberg Tables #57972
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
[feature](iceberg) Support Partition Evolution DDL for Iceberg Tables #57972
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
3c485cb to
7d998d1
Compare
|
run buildall |
zddr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reference regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy to add some partition evolution cases to validate the behavior of MTMV.
For example:
- Without changing the partition column, but switching from day to year, should be allowed. The materialized view should refresh normally and generate correct partitions.
- Changing the partition column from c1 to c2 should cause the materialized view refresh to fail.
- Switching from year to identity should cause the materialized view refresh to fail.
|
run buildall |
TPC-H: Total hot run time: 34247 ms |
TPC-DS: Total hot run time: 188470 ms |
ClickBench: Total hot run time: 27.43 s |
FE UT Coverage ReportIncrement line coverage |
|
run buildall |
TPC-H: Total hot run time: 34489 ms |
TPC-DS: Total hot run time: 187845 ms |
ClickBench: Total hot run time: 27.51 s |
FE UT Coverage ReportIncrement line coverage |
57da735 to
26b5aef
Compare
26b5aef to
f2cf736
Compare
|
run buildall |
TPC-H: Total hot run time: 34708 ms |
TPC-DS: Total hot run time: 187048 ms |
ClickBench: Total hot run time: 27.91 s |
|
run external |
fixed |
What problem does this PR solve?
Summary
This PR implements support for partition evolution in Iceberg tables, allowing users to dynamically modify table partition strategies without rewriting data files. This is a metadata-only operation that maintains multiple partition spec versions.
Background
Apache Iceberg supports partition evolution, which enables changing partition strategies on existing tables without data migration. Doris, as a query engine for Iceberg, needs to support SQL syntax for partition evolution operations to provide users with flexible partition management.
Features
Core Functionality
Design Principles
Syntax
Add Partition Field
Drop Partition Field
Replace Partition Field
Supported Partition Transforms
bucket(N, column)bucket(16, id)truncate(N, column)truncate(10, name)year(column)year(ts)month(column)month(ts)day(column)day(ts)hour(column)hour(ts)columncategoryRelease note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)