Dataflow Templates 2025-12-09-00_RC00

What's Changed

Improvements

• [SpannerToSourceDb] Integration test for CDC processing algorithm for Reverse migration template by @darshan-sj in #3027
• [SourceDbToSpanner] Fixing severe error file overwrite issue in Bulk migration template by @darshan-sj in #3039
• upgrade go to 1.24 to fix cicd issue by @derrickaw in #3048
• Add auto yaml to java generation by @derrickaw in #2984
• remove old todos for 2.68, gax, httpclient, and json-smart that are not used by @derrickaw in #2887
• feat: Denylist small worker sizes in reverse replication by @manitgupta in #3044
• Add support for GCP SchemaRegistry in Kafka Templates by @Dippatel98 in #3066

Bug fixes

• When Spanner has NULL value for SQL PK column, the error is thrown by change event by @aasthabharill in #3034

Package Upgrades

• update maven version in pom.xml by @apanich in #3068

New Contributors

• @NicolasRicheYopp made their first contribution in #3029
• @apanich made their first contribution in #3068
• @gpalanisamyYopp made their first contribution in #3059

Full Changelog: 2025-12-02-00_RC00...2025-12-09-00_RC00

Dataflow Templates 2025-12-02-00_RC00

What's Changed

Improvements

• Fix JSON/JSONB datatypes in postgres-to-sql template by @yair-harel in #2960
• Add IcebergResourceManager by @tarun-google in #2985
• Add support for additional fields in terraform templates for Reverse and Live migration by @aasthabharill in #3005
• [DatastreamToSpanner] Integration test for CDC processing algorithm for Live migration template by @darshan-sj in #3018

Bug fixes

• Remove preemptive cleanup by @rahulkumar0630 in #2981
• chore: Added latency metrics to Write to source DoFn by @pratickchokhani in #3009
• [spanner-to-sourcedb] Custom Transformation fails when SQL PK col is dropped in Spanner by @aasthabharill in #3003
• [spanner-to-sourcedb] DELETEs: Error fetching shard id column by @aasthabharill in #3024
• Fix dependency conflict on beam-sdks-java-core in Datastream to Firestore by @baeminbo in #3026

New Contributors

• @tarun-google made their first contribution in #2985
• @nmemond made their first contribution in #2992
• @aasthabharill made their first contribution in #3005
• @pratickchokhani made their first contribution in #3009
• @baeminbo made their first contribution in #3026

Full Changelog: 2025-11-11-00_RC00...2025-12-02-00_RC00

Dataflow Templates 2025-11-11-00_RC00

What's Changed

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2979

Full Changelog: 2025-11-06-00_RC00...2025-11-11-00_RC00

Dataflow Templates 2025-11-06-00_RC00

What's Changed

Improvements

• Delete v2/datastream-to-postgres directory by @dhercher in #2959
• [SpannerToSourceDb] Improving error classification in Reverse replication template by @darshan-sj in #2964
• Update to beam 2.69 by @Abacn in #2966

Bug fixes

• fixing retry count in reverse replication by @shreyakhajanchi in #2949

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2956

Full Changelog: 2025-10-28-01_RC00...2025-11-06-00_RC00

Dataflow Templates 2025-11-05-00_RC00

fix(build): update surefire report processing to handle time format (…

Dataflow Templates 2025-10-28-01_RC00

What's Changed

Improvements

• fix(kafka): handle domain-scoped project IDs in bootstrap server URL by @liferoad in #2908
• Yaml: Add yaml syntax UT by @derrickaw in #2872
• [SpannerToSourceDb] Fixing CDC events processing logic for Reverse replication template by @darshan-sj in #2919
• Apply GCP Application Default authentication when using a Managed Service for Apache Kafka cluster. by @fozzie15 in #2925

Bug fixes

• Remove special handling for only PK column tables by @manitgupta in #2929
• [yaml] - move syntax test verification to validate section by @derrickaw in #2936

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2899
• build: update jetty dependencies to 12.0.12 to fix CVE-2024-6763 by @stankiewicz in #2921
• build: update org.apache.commons:commons-configuration to resolve CVE-2024-29131 by @stankiewicz in #2922

New Contributors

• @LucaDai made their first contribution in #2898

Full Changelog: 2025-10-21-00_RC00...2025-10-28-01_RC00

Dataflow Templates 2025-10-28-00_RC00

bump org.apache.commons:commons-configuration to resolve CVE-2024-291…

Dataflow Templates 2025-10-21-00_RC00

What's Changed

Improvements

• KafkaToBigQueryYaml - change hidden to false by @derrickaw in #2879
• Add option to unwrap messages replayed from DLQ to avoid nesting loops. by @claudevdm in #2888

Bug fixes

• fix dependabot labeling workflow name by @derrickaw in #2886
• fix label dependabot workflow by @derrickaw in #2890

Full Changelog: 2025-10-14-00_RC00...2025-10-21-00_RC00

Dataflow Templates 2025-10-14-00_RC00

What's Changed

Improvements

• feat(kafka-to-bigquery): Fix dynamic destination connection churn by @anpag in #2873

Bug fixes

• changed mysql event ordering in datastream to spanner by @bharadwaj-aditya in #2867

New Contributors

• @tianz101 made their first contribution in #2871
• @anpag made their first contribution in #2873

Full Changelog: 2025-10-07-00_RC00...2025-10-14-00_RC00

Dataflow Templates 2025-10-07-00_RC00

What's Changed

Improvements

• Fixed a performance regression in the Spanner change streams to BigQuery template that was introduced in the 2024-09-10-00_RC00.
• Only read schema when the col in mod is not in the stored schema. by @jiangzzhu in #2840
• Skip reading schema if the value capture mode is NEW_ROW_AND_OLD_VALUES. by @jiangzzhu in #2839
• feat: Add support for property graph on view by @jennyjiang98 in #2813
• Remove jinja variables option code by @derrickaw in #2826

New Contributors

• @jiangzzhu made their first contribution in #2840

Full Changelog: 2025-10-01-00_RC00...2025-10-07-00_RC00