e2e: add upgrade test for clusters set up by promoted members #19634

fuweid · 2025-03-20T14:46:28Z

REF: #19557

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

cc @ahrtr @serathius @siyuanfoundation

Please wait for v3.5.20 release. This case can be pass with latest of release/3.5.

codecov · 2025-03-20T15:03:42Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.75%. Comparing base (ee2f4e0) to head (211f9e5).
Report is 10 commits behind head on main.

Additional details and impacted files

see 15 files with indirect coverage changes

@@           Coverage Diff           @@
##             main   #19634   +/-   ##
=======================================
  Coverage   68.75%   68.75%           
=======================================
  Files         419      419           
  Lines       35790    35790           
=======================================
  Hits        24608    24608           
+ Misses       9765     9763    -2     
- Partials     1417     1419    +2

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b76999...211f9e5. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tests/e2e/etcd_release_upgrade_test.go

ahrtr · 2025-03-20T15:27:36Z

Good, got the expected error "membership: too many learner members in cluster"

/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestClusterUpgradeAfterPromotingMembers-test-0) (60227): {"level":"fatal","ts":"2025-03-20T15:08:57.489048Z","caller":"etcdmain/etcd.go:183","msg":"discovery failed","error":"membership: too many learner members in cluster","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:183\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:283"}
    etcd_release_upgrade_test.go:192:

ahrtr · 2025-03-20T16:12:42Z

tests/e2e/etcd_release_upgrade_test.go

+	switch clusterVersion {
+	case e2e.CurrentVersion:
+		version, err = e2e.GetVersionFromBinary(e2e.BinPath.Etcd)
+		require.NoErrorf(t, err, "failed to get version from binary")
+	case e2e.LastVersion:
+		if !fileutil.Exist(e2e.BinPath.EtcdLastRelease) {
+			t.Skipf("%q does not exist", e2e.BinPath.EtcdLastRelease)
+		}
+
+		version, err = e2e.GetVersionFromBinary(e2e.BinPath.EtcdLastRelease)
+		require.NoErrorf(t, err, "failed to get version from last release binary")
+	default:
+		t.Fatalf("unexpected cluster version: %v", clusterVersion)
+	}


It seems that we can remove this code block to simplify the case. It's just getting the version and print it in log message.

ahrtr · 2025-03-20T20:10:39Z

@fuweid let's skip/disable the test so we can merge it for now. Once 3.5.20 is out, we can enable it again.

ahrtr · 2025-03-20T20:11:30Z

@fuweid let's skip/disable the test so we can merge it for now. Once 3.5.20 is out, we can enable it again.

or wait for 3.5.20 to be released tomorrow or next Monday

fuweid · 2025-03-21T00:09:36Z

or wait for 3.5.20 to be released tomorrow or next Monday

Let us wait for 3.5.20. No need to file two pull requests. Thanks

ahrtr · 2025-03-21T16:40:58Z

tests/e2e/etcd_release_upgrade_test.go

+	}
+}
+
+func createNewClusterByPromotingMembers(t *testing.T, clusterVersion *semver.Version, clusterSize int) *e2e.EtcdProcessCluster {


I think we can minor refactor this function as what I did in #19636.

We can create a separate PR to get it reviewed & merged separately

Note: I did some refactoring on this function in #19636, so that it can be reused by my e2e test cases.

Signed-off-by: Wei Fu <[email protected]>

fuweid · 2025-03-24T14:47:30Z

ping @ahrtr @serathius @siyuanfoundation it's ready to review. please take a look. thanks

ahrtr

LGTM

thx @fuweid

k8s-ci-robot · 2025-03-24T15:15:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, fuweid, siyuanfoundation

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr,fuweid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahrtr · 2025-03-24T15:16:12Z

Please also backport the PR to 3.6, thx

ahrtr · 2025-03-24T15:46:33Z

Please also backport the PR to 3.6, thx

It seems that you can only manually backport, because most of the implementation has already been included in 3.6.

k8s-ci-robot added area/testing approved size/L labels Mar 20, 2025

fuweid force-pushed the introduce-new-way-to-setup-cluster branch from 9276aeb to f85f42d Compare March 20, 2025 14:52

ahrtr reviewed Mar 20, 2025

View reviewed changes

tests/e2e/etcd_release_upgrade_test.go Outdated Show resolved Hide resolved

fuweid force-pushed the introduce-new-way-to-setup-cluster branch 2 times, most recently from beb0f6e to 1c611b2 Compare March 20, 2025 15:43

ahrtr reviewed Mar 20, 2025

View reviewed changes

siyuanfoundation approved these changes Mar 20, 2025

View reviewed changes

fuweid force-pushed the introduce-new-way-to-setup-cluster branch from 1c611b2 to eaeeccf Compare March 20, 2025 19:41

ahrtr reviewed Mar 21, 2025

View reviewed changes

fuweid force-pushed the introduce-new-way-to-setup-cluster branch 2 times, most recently from 8a262f8 to 7caad19 Compare March 24, 2025 13:39

e2e: add upgrade test for clusters set up by promoted members

211f9e5

Signed-off-by: Wei Fu <[email protected]>

fuweid force-pushed the introduce-new-way-to-setup-cluster branch from 7caad19 to 211f9e5 Compare March 24, 2025 13:49

ahrtr approved these changes Mar 24, 2025

View reviewed changes

ahrtr added the backport/v3.6 label Mar 24, 2025

ahrtr merged commit baeb60a into etcd-io:main Mar 24, 2025
32 checks passed

ahrtr mentioned this pull request Mar 21, 2025

[release-3.5] Learner promotion isn't persisted to v3store (bbolt) #19557

Closed

ivanvc mentioned this pull request Mar 24, 2025

Plan to release v3.6.0-rc.3 #19652

Closed

1 task

fuweid mentioned this pull request Mar 24, 2025

[release-3.6] e2e: add upgrade test for clusters set up by promoted members #19662

Merged

ahrtr mentioned this pull request May 14, 2025

testing upgrade scenarios similar to kubeadm etcd-io/etcd-operator#142

Closed

e2e: add upgrade test for clusters set up by promoted members #19634

e2e: add upgrade test for clusters set up by promoted members #19634

Uh oh!

Conversation

fuweid commented Mar 20, 2025

Uh oh!

codecov bot commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

ahrtr commented Mar 20, 2025

Uh oh!

ahrtr Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

fuweid Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

ahrtr commented Mar 20, 2025

Uh oh!

ahrtr commented Mar 20, 2025

Uh oh!

fuweid commented Mar 21, 2025

Uh oh!

ahrtr Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

ahrtr Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

fuweid commented Mar 24, 2025

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Mar 24, 2025

Uh oh!

ahrtr commented Mar 24, 2025

Uh oh!

Uh oh!

ahrtr commented Mar 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 20, 2025 •

edited

Loading