Skip to content

[USHIFT-6106] - Add RHEL 10 support to bootc test scenarios #5695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[USHIFT-6106] - Add RHEL 10 support to bootc test scenarios #5695

wants to merge 1 commit into from
+993 −0

Conversation

@kasturinarra
Copy link
Contributor

@kasturinarra kasturinarra commented Nov 5, 2025

Which issue(s) this PR addresses: Add RHEL10 support for bootc scenarios in releases directory

Closes #

@openshift-ci openshift-ci bot requested review from copejon and pmtk November 5, 2025 07:32
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 5, 2025
@kasturinarra
Copy link
Contributor Author

kasturinarra commented Nov 7, 2025

/test e2e-aws-tests-bootc-release

@kasturinarra
Copy link
Contributor Author

kasturinarra commented Nov 7, 2025

/test e2e-aws-tests-bootc-release-arm

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 7, 2025

@kasturinarra: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@kasturinarra
Copy link
Contributor Author

kasturinarra commented Nov 7, 2025

@agullon all tests passed, could you please help review ? thanks !!

@ggiguash
Copy link
Contributor

ggiguash commented Nov 8, 2025 •
edited
Loading

These are a lot of tests per job. We need to watch the hypervisor stability.
Currently, the job passes because most if not all these tests are skipped due to non-existent images.

I checked performance monitoring logs for amd64 and arm64 jobs.

Here's an example on how to check the system performance, watching the first (load average) and third (free memory) columns.

X86

$ pmstat -a . | awk '($1+0 > 30){print}'
   32.48      0   143g   6936 36262m    0    0    0  33K 898K 278K  55   9  37
   32.48      0   143g   6936 36262m    0    0   96  29K 768K 242K  64   9  27
   52.85      0   133g   6936 41764m    0    0  230  22K 588K 193K  78   9  13
   52.85      0   133g   6936 41764m    0    0  134 168K 598K 163K  69   8  23
   62.21      0   133g   6936 46455m    0    0    0 370K 611K 122K  56   7  37
   62.21      0   133g   6936 46455m    0    0    0 701K 382K  84K  35   5  61
   53.64      0   132g   6936 47430m    0    0    0   1M  67K  31K   6   1  93
   53.64      0   132g   6936 47430m    0    0    0 823K 297K  74K  17   3  80
   54.11      0   138g   6936 42073m    0    0    0 363K 615K 133K  33   5  61
   54.11      0   138g   6936 42073m    0    0  48K 492K 484K 101K  28   5  67
   48.83      0   137g   6936 42671m    0    0 115K 670K 304K  57K  21   4  76
   48.83      0   137g   6936 42671m    0    0  67K 511K 184K  55K  13   3  84
   41.70      0   141g   6936 41303m    0    0 1072 292K  20K  53K   2   1  97
   41.70      0   141g   6936 41303m    0    0  625 243K  13K  34K   2   1  97
   35.52      0   141g   6936 41327m    0    0   10 175K 4416 8450   1   0  99
   35.52      0   141g   6936 41327m    0    0    6 179K 3749 7305   1   0  99
   30.22      0   141g   6936 41388m    0    0    0 184K 2832 5728   1   0  99
   30.22      0   141g   6936 41388m    0    0    0 134K 4631 9194   1   0  99
   31.39      0   125g   6936 48094m    0    0    0 458K 186K  36K  15   2  84
   31.39      0   125g   6936 48094m    0    0    0 405K 211K  41K  19   2  79
   37.12      0   124g   6936 49388m    0    0    0 330K 246K  48K  26   2  72
   37.12      0   124g   6936 49388m    0    0    0 469K 206K  36K  20   2  79
   33.00      0   124g   6936 50541m    0    0    0 660K 151K  21K  11   1  87
   33.00      0   124g   6936 50541m    0    0    0 602K 157K  18K  11   1  88

ARM

$ pmstat -a . | awk '($1+0 > 30){print}'
   46.11      0 82023m   6936 35533m    0    0    0  21K  90K 158K  72   8  20
   46.11      0 82023m   6936 35533m    0    0    0 134K  92K 153K  73   7  20
   44.64      0 74731m   6936 43189m    0    0    0 433K  95K 142K  76   6  17
   44.64      0 74731m   6936 43189m    0    0 2289 510K  80K 116K  58   5  37
   38.31      0 83846m   6936 37416m    0    0 8337 713K  40K  47K  10   2  88
   38.31      0 83846m   6936 37416m    0    0 6048 704K  56K  74K  20   3  77
   42.30      0 80517m   6936 40373m    0    0    4 681K  98K 144K  46   6  49
   42.30      0 80517m   6936 40373m    0    0  28K 633K  81K 122K  34   5  61
   36.26      0 84168m   6936 38253m    0    0 102K 506K  37K  64K   4   2  94
   36.26      0 84168m   6936 38253m    0    0  74K 420K  28K  48K   3   2  95
   30.84      0 84363m   6936 38297m    0    0    0 193K 4872 7429   2   0  98
   30.84      0 84363m   6936 38297m    0    0    0 207K 5051 7948   2   0  98
   32.55      0 64561m   6936 45952m    0    0    0 402K  90K 140K  52   4  44
   32.55      0 64561m   6936 45952m    0    0    0 309K  76K 114K  45   3  52

I'm putting the PR on hold because it does not seem to be testing the real load it's going to cause.

We need to retest this PR after #5689 merges and make sure tests are not skipped.

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 8, 2025
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2025
@ggiguash
Copy link
Contributor

ggiguash commented Nov 8, 2025

/lgtm cancel

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 8, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kasturinarra

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kasturinarra
Copy link
Contributor Author

kasturinarra commented Nov 8, 2025

These are a lot of tests per job. We need to watch the hypervisor stability. Currently, the job passes because most if not all these tests are skipped due to non-existent images.

I checked performance monitoring logs for amd64 and arm64 jobs.

Here's an example on how to check the system performance, watching the first (load average) and third (free memory) columns.

X86

$ pmstat -a . | awk '($1+0 > 30){print}'
   32.48      0   143g   6936 36262m    0    0    0  33K 898K 278K  55   9  37
   32.48      0   143g   6936 36262m    0    0   96  29K 768K 242K  64   9  27
   52.85      0   133g   6936 41764m    0    0  230  22K 588K 193K  78   9  13
   52.85      0   133g   6936 41764m    0    0  134 168K 598K 163K  69   8  23
   62.21      0   133g   6936 46455m    0    0    0 370K 611K 122K  56   7  37
   62.21      0   133g   6936 46455m    0    0    0 701K 382K  84K  35   5  61
   53.64      0   132g   6936 47430m    0    0    0   1M  67K  31K   6   1  93
   53.64      0   132g   6936 47430m    0    0    0 823K 297K  74K  17   3  80
   54.11      0   138g   6936 42073m    0    0    0 363K 615K 133K  33   5  61
   54.11      0   138g   6936 42073m    0    0  48K 492K 484K 101K  28   5  67
   48.83      0   137g   6936 42671m    0    0 115K 670K 304K  57K  21   4  76
   48.83      0   137g   6936 42671m    0    0  67K 511K 184K  55K  13   3  84
   41.70      0   141g   6936 41303m    0    0 1072 292K  20K  53K   2   1  97
   41.70      0   141g   6936 41303m    0    0  625 243K  13K  34K   2   1  97
   35.52      0   141g   6936 41327m    0    0   10 175K 4416 8450   1   0  99
   35.52      0   141g   6936 41327m    0    0    6 179K 3749 7305   1   0  99
   30.22      0   141g   6936 41388m    0    0    0 184K 2832 5728   1   0  99
   30.22      0   141g   6936 41388m    0    0    0 134K 4631 9194   1   0  99
   31.39      0   125g   6936 48094m    0    0    0 458K 186K  36K  15   2  84
   31.39      0   125g   6936 48094m    0    0    0 405K 211K  41K  19   2  79
   37.12      0   124g   6936 49388m    0    0    0 330K 246K  48K  26   2  72
   37.12      0   124g   6936 49388m    0    0    0 469K 206K  36K  20   2  79
   33.00      0   124g   6936 50541m    0    0    0 660K 151K  21K  11   1  87
   33.00      0   124g   6936 50541m    0    0    0 602K 157K  18K  11   1  88

ARM

$ pmstat -a . | awk '($1+0 > 30){print}'
   46.11      0 82023m   6936 35533m    0    0    0  21K  90K 158K  72   8  20
   46.11      0 82023m   6936 35533m    0    0    0 134K  92K 153K  73   7  20
   44.64      0 74731m   6936 43189m    0    0    0 433K  95K 142K  76   6  17
   44.64      0 74731m   6936 43189m    0    0 2289 510K  80K 116K  58   5  37
   38.31      0 83846m   6936 37416m    0    0 8337 713K  40K  47K  10   2  88
   38.31      0 83846m   6936 37416m    0    0 6048 704K  56K  74K  20   3  77
   42.30      0 80517m   6936 40373m    0    0    4 681K  98K 144K  46   6  49
   42.30      0 80517m   6936 40373m    0    0  28K 633K  81K 122K  34   5  61
   36.26      0 84168m   6936 38253m    0    0 102K 506K  37K  64K   4   2  94
   36.26      0 84168m   6936 38253m    0    0  74K 420K  28K  48K   3   2  95
   30.84      0 84363m   6936 38297m    0    0    0 193K 4872 7429   2   0  98
   30.84      0 84363m   6936 38297m    0    0    0 207K 5051 7948   2   0  98
   32.55      0 64561m   6936 45952m    0    0    0 402K  90K 140K  52   4  44
   32.55      0 64561m   6936 45952m    0    0    0 309K  76K 114K  45   3  52

I'm putting the PR on hold because it does not seem to be testing the real load it's going to cause.

We need to retest this PR after #5689 merges and make sure tests are not skipped.

/hold

Let me also discuss with john and alejandro to see what they feel.

@kasturinarra
Copy link
Contributor Author

kasturinarra commented Nov 8, 2025

These are a lot of tests per job. We need to watch the hypervisor stability. Currently, the job passes because most if not all these tests are skipped due to non-existent images.
I checked performance monitoring logs for amd64 and arm64 jobs.
Here's an example on how to check the system performance, watching the first (load average) and third (free memory) columns.
X86

$ pmstat -a . | awk '($1+0 > 30){print}'
   32.48      0   143g   6936 36262m    0    0    0  33K 898K 278K  55   9  37
   32.48      0   143g   6936 36262m    0    0   96  29K 768K 242K  64   9  27
   52.85      0   133g   6936 41764m    0    0  230  22K 588K 193K  78   9  13
   52.85      0   133g   6936 41764m    0    0  134 168K 598K 163K  69   8  23
   62.21      0   133g   6936 46455m    0    0    0 370K 611K 122K  56   7  37
   62.21      0   133g   6936 46455m    0    0    0 701K 382K  84K  35   5  61
   53.64      0   132g   6936 47430m    0    0    0   1M  67K  31K   6   1  93
   53.64      0   132g   6936 47430m    0    0    0 823K 297K  74K  17   3  80
   54.11      0   138g   6936 42073m    0    0    0 363K 615K 133K  33   5  61
   54.11      0   138g   6936 42073m    0    0  48K 492K 484K 101K  28   5  67
   48.83      0   137g   6936 42671m    0    0 115K 670K 304K  57K  21   4  76
   48.83      0   137g   6936 42671m    0    0  67K 511K 184K  55K  13   3  84
   41.70      0   141g   6936 41303m    0    0 1072 292K  20K  53K   2   1  97
   41.70      0   141g   6936 41303m    0    0  625 243K  13K  34K   2   1  97
   35.52      0   141g   6936 41327m    0    0   10 175K 4416 8450   1   0  99
   35.52      0   141g   6936 41327m    0    0    6 179K 3749 7305   1   0  99
   30.22      0   141g   6936 41388m    0    0    0 184K 2832 5728   1   0  99
   30.22      0   141g   6936 41388m    0    0    0 134K 4631 9194   1   0  99
   31.39      0   125g   6936 48094m    0    0    0 458K 186K  36K  15   2  84
   31.39      0   125g   6936 48094m    0    0    0 405K 211K  41K  19   2  79
   37.12      0   124g   6936 49388m    0    0    0 330K 246K  48K  26   2  72
   37.12      0   124g   6936 49388m    0    0    0 469K 206K  36K  20   2  79
   33.00      0   124g   6936 50541m    0    0    0 660K 151K  21K  11   1  87
   33.00      0   124g   6936 50541m    0    0    0 602K 157K  18K  11   1  88

ARM

$ pmstat -a . | awk '($1+0 > 30){print}'
   46.11      0 82023m   6936 35533m    0    0    0  21K  90K 158K  72   8  20
   46.11      0 82023m   6936 35533m    0    0    0 134K  92K 153K  73   7  20
   44.64      0 74731m   6936 43189m    0    0    0 433K  95K 142K  76   6  17
   44.64      0 74731m   6936 43189m    0    0 2289 510K  80K 116K  58   5  37
   38.31      0 83846m   6936 37416m    0    0 8337 713K  40K  47K  10   2  88
   38.31      0 83846m   6936 37416m    0    0 6048 704K  56K  74K  20   3  77
   42.30      0 80517m   6936 40373m    0    0    4 681K  98K 144K  46   6  49
   42.30      0 80517m   6936 40373m    0    0  28K 633K  81K 122K  34   5  61
   36.26      0 84168m   6936 38253m    0    0 102K 506K  37K  64K   4   2  94
   36.26      0 84168m   6936 38253m    0    0  74K 420K  28K  48K   3   2  95
   30.84      0 84363m   6936 38297m    0    0    0 193K 4872 7429   2   0  98
   30.84      0 84363m   6936 38297m    0    0    0 207K 5051 7948   2   0  98
   32.55      0 64561m   6936 45952m    0    0    0 402K  90K 140K  52   4  44
   32.55      0 64561m   6936 45952m    0    0    0 309K  76K 114K  45   3  52

I'm putting the PR on hold because it does not seem to be testing the real load it's going to cause.
We need to retest this PR after #5689 merges and make sure tests are not skipped.
/hold

Let me also discuss with john and alejandro to see what they think as well.

@agullon
Copy link
Contributor

agullon commented Nov 10, 2025

hi @kasturinarra the only issue I can foresee with this PR is what it's already discussed with Giga: we can not run that many scenarios in parallel. So I guess we need to remove some of them or think to how to rotate which run every time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@copejon copejon Awaiting requested review from copejon

@pmtk pmtk Awaiting requested review from pmtk

Assignees

@ggiguash ggiguash

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kasturinarra @ggiguash @agullon