Upgrade base images to Ubuntu 24.04

[adambabik]

It seems that the base images are still using Ubuntu 22.04. Please upgrade them to the latest LTS which is 24.04.

From time to time, we observe a kernel issue when running the Monitor at scale in the Cloud.

[amnonh]

@yaronkaikov if you can handle it before 4.12 is officially released, it would be great

[mykaul]

It seems that the base images are still using Ubuntu 22.04. Please upgrade them to the latest LTS which is 24.04.

From time to time, we observe a kernel issue when running the Monitor at scale in the Cloud.

22 is also LTS. There shouldn't be a kernel issue. If there is, please report it.

[adambabik]

This is the console output we managed to capture:

[ 5103.423319] rcu: INFO: rcu_sched self-detected stall on CPU
[ 5103.427318] rcu:     1-....: (1266109 ticks this GP) idle=0b8c/1/0x4000000000000000 softirq=10/10 fqs=632882
[ 5103.431319]  (t=1275428 jiffies g=-1167 q=2 ncpus=4)
[ 5103.435319] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G      D      L     6.2.0-1015-aws #15~22.04.1-Ubuntu
[ 5103.443318] Hardware name: Amazon EC2 m6i.xlarge/, BIOS 1.0 10/16/2017
[ 5103.447319] RIP: 0010:queued_write_lock_slowpath+0x8f/0x13f
[ 5103.451320] Code: 31 d2 45 31 db c3 cc cc cc cc f0 41 81 0c 24 00 01 00 00 ba ff 00 00 00 b9 00 01 00 00 41 8b 04 24 3d 00 01 00 00 74 0d
 f3 90 <41> 8b 04 24 3d 00 01 00 00 75 f3 89 c8 f0 41 0f b1 14 24 74 9e eb
[ 5103.463319] RSP: 0000:ffffb80c40017d90 EFLAGS: 00000206
[ 5103.467318] RAX: 00000000000001ff RBX: ffffffff9f76b1d4 RCX: 0000000000000100
[ 5103.471318] RDX: 00000000000000ff RSI: 00000000bffe9000 RDI: ffffffff9f76b1d0
[ 5103.475318] RBP: ffffb80c40017da0 R08: 0000000000000000 R09: 0000000000000000
[ 5103.479318] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9f76b1d0
[ 5103.487317] R13: ffffffff9e2804a0 R14: ffff98a8408856c0 R15: 00000000000001a2
[ 5103.491317] FS:  0000000000000000(0000) GS:ffff98ab5f080000(0000) knlGS:0000000000000000
[ 5103.495318] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5103.499317] CR2: 0000000000000000 CR3: 0000000358210001 CR4: 00000000007706e0
[ 5103.503317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5103.507324] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5103.515317] PKRU: 55555554
[ 5103.515317] Call Trace:
[ 5103.519318]  <IRQ>
[ 5103.523318]  ? show_regs+0x72/0x90
[ 5103.523318]  ? dump_cpu_task+0x74/0x90
[ 5103.527318]  ? rcu_dump_cpu_stacks+0xe8/0x180
[ 5103.531318]  ? print_cpu_stall+0x131/0x2b0
[ 5103.535318]  ? check_cpu_stall+0x113/0x280
[ 5103.539318]  ? rcu_sched_clock_irq+0x2cd/0x3f0
[ 5103.539318]  ? update_process_times+0x63/0xb0
[ 5103.543318]  ? tick_periodic+0x2b/0x90
[ 5103.547318]  ? tick_handle_periodic+0x25/0x80
[ 5103.551318]  ? __sysvec_apic_timer_interrupt+0x5e/0x110
[ 5103.555318]  ? sysvec_apic_timer_interrupt+0x91/0xb0
[ 5103.559318]  </IRQ>
[ 5103.559318]  <TASK>
[ 5103.563318]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[ 5103.567318]  ? queued_write_lock_slowpath+0x8f/0x13f
[ 5103.571318]  _raw_write_lock+0x26/0x40
[ 5103.575318]  reserve_region_with_split+0x2f/0x200
[ 5103.579317]  e820__reserve_resources_late+0xeb/0x120
[ 5103.579317]  pcibios_resource_survey+0xdd/0x100
[ 5103.583317]  ? __pfx_pci_subsys_init+0x10/0x10
[ 5103.587317]  pcibios_init+0x37/0x90
[ 5103.591318]  pci_subsys_init+0x66/0xa0
[ 5103.595317]  do_one_initcall+0x46/0x210
[ 5103.595317]  do_initcalls+0xe3/0x1e0
[ 5103.599318]  kernel_init_freeable+0x12f/0x1f0
[ 5103.603327]  ? __pfx_kernel_init+0x10/0x10
[ 5103.607318]  kernel_init+0x1b/0x200
[ 5103.611318]  ? __pfx_kernel_init+0x10/0x10
[ 5103.615318]  ret_from_fork+0x29/0x50
[ 5103.615318]  </TASK>

The instance is stuck and never progresses.

[mykaul]

For some reason you are not using the LTS kernel version (5.15.x - https://launchpad.net/ubuntu/jammy/+source/linux-aws ) , but 6.2.0-x - which is not even on LTS? It should have been either 5.15.x, or 6.8.0-something (see https://launchpad.net/ubuntu/jammy/+package/linux-aws )

If I'm looking at 6.2.0, this is it - https://launchpad.net/ubuntu/+source/linux-aws-6.2 - and it shows you are using an outdated version. Even -1018 was published on 2024-01-24 !

@yaronkaikov - I reiterate my request to have a single 'base AMI' for all our artifacts. That would have solved this, I hope.

[yaronkaikov]

For some reason you are not using the LTS kernel version (5.15.x - https://launchpad.net/ubuntu/jammy/+source/linux-aws ) , but 6.2.0-x - which is not even on LTS? It should have been either 5.15.x, or 6.8.0-something (see https://launchpad.net/ubuntu/jammy/+package/linux-aws )

If I'm looking at 6.2.0, this is it - https://launchpad.net/ubuntu/+source/linux-aws-6.2 - and it shows you are using an outdated version. Even -1018 was published on 2024-01-24 !

@yaronkaikov - I reiterate my request to have a single 'base AMI' for all our artifacts. That would have solved this, I hope.

We always use the latest image available when building AMI (in Scylla), Not sure why do we want now to maintain another image which probably need to be update on daily/weekly basis. Let's first define a task for it with all the requirements.

[adambabik]

@yaronkaikov Can you prioritize this? It seems like a config change and we hit this problem almost every week in our nightly tests.

[yaronkaikov]

@yaronkaikov Can you prioritize this? It seems like a config change and we hit this problem almost every week in our nightly tests.

will try to fit it in our next sprint

[yaronkaikov]

@yaronkaikov Can you prioritize this? It seems like a config change and we hit this problem almost every week in our nightly tests.

will try to fit it in our next sprint

Created https://scylladb.atlassian.net/browse/RELENG-5 so it will be visible when i plan the sprint