Arch x86_64_v2 is unsupported

[Gunni]

• This is not a support question, I have read about opensource and will send support questions to the IRC channel, GitHub Discussions or the mailing list.

• I have read and understood the 'out in the open' support policy

• I have read and understood the PowerDNS AI policy

• Program: Recursor

• Issue type: Bug report

Short description

Setup new AlmaLinux v10 on x86_64_v2, tried to install from master but RHEL10 versions don't have the master version... So I installed stable, and got this crash.

systemd output

Nov 05 09:39:23 watcher systemd[1]: pdns-recursor.service: Scheduled restart job, restart counter is at 235.
Nov 05 09:39:23 watcher systemd[1]: Starting pdns-recursor.service - PowerDNS Recursor...
Nov 05 09:39:23 watcher systemd-coredump[30852]: [🡕] Process 30847 (pdns_recursor) of user 995 dumped core.
	
    Module libpcre2-8.so.0 from rpm pcre2-10.44-1.el10.3.x86_64
    Module libselinux.so.1 from rpm libselinux-3.8-2.el10_0.x86_64
    Module libbrotlicommon.so.1 from rpm brotli-1.1.0-6.el10.x86_64
    Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-27.el10.x86_64
    Module libevent-2.1.so.7 from rpm libevent-2.1.12-16.el10.x86_64
    Module libkeyutils.so.1 from rpm keyutils-1.6.3-5.el10.x86_64
    Module libkrb5support.so.0 from rpm krb5-1.21.3-8.el10_0.x86_64
    Module libcom_err.so.2 from rpm e2fsprogs-1.47.1-3.el10.x86_64
    Module libk5crypto.so.3 from rpm krb5-1.21.3-8.el10_0.x86_64
    Module libkrb5.so.3 from rpm krb5-1.21.3-8.el10_0.x86_64
    Module libunistring.so.5 from rpm libunistring-1.1-10.el10.x86_64
    Module libbrotlidec.so.1 from rpm brotli-1.1.0-6.el10.x86_64
    Module libgssapi_krb5.so.2 from rpm krb5-1.21.3-8.el10_0.x86_64
    Module libpsl.so.5 from rpm libpsl-0.21.5-6.el10.x86_64
    Module libssh.so.4 from rpm libssh-0.11.1-4.el10_0.x86_64
    Module libidn2.so.0 from rpm libidn2-2.3.7-3.el10.x86_64
    Module libnghttp2.so.14 from rpm nghttp2-1.64.0-2.el10.x86_64
    Module libcrypt.so.2 from rpm libxcrypt-4.4.36-10.el10.x86_64
    Module libz.so.1 from rpm zlib-ng-2.2.3-1.el10.x86_64
    Module libboost_atomic.so.1.83.0 from rpm boost-1.83.0-5.el10.x86_64
    Module libcap.so.2 from rpm libcap-2.69-7.el10.x86_64
    Module libcurl.so.4 from rpm curl-8.9.1-5.el10.x86_64
    Module libfstrm.so.0 from rpm fstrm-0.6.1-12.el10.x86_64
    Module libssl.so.3 from rpm openssl-3.2.2-16.el10_0.4.alma.1.x86_64
    Module libsodium.so.26 from rpm libsodium-1.0.20-2.el10_0.alma_altarch.x86_64
    Module libcrypto.so.3 from rpm openssl-3.2.2-16.el10_0.4.alma.1.x86_64
    Module libboost_context.so.1.83.0 from rpm boost-1.83.0-5.el10.x86_64
    Module libsystemd.so.0 from rpm systemd-257-9.el10_0.1.alma.1.x86_64
    Module libluajit-5.1.so.2 from rpm luajit-2.1.1720049189-1.el10_0.alma_altarch.x86_64
    Module libboost_filesystem.so.1.83.0 from rpm boost-1.83.0-5.el10.x86_64
    Stack trace of thread 30847:
    #0  0x000055d34550c3d2 n/a (n/a + 0x0)
    #1  0x000055d345159925 n/a (n/a + 0x0)
    ELF object binary architecture: AMD x86-64
Nov 05 09:39:23 watcher systemd[1]: pdns-recursor.service: Main process exited, code=dumped, status=4/ILL
Nov 05 09:39:23 watcher systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
Nov 05 09:39:23 watcher systemd[1]: Stopped pdns-recursor.service - PowerDNS Recursor.

p.s. seriously put a RestartSec or StartLimitIntervalSec in the unit, the default restart interval is stupid! ~2 per second! Maybe look into using systemd/systemd#26902 on supported systems.

Environment

• Operating system: AlmaLinux 10.0
• Software version: 5.3.1
• Software source: PowerDNS repository

Steps to reproduce

1. Install OS
2. Install pdns-rc
3. Start pdns-rc

Expected behaviour

pdns-rc starts normally...

Actual behaviour

It crashed.

Other information

I changed config, if it matters

# cat /etc/pdns-recursor/recursor.conf
dnssec:
  validation: validate
recursor:
  include_dir: /etc/pdns-recursor/recursor.d
  setuid: pdns-recursor
  setgid: pdns-recursor
incoming:
  listen:
  - 127.0.0.1
  - ::1
#outgoing:
  # source_address:
  # - 0.0.0.0

[rgacogne]

Thanks for reporting this! Would you be able to test a version compiled on the machine itself, by any chance?

[omoerbeek]

I installed Almalinux 10 on a bare metal Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz

$ uname -a
Linux 2a10-3781-2e5e.connected.by.freedominter.net 6.12.0-55.40.1.el10_0.x86_64_v2 #1 SMP PREEMPT_DYNAMIC Tue Oct 21 10:19:15 UTC 2025 x86_64 GNU/Linux

Recursor 5.3.1 from the powerdns repo is running fine:

$ sudo systemctl status pdns-recursor|cat
● pdns-recursor.service - PowerDNS Recursor
     Loaded: loaded (/usr/lib/systemd/system/pdns-recursor.service; disabled; preset: disabled)
     Active: active (running) since Fri 2025-11-07 10:07:37 CET; 7min ago
 Invocation: f351f2d817df478aa2e9689e13de9138
       Docs: man:pdns_recursor(1)
             man:rec_control(1)
             https://doc.powerdns.com
   Main PID: 1572 (pdns_recursor)
      Tasks: 6 (limit: 151126)
     Memory: 22M (peak: 23.7M)
        CPU: 84ms
     CGroup: /system.slice/pdns-recursor.service
             └─1572 /usr/sbin/pdns_recursor --daemon=no --write-pid=no --disable-syslog --log-timestamp=no

Nov 07 10:07:37 2a10-3781-2e5e.connected.by.freedominter.net pdns-recursor[1572]: msg="Launching worker threads" subsystem="config" level="0" prio="Notice" tid="0" ts="1762506457.037" count="2"
Nov 07 10:07:37 2a10-3781-2e5e.connected.by.freedominter.net pdns-recursor[1572]: msg="Launching tcpworker threads" subsystem="config" level="0" prio="Notice" tid="0" ts="1762506457.037" count="1"
Nov 07 10:07:37 2a10-3781-2e5e.connected.by.freedominter.net pdns-recursor[1572]: msg="Enabled multiplexer" subsystem="runtime" level="0" prio="Info" tid="0" ts="1762506457.037" name="epoll"
Nov 07 10:07:37 2a10-3781-2e5e.connected.by.freedominter.net systemd[1]: Started pdns-recursor.service - PowerDNS Recursor.
Nov 07 10:07:42 2a10-3781-2e5e.connected.by.freedominter.net pdns-recursor[1572]: msg="Polled security status of version, no known issues reported" subsystem="housekeeping" level="0" prio="Notice" tid="0" ts="1762506462.312" query="recursor-5.3.1.security-status.secpoll.powerdns.com" securitymessage="OK" status="1" version="5.3.1"

$ cat /proc/cpuinfo | grep flags | head -1
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d

Can you tell us more about your setup? VM or bare metal, CPU model, cpu flags, etc?

[omoerbeek]

You can also use the script in https://unix.stackexchange.com/questions/631217/how-do-i-check-if-my-cpu-supports-x86-64-v2#631226 to show what your CPU is capable of. Mine shows:

CPU supports x86-64-v3

[Gunni]

It is a VM on Proxmox, on the below CPU.

CPU supports x86-64-v2

cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz
stepping        : 4
microcode       : 0x42e
cpu MHz         : 2399.998
cache size      : 16384 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase tsc_adjust smep erms xsaveopt arat vnmi umip md_clear flush_l1d arch_capabilities
vmx flags       : vnmi preemption_timer posted_intr invvpid ept_x_only ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid shadow_vmcs
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs mmio_unknown bhi ibpb_no_ret
bogomips        : 4799.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

lscpu

lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   1
  On-line CPU(s) list:    0
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz
    CPU family:           6
    Model:                62
    Thread(s) per core:   1
    Core(s) per socket:   1
    Socket(s):            1
    Stepping:             4
    BogoMIPS:             4799.99
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 pdcm pcid sse4_1 sse4_2
                          x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase tsc_adjust smep erms xsaveopt arat vnmi umip md_clear flush_l1d arch_capabilities
Virtualization features:
  Virtualization:         VT-x
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):
  L1d:                    32 KiB (1 instance)
  L1i:                    32 KiB (1 instance)
  L2:                     4 MiB (1 instance)
  L3:                     16 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Mitigation; PTE Inversion; VMX flush not necessary, SMT disabled
  Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Unknown: No mitigations
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
  Srbds:                  Not affected
  Tsx async abort:        Not affected

[omoerbeek]

Is this the info from the VM guest? Hypervisors do not necessarily expose all features to guests.

[Gunni]

Yes this is the vm guest, I do not have hypervisor access for this VM.

I do know the cpu setting in proxmox is set to "host" so it exposes the cpu as-is afaik.

[hlindqvist]

@omoerbeek

I suspect the central issue is that RHEL10, and el10 variants in general, require x86_64_v3 (ie AVX2 is required). From the looks of it these are also the build flags picked up in the el10 pdns package builds?

However, Almalinux 10 also ships a separate x86_64_v2 version (handled as a whole separate architecture!) for compatibility with older systems.

See https://almalinux.org/blog/2025-05-27-welcoming-almalinux-10/#extended-x86-64-v2-life and https://almalinux.org/blog/2025-05-27-welcoming-almalinux-10/#isos-live-images-cloud-and-containers

It would appear the reported problem is specific to taking the pdns el10 packages and using them on a x86_64_v2 system that runs the Almalinux 10 x86_64_v2 variant.

[omoerbeek]

Flags used by the el-10 build:

CXXFLAGS='-O2 -flto=thin -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS --config=/usr/lib/rpm/redhat/redhat-hardened-clang.cfg -fstack-protector-strong   -m64 -march=x86-64-v3 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection   '

So indeed, our executable actually is a v3 one. Thanks for the hint!

@gunn: until we decide how (and if) we are going to solve this, I suggest to build recursor yourself on your v2 system. See https://docs.powerdns.com/recursor/appendices/compiling.html . It might very well be that we decide to not support this flavor of an el-10 derived system.

[omoerbeek]

The conclusion is that we are not going to build packages specifically for AlmaLinux 10 x86_64_v2. To run our packages from repo.powerdns.com on el-10 derived x86_64 system, your CPU must support x86_64_v3.

[Gunni]

Maybe then mark the repo with v3 and put a v2 with a readme or something? Or maybe the repo thing has a feature to tell you your arch is unsupported instead of trying to run and crashing?

[Habbie]

https://wiki.almalinux.org/release-notes/10.0.html explicitly notes:

All 3rd party packages for RHEL10 will target x86-64-v3, while the x86-64-v2 release of AlmaLinux OS 10 will only be suitable for workloads where using the default OS package set is enough, or where users will be able to rebuild any additional packages they require for x86-64-v2 architecture themselves.

Alma does offer a -v2 rebuild of EPEL, but sadly pdns-recursor is not currently in EPEL10. Apparently somebody did already request it, but nothing has happened yet.