521.wrf_r segfaults when built with -flto=full -O2

[bryanpkc]

After upgrading to LLVM 17, the 521.wrf_r benchmark in SPEC CPU2017 is miscompiled at O3 with LTO enabled. Examination of the crash site shows that a large amount of code that used to be inlined into wrf_init (via alloc_and_configure_domain) has been deleted, leaving the head_grid pointer uninitialized.

The root cause of this problem is that upstream LLVM had strengthened the pruning of unreachable code to handle undef and poison. Specifically, these three patches are found to be relevant:

• llvm/llvm-project@1fc425380e
• llvm/llvm-project@70aca7b122
• llvm/llvm-project@41895843b5

Classic Flang leaves variables uninitialized by default. The corresponding IR values are therefore considered to contain undef. In our downstream compiler, we have tested with initializing all variables to zeroes, and that successfully avoids the problem.

628.pop2_s is also affected by this problem.

[bryanpkc]

As discussed at the call, I will follow up in this issue with information about which variables are left uninitialized in the source, and whether the problem occurs at -O2/-O3.

[bryanpkc]

The problem does occur at -O2 and -O3.

The compiler flags I used are:

-Mallocatable=03 -flto=full -O2 -mcpu=native -ffast-math -Mbyteswapio

The linker flags I used (with ld.lld) are:

-flto=full -O2 -mcpu=native -ffast-math

[bryanpkc]

The problematic undef values that caused some basic blocks to be pruned correspond to the dummy arguments ipex and jpex in the function compute_memory_dims_rsl_lite (module_dm.F90, line 1088). This function is called by patch_domain_rsl_lite, which passes a whole bunch of actual arguments, none of which is initialized before the call.

[bryanpkc]

@pawosm-arm @shivaramaarao Could you check things on your side to see if you can reproduce the problem, or determine why you don't see the problem?

[pawosm-arm]

Hi @bryanpkc assuming the problem manifests itself at runtime, I need to ask, what input data set are you using? (test, ref, or something else). Also, with -mcpu=native being used at build time, what are the CPU features (namely, NEON vs SVE)?

[bryanpkc]

@pawosm-arm I have confirmed that the problem can be reproduced without specifying any -mcpu= option. The run-time segmentation fault happens during benchmark start-up; I am using the test data set.

[pawosm-arm]

Been testing it all night in our CI, with your flags, with O3 and even with Ofast, and it didn't fail. It's time to do it manually, but bear in mind, building spec's wrf with LTO take ages, and I want to try two different builds of the compiler, so it may take days before I'd confirm anything.

[pawosm-arm]

Just noticed, our CI always adds -funroll-loops to the explicitly specified set of flags, but I doubt it matters

[pawosm-arm]

...also we don't seem to use (or even build) lld, but it also should make no difference on what LTO does when optimizing the code

[pawosm-arm]

Should 628.pop2_s also fail during benchmark start-up? It takes forever to execute it, so I guess it's not affected in my builds...

[bryanpkc]

@pawosm-arm Can you confirm in the 521.wrf_r source files that the actual arguments I mentioned are not initialized? I am using an older version of the benchmark (v1.0.5) and perhaps SPEC has changed the source since then?

[shivaramaarao]

I could reproduce the issue with -O2 and -flto. I need to still check why we don’t see the issue in our downstream compiler. Will update when I have additional information. Regards, Shivaram