Parallel make install is missing dependencies

[mweinelt]

For nixpkgs we are converting our install hook to parallelize installation, which yields great speedups with e.g., openssl.

For solanum it was noticed, that it does not play well with for example make -j6 install. It consistently fails like this:

solanum> /nix/store/1qasgqvab0xh2jcy00x9b1zh39dw7m8f-binutils-2.40/bin/ld: cannot find ./.libs/libircd.so: No such file or directory
solanum> collect2: error: ld returned 1 exit status
solanum> make[4]: *** [Makefile:633: solanum] Error 1
solanum> make[4]: *** Waiting for unfinished jobs....

Solanum is one of 12 packages, for which we had to disable parallel install. Would be amzing if this dependency could get set up.

[aaronmdjones]

I'm having trouble reproducing this.

Specifically; the following has worked for 10 consecutive times:

$ rm -rf /home/aaron/ircd/{bin,lib,modules}/ && \
    ./configure --prefix=/home/aaron/ircd/ && \
    make -j install && \
    make distclean && \
    echo ok

Furthermore, if I'm reading the output of your error log properly, the problem is that it's trying to build ircd/solanum before building ircd/libircd.la (from ircd/Makefile.am). However, solanum_LDADD has libircd.la in it, so automake ought not to be generating a Makefile that attempts to do so, which I'm preliminarily confirming by looking at ircd/Makefile.in and observing solanum_DEPENDENCIES = libircd.la

Can you provide more information?

[trofi]

Initially we observed the install failure on make install -j2. Here is the full build log: https://hydra.nixos.org/log/6d9j7mvknsyvkhf38v238m4g18ra30cx-solanum-unstable-2022-07-12.drv

A few suggestions to trigger it effectively:

• use --shuffle option for GNU make 4.4 or later to reorder prerequisite execution
• along with high parallelism also try to use low parallelism level, like -j2. that gives more chance to execute only subset of prerequisites
• make sure your /usr/lib/libircd.so (or other system pah) is not present. Otherwise you would not be able to reproduce the failure as relinking will accidentally happen against the system library.

For me using the above against solanum from e593329 commit (latest main) I get mainly install failures (and less frequently successes):

# a bit too specific to my setup
$ while :; do nix build --impure --expr 'with import ~/n {}; solanum.overrideAttrs (oa: { src = builtins.fetchGit ./.; })' --cores 2 --rebuild; echo again; done

error: builder for '/nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv' failed with exit code 2;
       last 10 log lines:
       > more information, such as the ld(1) and ld.so(8) manual pages.
       > ----------------------------------------------------------------------
       > make[4]: Leaving directory '/build/source/ircd'
       > make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2912249958
       > make[3]: Leaving directory '/build/source/ircd'
       > make[2]: *** [Makefile:824: install] Error 2 shuffle=2912249958
       > make[2]: Leaving directory '/build/source/ircd'
       > make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2912249958
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:776: install] Error 2 shuffle=2912249958
       For full logs, run 'nix log /nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv'.
error:
again
error: builder for '/nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv' failed with exit code 2;
       last 10 log lines:
       > more information, such as the ld(1) and ld.so(8) manual pages.
       > ----------------------------------------------------------------------
       > make[4]: Leaving directory '/build/source/ircd'
       > make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2889377224
       > make[3]: Leaving directory '/build/source/ircd'
       > make[2]: *** [Makefile:824: install] Error 2 shuffle=2889377224
       > make[2]: Leaving directory '/build/source/ircd'
       > make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2889377224
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:776: install] Error 2 shuffle=2889377224
       For full logs, run 'nix log /nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv'.
again
error: builder for '/nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv' failed with exit code 2;
       last 10 log lines:
       > more information, such as the ld(1) and ld.so(8) manual pages.
       > ----------------------------------------------------------------------
       > make[4]: Leaving directory '/build/source/ircd'
       > make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2869706743
       > make[3]: Leaving directory '/build/source/ircd'
       > make[2]: *** [Makefile:824: install] Error 2 shuffle=2869706743
       > make[2]: Leaving directory '/build/source/ircd'
       > make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2869706743
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:776: install] Error 2 shuffle=2869706743
       For full logs, run 'nix log /nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv'.
error:

Thus I should be able to verify the fix if you will have a hard time triggering it on your machine.

Looking at the failure specifics (against current main) this is the full install log:

installing
install flags: -j2 SHELL=/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash install
make  install-recursive
make[1]: Entering directory '/build/source'
Making install in librb
make[2]: Entering directory '/build/source/librb'
make  install-recursive
make[3]: Entering directory '/build/source/librb'
Making install in src
make[4]: Entering directory '/build/source/librb/src'
make  install-am
make[5]: Entering directory '/build/source/librb/src'
make[6]: Entering directory '/build/source/librb/src'
make[6]: Nothing to be done for 'install-data-am'.
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/mkdir -p '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
 /nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash ../libtool --silent   --mode=install /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c   librb.la '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
make[6]: Leaving directory '/build/source/librb/src'
make[5]: Leaving directory '/build/source/librb/src'
make[4]: Leaving directory '/build/source/librb/src'
make[4]: Entering directory '/build/source/librb'
make[5]: Entering directory '/build/source/librb'
make[5]: Nothing to be done for 'install-exec-am'.
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/mkdir -p '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/pkgconfig'
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c -m 644 librb.pc '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/pkgconfig'
make[5]: Leaving directory '/build/source/librb'
make[4]: Leaving directory '/build/source/librb'
make[3]: Leaving directory '/build/source/librb'
make[2]: Leaving directory '/build/source/librb'
Making install in ircd
make[2]: Entering directory '/build/source/ircd'
/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash version.c.SH
Extracting solanum/src/version.c...
/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/cp version.c version.c.last
touch version.c.SH
make  install-am
make[3]: Entering directory '/build/source/ircd'
/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash version.c.SH
Extracting solanum/src/version.c...
/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/cp version.c version.c.last
touch version.c.SH
  CC       version.lo
  CCLD     libircd.la
  CCLD     solanum
make[4]: Entering directory '/build/source/ircd'
/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash version.c.SH
Extracting solanum/src/version.c...
/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/cp version.c version.c.last
touch version.c.SH
  CC       version.lo
  CCLD     libircd.la
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/mkdir -p '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
 /nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash ../libtool   --mode=install /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c   libircd.la '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
  CCLD     solanum
libtool: warning: relinking 'libircd.la'
libtool: install: (cd /build/source/ircd; /nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash "/build/source/libtool"  --silent --tag CC --mode=relink gcc -g -O2 -avoid-version -no-undefined -Wl,-rpath=/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib -o libircd.la -rpath /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib authproc.lo bandbi.lo cache.lo capability.lo channel.lo chmode.lo class.lo client.lo dns.lo extban.lo getopt.lo hash.lo hook.lo hostmask.lo ircd.lo ircd_parser.lo ircd_lexer.lo ircd_signal.lo listener.lo logger.lo match.lo modules.lo monitor.lo msgbuf.lo newconf.lo operhash.lo packet.lo parse.lo privilege.lo ratelimit.lo reject.lo restart.lo s_conf.lo s_newconf.lo s_serv.lo s_user.lo scache.lo send.lo snomask.lo sslproc.lo substitution.lo supported.lo tgchange.lo version.lo whowas.lo wsproc.lo -lltdl -L../librb/src -lrb )
/nix/store/r7yp62k0cqhm59nrvn8wn1xsz0kl3sgv-binutils-2.40/bin/ld: cannot find ./.libs/libircd.so: No such file or directory
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:633: solanum] Error 1 shuffle=2854295066
make[4]: *** Waiting for unfinished jobs....
libtool: install: /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c .libs/libircd.soT /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/libircd.so
libtool: install: /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c .libs/libircd.lai /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/libircd.la
libtool: finish: PATH="/nix/store/xwd7pd4fd4zl28m50jczvf7k5rynsi71-autoconf-2.69/bin:/nix/store/q4pv8wl5p7wwx34ggcm6fv8li7qr8wkb-automake-1.16.5/bin:/nix/store/wp9hly5270bxy5ldnv4n6rrb7yc20zxb-gettext-0.21/bin:/nix/store/m583s17s13n9d81qnzd37c0d37li2z7p-libtool-2.4.7/bin:/nix/store/yk6g0lc7sgqghf7dxv654kz4sy8ab7iv-gnum4-1.4.19/bin:/nix/store/hx73nmjw84hxk5jnf081ncp1jybish8w-file-5.44/bin:/nix/store/x3w26b833bhmvxd3wyzhfva94dvllvnm-bison-3.8.2/bin:/nix/store/4kqa6bdhnnllpglmdhy0n0zw1im4lair-flex-2.6.4/bin:/nix/store/k66cg7smlvlc4dylny6fsp8cw2c9g52g-pkg-config-wrapper-0.29.2/bin:/nix/store/i8sfwz7pmxar03k98k5vwqkv0w00inmy-util-linux-2.38.1-bin/bin:/nix/store/fc0ypma9gi7r85qvn22w5x5934br6a3r-patchelf-0.17.2/bin:/nix/store/69iqwrsr5dcfi2fsqf5w3hmx15pmfaqk-gcc-wrapper-13.0.0/bin:/nix/store/v626s1w0s6hdnjxlaig37q9wzqm63ah9-gcc-13.0.0/bin:/nix/store/w7kz70f1dw8waf803lddv3isb9bhjpfk-glibc-2.35-224-bin/bin:/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin:/nix/store/vrk8hj570x5nm2j65pc3j634ygcc63hx-binutils-wrapper-2.40/bin:/nix/store/r7yp62k0cqhm59nrvn8wn1xsz0kl3sgv-binutils-2.40/bin:/nix/store/173mgp8qnpzs5bznhh69kdyy4mlk62bn-openssl-3.0.8-bin/bin:/nix/store/35nb54v62wnmh2ix4zp84ycnmgvavjm8-sqlite-3.41.1-bin/bin:/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin:/nix/store/g5pbad773hzx4j54i5dn22bkjzzigb6h-findutils-4.9.0/bin:/nix/store/285vm1zn78dy87adny0z0sbwkwbyrwvi-diffutils-3.9/bin:/nix/store/7xa8bc3ipyz1y9jl1p6iyjb30vxfk1zq-gnused-4.9/bin:/nix/store/2iaqkjcdma1bw19g0dg1zybc9w2jq413-gnugrep-3.7/bin:/nix/store/1l3vkqzp7bppjhh06w7lgfp1662nk26h-gawk-5.2.1/bin:/nix/store/cs2m8536x9hrbxd4yzg2haspy368757k-gnutar-1.34/bin:/nix/store/iykzhqbr570zisqcvdyq6s3hh42zns4f-gzip-1.12/bin:/nix/store/knz3cacv109gzvf75mgag6bvjqvk9ra2-bzip2-1.0.8-bin/bin:/nix/store/2p4ri16y7rjyir3l9qdxic5mdnshix5b-gnumake-4.4.1/bin:/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin:/nix/store/b77k73j5is2s7c5af87rv7p06765bhwi-patch-2.7.6/bin:/nix/store/nicb37s3z16116kqyaq959i8gs0pvmi6-xz-5.4.1-bin/bin:/nix/store/pqihdci4c2d8g1ryihk8wh654pnwxfvx-file-5.44/bin:/sbin" ldconfig -n /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib
----------------------------------------------------------------------
Libraries have been installed in:
   /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[4]: Leaving directory '/build/source/ircd'
make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2854295066
make[3]: Leaving directory '/build/source/ircd'
make[2]: *** [Makefile:824: install] Error 2 shuffle=2854295066
make[2]: Leaving directory '/build/source/ircd'
make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2854295066
make[1]: Leaving directory '/build/source'
make: *** [Makefile:776: install] Error 2 shuffle=2854295066

I find it suspicions that libircd.la is attempted to be built multiple times (and CC is executed at all during the install). I wonder if it gets deleted as a temporary eventually. I'll add V=1 to get more detailed logs.

[trofi]

solanum-unstable-V-1.log.txt is the GNUMAKEFLAGS=V=1 make install -j2 with a similar failure. Could it be that version.c.SH rebuild causes removal of .libs/libircd.so?

I find this multiple removals suspicious:

$ cat /tmp/solanum-unstable-V-1.log.txt | grep -P 'rm.*libircd.so'

libtool: link:  gcc -shared  -fPIC -DPIC  .libs/extb_usermode.o   -Wl,-rpath -Wl,/build/source/ircd/.libs -Wl,-rpath -Wl,/nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib -Wl,-rpath -Wl,/build/source/librb/src/.libs -Wl,-rpath -Wl,/nix/store/xvf1sawg92x6cp8plw34ica3l5y88qvb-solanum-unstable-2022-07-12/lib -Wl,-rpath -Wl,/nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib -L/build/source/librb/src/.libs ../ircd/.libs/libircd.so -L/nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib -L../librb/src -L/nix/store/v3y5ds8z0y128gspghnmkrxxdh3xc4j8-openssl-3.0.8-dev/lib /nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib/libltdl.so /build/source/librb/src/.libs/librb.so -lssl -lcrypto  -g -O2 -Wl,-rpath=/nix/store/xvf1sawg92x6cp8plw34ica3l5y88qvb-solanum-unstable-2022-07-12/lib   -Wl,-soname -Wl,extb_usermode.so -Wl,-version-script -Wl,.libs/extb_usermode.ver -o .libs/extb_usermode.so
libtool: link: rm -fr  .libs/libircd.la .libs/libircd.lai .libs/libircd.so
libtool: link: rm -fr  .libs/libircd.la .libs/libircd.lai .libs/libircd.so
libtool: link: rm -fr  .libs/libircd.la .libs/libircd.lai .libs/libircd.so

[trofi]

I think it gets rebuild indefinitely because of touch version.c.SH:

version.c: version.c.SH ../CREDITS ../include/patchlevel.h ../include/serno.h
        $(SHELL) version.c.SH
        $(CP) version.c version.c.last
        touch version.c.SH

That touch version.c.SH makes this rule fire forever. Should it be touch version.c instead?

[trofi]

On top of that removal of the source in the middle of install looks fishy as well:

ircd/Makefile.am-install-exec-hook: install-libircdLTLIBRARIES
ircd/Makefile.am:       $(RM) -f version.c

[aaronmdjones]

If you remove the touch version.c.SH line entirely (version.c.SH already updates version.c) and remove that $(RM) -f version.c line (but not the line above it expressing an installation dependency), is the problem resolved? I am thus far unable to reproduce this on any parallelism level and my make isn't new enough for the shuffle feature.

[trofi]

It was not enough to fix the install. I tried the following change:

--- a/ircd/Makefile.am
+++ b/ircd/Makefile.am
@@ -12,7 +12,6 @@ BUILT_SOURCES = ircd_parser.h version.c
 version.c: version.c.SH ../CREDITS ../include/patchlevel.h ../include/serno.h
        $(SHELL) version.c.SH
        $(CP) version.c version.c.last
-       touch version.c.SH

 libircd_la_SOURCES =                  \
   authproc.c                   \
@@ -71,4 +70,3 @@ solanum_LDFLAGS = $(EXTRA_FLAGS) -dlopen self
 solanum_LDADD = libircd.la -L$(top_srcdir)/librb/src -lrb

 install-exec-hook: install-libircdLTLIBRARIES
-       $(RM) -f version.c

Still fails as:
solanum-unstable-V-1-patched.log.txt

Looking at the log it sill tries to relink libircd.la twice:

libtool: warning: relinking 'libircd.la'
...
libtool: warning: relinking 'libircd.la'
...
ldconfig: File /nix/store/4d5imqc90dqhmi5g22y2mlrfdxxyyxyy-solanum-unstable-2022-07-12/lib/libircd.so is empty, not checked.

[trofi]

But if I remove install-exec-hook: install-libircdLTLIBRARIES dependency my rebuild tests survives 10 rebuilds without failures:

--- a/ircd/Makefile.am
+++ b/ircd/Makefile.am
@@ -12,7 +12,6 @@ BUILT_SOURCES = ircd_parser.h version.c
 version.c: version.c.SH ../CREDITS ../include/patchlevel.h ../include/serno.h
        $(SHELL) version.c.SH
        $(CP) version.c version.c.last
-       touch version.c.SH

 libircd_la_SOURCES =                  \
   authproc.c                   \
@@ -69,6 +68,3 @@ bin_PROGRAMS = solanum
 solanum_SOURCES = main.c
 solanum_LDFLAGS = $(EXTRA_FLAGS) -dlopen self
 solanum_LDADD = libircd.la -L$(top_srcdir)/librb/src -lrb
-
-install-exec-hook: install-libircdLTLIBRARIES
-       $(RM) -f version.c

Looking at https://www.gnu.org/software/automake/manual/html_node/Extending-Installation.html I think the desired ordering should already be implied.