Use BinaryBuilder & BinaryProvider

[giordano]

After #15 is fixed, would it make sense to build ROOT with BinaryBuilder.jl and install the dependency here with BinaryProvider.jl? I'll probably be able to have a look at this if you're interested

[oschulz]

I think so, yes. Having ROOT installed automatically (only if not present on PATH, so people can choose a specific build) was always kind of in the back of my mind. Would also help with CI testing. Would it make sense to use the official CERN builds where applicable?

[giordano]

The way BinaryBuilder + BinaryProvider work is that one builds from source the library for all platforms (or at least those for which they succeed to build the library), currently:

julia> using BinaryBuilder

julia> supported_platforms()
13-element Array{Platform,1}:
 Linux(:i686, libc=:glibc)                    
 Linux(:x86_64, libc=:glibc)                  
 Linux(:aarch64, libc=:glibc)                 
 Linux(:armv7l, libc=:glibc, call_abi=:eabihf)
 Linux(:powerpc64le, libc=:glibc)             
 Linux(:i686, libc=:musl)                     
 Linux(:x86_64, libc=:musl)                   
 Linux(:aarch64, libc=:musl)                  
 Linux(:armv7l, libc=:musl, call_abi=:eabihf) 
 MacOS(:x86_64)                               
 FreeBSD(:x86_64)                             
 Windows(:i686)                               
 Windows(:x86_64)

Then they upload the resulting artifacts somewhere (usually a release page in a GitHub repository) and BinaryProvider downloads and unpacks the tarball for the desired platform.

only if not present on PATH

The idea of BinaryProvider is to always install the libraries in this way, not "only if not already available". I can see that this is not really ideal, but it has its advantages in terms of reproducibility (a given version of the Julia package is linked to a specific version of the binary library it uses) and ease of installation (you're almost sure that the library will be installed correctly)

Would it make sense to use the official CERN builds where applicable?

Well, theoretically we could ignore not use BinaryBuilder to build the package and just reuse the prebuilt tarballs on CERN's website. The only problem is that BinaryProvider expects the archive to have a specific structure (e.g., no top directory), so we couldn't directly use the official tarballs as they are because they have a "Root" top directory.

---

I just realised that actually you build a ROOT-compatible Julia, rather than a Julia-compatible ROOT. Perhaps @staticfloat might be interested in this use case.

[staticfloat]

The only problem is that BinaryProvider expects the archive to have a specific structure (e.g., no top directory)

Note that this can be worked around by setting up the products in your build.jl properly, but by default, yes this is true. :) (these little tricks abound!)

I just realised that actually you build a ROOT-compatible Julia

What needs to be special about Julia in order to work with ROOT?

[oschulz]

What needs to be special about Julia in order to work with ROOT?

ROOT also uses LLVM, and it uses a specially modified version.

ROOT is unhappy if it finds an active LLVM instance when it starts - it will print a warning, and dependent on the current LLVM version in ROOT and in Julia, you get segfaults and crashes (or not, for some combination of LLVM versions). The workaround ROOT.jl uses (@jpata came up with it) is to build a special julia executable that loads ROOT before it starts Julia. I tried other approaches (like LD_PRELOAD), but that's the only way that has worked consistently so far.

I'm pretty sure this is ROOT's fault, not Julia's.

[staticfloat]

Would it be helpful to try building ROOT against the exact version of LLVM that Julia uses? Julia is also quite opinionated about the LLVM it uses, as I'm sure you're aware. ;)

If you think that might help, you can get prebuilt tarballs of the versions of LLVM that Julia uses from here; where each of those tarballs is compiled for a different platform, linked against different compiler support libraries (e.g. if you wanted to ROOT on an x86_64-linux-gnu machine that is using GCC 4, you'd use this tarball).

[oschulz]

I just realised that actually you build a ROOT-compatible Julia, rather than a Julia-compatible ROOT.

That would be interesting - if we could find a way to do that in a clean fashion, I think we could even get the required changes merged into ROOT.

However, I don't know how tricky this is - I never looks into what exactly goes wrong there, deep down, and I'm not an LLVM expert, so I can't judge, really.

[oschulz]

Would it be helpful to try building ROOT against the exact version of LLVM that Julia uses? Julia is also quite opinionated about the LLVM it uses, as I'm sure you're aware. ;)

Yes, I know. :-) There are, however, some problems here:

• I'm not sure if in the instances where things did work (despite ROOT's warning), the LLVM versions were very similar or very different. We can try though.

• ROOT's LLVM is modified for ROOT's own C++ REPL and some other things. So I don't think we could build ROOT against the Julia's LLVM binary. It may be enough if we use the same LLVM base version, though.

• AFAIK specific ROOT versions only work with specific LLVM versions. This means that we'd have to find a ROOT version that brings the right LLVM version with it (and there might not be one, if the match has to be exact). Also, physicists often need to (or are required to) work with a specific ROOT version that is kept fixed for a long time, esp. in larger collaborations. Depending on the environment, that ROOT version may also need to be built with specific options.

Still, it may be workable, but it might get tricky.

[oschulz]

@staticfloat, If you're interested to take a look at this, I can put together a short Julia script that will load Cxx and ROOT and demonstrate the warning and the crash (without ROOT.jl, even).

[oschulz]

Here's a script that will demonstrate the problem:

const ROOT_INCDIR = String(strip(read(`root-config --incdir`, String)))
const ROOT_LIBDIR = String(strip(read(`root-config --libdir`, String)))
const SHEXT = "so"

using Cxx
using Libdl: Libdl

function load_rootlib(libname)
    libpath = joinpath(ROOT_LIBDIR, "lib$libname.$SHEXT")
    isfile(libpath) || error("could not find ROOT library $libname at \"$libpath\"")
    Libdl.dlopen(libpath, Libdl.RTLD_GLOBAL)
end


addHeaderDir(ROOT_INCDIR, kind=C_System)

load_rootlib.([
    "Core", "Gpad", "Graf", "Graf3d", "Hist", "MathCore", "Matrix", "MultiProc", "Net",
    "Physics", "Postscript", "Rint", "RIO", "Thread", "Tree"
])


cxxinclude("TFile.h")

f = icxx"""new TFile("out.root", "recreate");"""

This results in

Error in <UnknownClass::InitInterpreter()>: LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!

 *** Break *** segmentation violation
 [...]

(tested just now on CentOS-7 with Julia v1.2.0-rc1.0 and ROOT 6.16/00, but observed with many combinations of Julia and ROOT versions in the past).

To run the script about, ROOT's "bin" directory, containing root-config, but be on $PATH.

Official binary builds of ROOT for Ubuntu, CentOS, Fedora, OS-X and Windows are available here: https://root.cern.ch/downloading-root

[StefanKarpinski]

I wonder if it's possible to build an LLVM that is compatible with both Julia and ROOT and then arrange for that to be the LLVM that Julia uses and then load ROOT via binary builder. The tricky part would be arranging for Julia to load the right LLVM, but it might be possible by installing LLVM_ROOT_jll or whatever it's called and then setting the DL_LIBRARY_PATH to point to that and restarting Julia, which wouldn't be a completely horrible user experience.

[oschulz]

That would be a nice clean solution. I'm not sure how much it would limit Julia and ROOT versions, since both are kinda opinionated about that (ROOT likely a lot more than JULIA). I'm definitely not an LLVM expert though. @StefanKarpinski, would someone on the Julia core team have time and interest to talk about this with the ROOT team? That might yield clarity very quickly - but I realize you guys are quite busy.

[oschulz]

If so, I could definitely get you in contact with the right people at CERN.

[StefanKarpinski]

I think the thing to make progress would be to get a description of what version of LLVM ROOT needs and how it modifies LLVM.

[oschulz]

@lmoneta, may I pull you into this discussion, as a member of the ROOT team?

[vgvassilev]

Hi all, thanks for looking into this!

ROOT's fork of LLVM and Clang are based on version 9. The LLVM fork is essentially patch free, however the Clang fork has a set of patches which can be seen here.

I believe the requirement how LLVM is loaded (per #17 (comment)) was relaxed. Could you try again?