Software/EasyBuild

Compile BDSIM

James Chappell's question

Dear Laurie,

I’m trying to learn how to use the version of easy build that you setup to run some BDSIM simulations for AWAKE. I have a few questions for you.

1) The version of bdsim that you wrote the shell script for within /unix/pbt/software/eb/bdsim.sh does not have the AWAKE module compiled. Do you know if there is another version that I can link to which does for testing purposes for now?

2) Having looked through your eb files in /unix/pbt/software/eb/accsoft-eb-nevay, I can see that you define sources using a url and I’m assuming that in the process of compiling all the software, this source code is downloaded and then compiled. Is there a way to define the source from within the local system directories and then compile this version? Furthermore, to compile the AWAKE module, normally I use “ccmake .” to edit the compilation options - is the equivalent using the “configopts” command within an *.eb script? I’m assuming the command will be something like: “-USE_AWAKE=on”?

3) For our simulations, we edit the source code of the AWAKE module to make changes to the spectrometer and then recompile BDSIM before using it. How will this setup need to change when using easy build? Will it require recompiling the whole system of dependencies every time (I can see from your email to Simon previously that this took 5 hours)? Is it the case that we can compile everything up to BDSIM in one place (i.e. Geant4 and all its dependencies), and then just load our own version of BDSIM (or even just the AWAKE module) on top of this?

Sorry for all the questions, I’m just trying to get my head around how it all works! Thanks for your help, James.

The answer

Dear James,

1) The version of bdsim that you wrote the shell script for within /unix/pbt/software/eb/bdsim.sh does not have the AWAKE module compiled. Do you know if there is another version that I can link to which does for testing purposes for now?

My bad -> I'll recompile it with awake on. I need a time slot when you won't be using it as it'll disappear during the rebuilding - pretty bad if someone's running jobs when I do it! Let me know when I can do this - only takes about 15 mins.

Is there a way to define the source from within the local system directories and then compile this version? It looks like the system really only is designed to download things from what I can read online... Despite the url, it caches them in a sources directory, so if it was already there it would use that (a fudge though).

Furthermore, to compile the AWAKE module, normally I use “ccmake .” to edit the compilation options - is the equivalent using the “configopts” command within an *.eb script? I’m assuming the command will be something like: “-USE_AWAKE=on”?

We have so far in the Bdsim-1.1-GCC-4.9.4.eb easy build file:

configopts = ' -DCMAKE_CXX_FLAGS="-std=c++11"'

We can add to this:

configopts += '-DUSE_AWAKE=ON'

See ROOT-v6.08.06-GCC-4.9.3-Python-2.7.12.eb for lots of examples.

3) For our simulations, we edit the source code of the AWAKE module to make changes to the spectrometer and then recompile BDSIM before using it. How will this setup need to change when using easy build? Will it require recompiling the whole system of dependencies every time (I can see from your email to Simon previously that this took 5 hours)? Is it the case that we can compile everything up to BDSIM in one place (i.e. Geant4 and all its dependencies), and then just load our own version of BDSIM (or even just the AWAKE module) on top of this?

If you're modifying the C++ often I'd make your own build of BDSIM + AWAKE separately from the EasyBuild system and not bother trying to package it into a module. The module system is really for things that won't change.

You can get all the dependencies from EasyBuild and then make your own independent build of whatever you want. I think this was done for Gate for example and we do this just now for our nightly testing of BDSIM. I load the BDSIM module, then unload it - this leaves all the dependencies loaded.

module purge module load Bdsim module unload Bdsim module list

1. still has all libraries loaded including Geant4 etc..
2. somewhere else, make a build dir and build from your source.

When you load Geant4 and ROOT, note you should really source their setup scripts:

source $EBROOTGEANT4/bin/geant4.sh source $EBROOTROOT/bin/thisroot.sh

--->>> You should make a copy of the bdsim.sh script that does most of this and just add the unload to the end.

then...

mkdir build cd build cmake ../relative/path/to/bdsim -DCMAKE_CXX_COMPILER=/home/accsoft/SL68/software/GCC/4.9.3/bin/g++

-DCMAKE_C_COMPILER=/home/accsoft/SL68/software/GCC/4.9.3/bin/gcc
-DCLHEP_LIBRARY_DIR=/home/accsoft/SL68/software/CLHEP/2.3.1.1-GCC-4.9.3/lib/

make -j10

• paths are example on my system at RHUL

Normally you wouldn't have to specify these options as CMake would find these on its own. However, practice shows it often finds a system compiler before it finds the one in easybuild (this is required to be the same for all the software to be consistent), so we explicitly specify them.

Note once a build is make and it finds a compiler, the compiler can never be changed in CMake. You must wipe the build directory contents to try again.

How do you use this new build? Simple, you must have the same environment loaded as when it was built (so hence the modified bdsim.sh) then execute from that directory. This could easily be done in your shell script for the simulation or via an alias (fictional paths here as an example)...

alias bdsim-awake-devel="/home/jchappell/my-builds/build123/bdsim" bdsim-awake-devel --help

Note if you haven't changed the output or the analysis you can just use the regular bdsim from easybuild if you do any analysis later on the output. If you have changed it, you'll need to use your build and modify your bdsim.sh - pretty clear where to change for the analysis stuff.

Ok, hopefully that should help you both get the build working and help you understand.

Cheers, Laurie

Use modules

As Simon says, I've made modules ('environment modules for unix') on the system using EasyBuild. I've installed the latest Geant4 for BDSIM (our application) but I've also now put on the latest patch version of all recent Geant4 versions (was quite easy). You can access them from a CentOS7 node (I'm presuming Simon's here):

module use /unix/pbt/software/eb/CentOS7/modules/* module avail Geant4

.. this prints out the following

------------------------------------------------ /unix/pbt/software/eb/CentOS7/modules/all -------------------------------------------------
Geant4/10.00.p04-GCC-4.9.3 Geant4/10.01.p03-GCC-4.9.3 Geant4/10.02.p03-GCC-4.9.3 Geant4/10.03.p03-GCC-4.9.3 Geant4/10.04.p02-GCC-4.9.3

module load Geant4/10.04.p02-GCC-4.9.3 source $EBROOTGEANT4/bin/geant4.sh

The last line sources the required environmental variable for Geant4 to be found. This will load the corresponding CLHEP version required (multiple available).

To see what's loaded:

module list

Currently Loaded Modulefiles:

1) GCC/4.9.3                            9) bzip2/1.0.6-GCC-4.9.3               17) Mesa/11.2.1-GCC-4.9.3
2) expat/2.1.1-GCC-4.9.3               10) freetype/2.6.3-GCC-4.9.3            18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1
3) CLHEP/2.3.3.0-GCC-4.9.3             11) fontconfig/2.11.95-GCC-4.9.3        19) libffi/3.2.1-GCC-4.9.3
4) cURL/7.44.0-GCC-4.9.3               12) X11/20160819-GCC-4.9.3              20) gettext/0.19.6-GCC-4.9.3
5) Xerces-C++/3.1.2-GCC-4.9.3          13) libdrm/2.4.68-GCC-4.9.3             21) PCRE/8.38-GCC-4.9.3
6) zlib/1.2.8-GCC-4.9.3                14) ncurses/6.0-GCC-4.9.3               22) GLib/2.49.5-GCC-4.9.3
7) libxml2/2.9.3-GCC-4.9.3             15) LLVM/3.8.0-GCC-4.9.3                23) Qt5/5.7.0-GCC-4.9.3
8) libpng/1.6.21-GCC-4.9.3             16) eudev/3.1.5-GCC-4.9.3               24) Geant4/10.04.p02-GCC-4.9.3

To get rid of these loaded modules:

module purge

To see all modules:

module avail

There's the latest ROOT 6 on there too as well as a few versions of CLHEP (note Geant4 versions use different ones).

The module use and module load should be done in each new environment / terminal.

I tried making an easybuild recipe for GATE, but it fails at the end (I even tried a patch to remove their hard-coded make files for gjm) but still no success.

Install modules

• BDSIM has to use the single threaded version -> "high throughput" only vs "high performance"
• BDSIM requires external CLHEP so Geant4's internal one isn't mixed in -> otherwise, whilst BDSIM would work, the random number generator found is unclear and the results are not strongly reproducible.

Example command: eb Bdsim-1.2.1-GCC-4.9.3.eb --prefix=/unix/pbt/software/eb/CentOS7 -robot --robot-path=./: --modules-tool=EnvironmentModulesC --module-syntax=Tcl --allow-modules-tool-mismatch --optarch=GENERIC --buildpath=/tmp/easybuild/

The trick to get it to not replace a current version is to rename the module in the easybuild file. When you make one, easybuild checks for ones of the same name and version and if one with the same name and recipe is built it won't build it. If the options are updated, I believe it will update the build. Anyway, if you change the name it will create a new one. We used the name also as part of the source URL, so you should just hard code that. I've made an example here:

/unix/pbt/software/eb/accsoft-eb-nevay/Geant4-10.04.p02-mt-GCC-4.9.3.eb (copied from the regular geant4 one, turned multithreading on, changed version to have "-mt" and hardcoded source url)

When you build a module, you should do the module use command so it can find the others already available (it checks the requirements of the one you're building) and also the EasyBuild module itself. You should have a clean environment with no other modules loaded.

/unix/pbt/software/eb/easybuildonly.sh

seems to provide this.

I tried my example and found you should edit the version rather than the name for it to work. I've done this and it's setup now. You can import it as.

module load Geant4/10.04.p02-mt-GCC-4.9.3

Cheers, Laurie

PS we keep a note of the big incantation. You could get around most of these with some environmental variables, but we got with this so far.

PPS, remember you should source the geant4.sh from that installation before use. source $EBROOTGEANT4/bin/geant4.sh