Software/EasyBuild

Compile BDSIM

James Chappell's question

Dear Laurie,

I’m trying to learn how to use the version of easy build that you setup to run some BDSIM simulations for AWAKE. I have a few questions for you.

1) The version of bdsim that you wrote the shell script for within /unix/pbt/software/eb/bdsim.sh does not have the AWAKE module compiled. Do you know if there is another version that I can link to which does for testing purposes for now?

2) Having looked through your eb files in /unix/pbt/software/eb/accsoft-eb-nevay, I can see that you define sources using a url and I’m assuming that in the process of compiling all the software, this source code is downloaded and then compiled. Is there a way to define the source from within the local system directories and then compile this version? Furthermore, to compile the AWAKE module, normally I use “ccmake .” to edit the compilation options - is the equivalent using the “configopts” command within an *.eb script? I’m assuming the command will be something like: “-USE_AWAKE=on”?

3) For our simulations, we edit the source code of the AWAKE module to make changes to the spectrometer and then recompile BDSIM before using it. How will this setup need to change when using easy build? Will it require recompiling the whole system of dependencies every time (I can see from your email to Simon previously that this took 5 hours)? Is it the case that we can compile everything up to BDSIM in one place (i.e. Geant4 and all its dependencies), and then just load our own version of BDSIM (or even just the AWAKE module) on top of this?

Sorry for all the questions, I’m just trying to get my head around how it all works! Thanks for your help, James.

The answer

Dear James,

1) The version of bdsim that you wrote the shell script for within /unix/pbt/software/eb/bdsim.sh does not have the AWAKE module compiled. Do you know if there is another version that I can link to which does for testing purposes for now?

My bad -> I'll recompile it with awake on. I need a time slot when you won't be using it as it'll disappear during the rebuilding - pretty bad if someone's running jobs when I do it! Let me know when I can do this - only takes about 15 mins.

Is there a way to define the source from within the local system directories and then compile this version? It looks like the system really only is designed to download things from what I can read online... Despite the url, it caches them in a sources directory, so if it was already there it would use that (a fudge though).

Furthermore, to compile the AWAKE module, normally I use “ccmake .” to edit the compilation options - is the equivalent using the “configopts” command within an *.eb script? I’m assuming the command will be something like: “-USE_AWAKE=on”?

We have so far in the Bdsim-1.1-GCC-4.9.4.eb easy build file:

configopts = ' -DCMAKE_CXX_FLAGS="-std=c++11"'

We can add to this:

configopts += '-DUSE_AWAKE=ON'

See ROOT-v6.08.06-GCC-4.9.3-Python-2.7.12.eb for lots of examples.

3) For our simulations, we edit the source code of the AWAKE module to make changes to the spectrometer and then recompile BDSIM before using it. How will this setup need to change when using easy build? Will it require recompiling the whole system of dependencies every time (I can see from your email to Simon previously that this took 5 hours)? Is it the case that we can compile everything up to BDSIM in one place (i.e. Geant4 and all its dependencies), and then just load our own version of BDSIM (or even just the AWAKE module) on top of this?

If you're modifying the C++ often I'd make your own build of BDSIM + AWAKE separately from the EasyBuild system and not bother trying to package it into a module. The module system is really for things that won't change.

You can get all the dependencies from EasyBuild and then make your own independent build of whatever you want. I think this was done for Gate for example and we do this just now for our nightly testing of BDSIM. I load the BDSIM module, then unload it - this leaves all the dependencies loaded.

module purge module load Bdsim module unload Bdsim module list

1. still has all libraries loaded including Geant4 etc..
2. somewhere else, make a build dir and build from your source.

When you load Geant4 and ROOT, note you should really source their setup scripts:

source $EBROOTGEANT4/bin/geant4.sh source $EBROOTROOT/bin/thisroot.sh

--->>> You should make a copy of the bdsim.sh script that does most of this and just add the unload to the end.

then...

mkdir build cd build cmake ../relative/path/to/bdsim -DCMAKE_CXX_COMPILER=/home/accsoft/SL68/software/GCC/4.9.3/bin/g++

-DCMAKE_C_COMPILER=/home/accsoft/SL68/software/GCC/4.9.3/bin/gcc
-DCLHEP_LIBRARY_DIR=/home/accsoft/SL68/software/CLHEP/2.3.1.1-GCC-4.9.3/lib/

make -j10

• paths are example on my system at RHUL

Normally you wouldn't have to specify these options as CMake would find these on its own. However, practice shows it often finds a system compiler before it finds the one in easybuild (this is required to be the same for all the software to be consistent), so we explicitly specify them.

Note once a build is make and it finds a compiler, the compiler can never be changed in CMake. You must wipe the build directory contents to try again.

How do you use this new build? Simple, you must have the same environment loaded as when it was built (so hence the modified bdsim.sh) then execute from that directory. This could easily be done in your shell script for the simulation or via an alias (fictional paths here as an example)...

alias bdsim-awake-devel="/home/jchappell/my-builds/build123/bdsim" bdsim-awake-devel --help

Note if you haven't changed the output or the analysis you can just use the regular bdsim from easybuild if you do any analysis later on the output. If you have changed it, you'll need to use your build and modify your bdsim.sh - pretty clear where to change for the analysis stuff.

Ok, hopefully that should help you both get the build working and help you understand.

Cheers, Laurie

Additional info

Hello,

Ok, I've updated the general BDSIM build in easy build now. It should have the AWAKE module turned on. Note, this is the code of v1.1 and not the latest develop version.

I made an example environment only script in:

/unix/pbt/software/eb/bdsim-own-build.sh

I tried making my own build and it worked just fine. I only needed to do also:

module load CMake

to get a more up to date version. From my home dir:

mkdir build-custom-build cd build-custom-build cmake ../bdsim

1. check compiler is GCC4.9.3 form easy build, and ROOT and Geant4 and CLHEP -> all ok

ccmake . turn on USE_AWAKE <c> <g> make -j15 works!

It's in /home/lnevay/bdsim-custom-build

Cheers, Laurie

Use modules

Set path to modules

Before modules can be accessed, the module tool needs to be given the path to the desired set of modules. For example,

module use /unix/pbt/software/eb/CentOS7/modules/*

List available modules

You can then list all modules available with

module avail

or check what versions of a given module are available using

module avail [modulename]

For example,

module avail Geant4

prints out the following

-------------------------------------------------------------------- /unix/pbt/software/eb/CentOS7/modules/phys --------------------------------------------------------------------
Geant4/10.00.p04-GCC-4.9.3    Geant4/10.01.p03-GCC-4.9.3    Geant4/10.02.p03-GCC-4.9.3    Geant4/10.03.p03-GCC-4.9.3    Geant4/10.04.p02-GCC-4.9.3    Geant4/10.04.p02-mt-GCC-4.9.3

-------------------------------------------------------------------- /unix/pbt/software/eb/CentOS7/modules/all ---------------------------------------------------------------------
Geant4/10.00.p04-GCC-4.9.3    Geant4/10.01.p03-GCC-4.9.3    Geant4/10.02.p03-GCC-4.9.3    Geant4/10.03.p03-GCC-4.9.3    Geant4/10.04.p02-GCC-4.9.3    Geant4/10.04.p02-mt-GCC-4.9.3

Load modules

Load a given module using the command

module load

This will load the specified module along with all its dependencies. For instance,

module load Geant4/10.04.p02-GCC-4.9.3

loads Geant4 and all of the following

 1) GCC/4.9.3                            7) libxml2/2.9.3-GCC-4.9.3             13) libdrm/2.4.68-GCC-4.9.3             19) libffi/3.2.1-GCC-4.9.3
 2) expat/2.1.1-GCC-4.9.3                8) libpng/1.6.21-GCC-4.9.3             14) ncurses/6.0-GCC-4.9.3               20) gettext/0.19.6-GCC-4.9.3
 3) CLHEP/2.3.3.0-GCC-4.9.3              9) bzip2/1.0.6-GCC-4.9.3               15) LLVM/3.8.0-GCC-4.9.3                21) PCRE/8.38-GCC-4.9.3
 4) cURL/7.44.0-GCC-4.9.3               10) freetype/2.6.3-GCC-4.9.3            16) eudev/3.1.5-GCC-4.9.3               22) GLib/2.49.5-GCC-4.9.3
 5) Xerces-C++/3.1.2-GCC-4.9.3          11) fontconfig/2.11.95-GCC-4.9.3        17) Mesa/11.2.1-GCC-4.9.3               23) Qt5/5.7.0-GCC-4.9.3
 6) zlib/1.2.8-GCC-4.9.3                12) X11/20160819-GCC-4.9.3              18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1  24) Geant4/10.04.p02-GCC-4.9.3

Don't forget to source the relevant environment variables where required, e.g.

source $EBROOTGEANT4/bin/geant4.sh

Show loaded modules

To see what's loaded use

module list

Remove loaded modules

To get rid of all loaded modules:

module purge

Install modules

Installing modules

Example command to install module:

eb Bdsim-1.2.1-GCC-4.9.3.eb --prefix=/unix/pbt/software/eb/CentOS7 -robot --robot-path=./: --modules-tool=EnvironmentModulesC --module-syntax=Tcl --allow-modules-tool-mismatch --optarch=GENERIC --buildpath=/tmp/easybuild/

Two types of the same version (i.e. one single-threaded and one multi-threaded) can be installed side-by-side. To get EasyBuild to not replace a current version, the module can be renamed in the EasyBuild file. When a module is built, EasyBuild checks for modules of the same name and version and if one already exists, the build won't complete. If the options are updated, the build will be updated.

When you build a module, you should do the `module use` command so it can find the others already available (it checks the requirements of the one you're building) and also the EasyBuild module itself. You should have a clean environment with no other modules loaded.

/unix/pbt/software/eb/easybuildonly.sh

seems to provide this.

I tried my example and found you should edit the version rather than the name for it to work. I've done this and it's setup now. You can import it as.

module load Geant4/10.04.p02-mt-GCC-4.9.3

Cheers, Laurie

PS we keep a note of the big incantation. You could get around most of these with some environmental variables, but we got with this so far.

PPS, remember you should source the geant4.sh from that installation before use. source $EBROOTGEANT4/bin/geant4.sh

PS we keep a note of the big incantation. You could get around most of these with some environmental variables, but we got with this so far.

PPS, remember you should source the geant4.sh from that installation before use. source $EBROOTGEANT4/bin/geant4.sh

The question

Dear Laurie,

I'm trying to compile a multi-threaded version of Geant4 using EasyBuild but would like to make sure I don't break everything.

Looking at the easyconfig file for the most recent Geant4 version installed on our system and the directory structure of the `eb/CentOS7/software` directory, I can see that the location of the installation depends on the name and the version of the package and on the toolchain used to compile it. This leads me to believe that if I were to set `GEANT4_BUILD_MULTITHREADED=ON` in the easyconfig file and run the following command,

eb Geant4-10.04.p02-GCC-4.9.3.eb --robot

it would overwrite the current (non-mutli-threaded) version of Geant4. I have two questions relating to this:

1) Am I correct in thinking it would overwrite it? 2) If so, would you mind showing me how I can compile it alongside the existing version?

The best solution I could come up with would be to install it in a different location using the command line option `--installpath-software=INSTALLPATH-SOFTWARE` but I'm unsure if that breaks compatibility with the modules tool and I'd also like to avoid overwriting the existing easyconfig file, which I don't know how to ensure.

Note on how to install Geant4 on Mac

Hello,

Ah we tried it briefly on mac, but it was a little frustrating so we stuck with our usual way, which is macports. For your mac, I'd recommend that. I know it's different than EasyBuild, but it's generally easier for mac and for scientific software.

https://www.macports.org

Before getting it you should xcode + the command line tools (from app store) installed (test is you can run 'make' on the terminal). You should also have XQuartz (https://www.xquartz.org) installed for X11 support. Then download and install the macports software (small, little mac installer). Gives you a command on the terminal 'port'. Everything is by default installed in /opt/local so you can nuke that directory if you don't want it anymore. It works by adding a line to your profile that adds that directory to your paths.

port search packagename

sudo port install py27-matplotlib

I'd install everything up to CLHEP from macports, then make my own geant4 installation. I think there is a geant4 port on there, but I haven't used it. For the geant4 installation you download hte source somewhere, make a build directory (outside the source) and then an install directory. Usual CMake pattern.

I'd start with a high level package in macports and it'll work out all the requirements. Like...

sudo port install clhep qt5 py27-matplotlib

I do everything apart from geant4 and bdsim through macports.

Hope that helps, Cheers, Laurie

Summary from Raffy about EasyBuild, Environment Modules and GEANT4

Geant4

Reference

http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/gettingstarted.html http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/installguide.html

GEANT4 OS/Software Prerequisites

the following source/software must be present to build Geant4:

• Geant4 Toolkit Source Code.
• C++ Compiler and Standard Library supporting the C++11 Standard
• Linux (Scientific Linux CERN 6 and Linux CentOS 7 ): GNU Compiler Collection 4.8.5 or higher ( recommended to use the GCC compiler supplied by the package management system of your distribution unless this does not meet the minimum version requirement. The minimum required versions of GCC may be installed on SLC6 systems via the freeDeveloper Toolset)
• CMake 3.3 or higher (as provided through the package management system of your distribution, unless it does not meet the minimum version requirement.)

The main requirement is that the system has a GCC of sufficient version to support C++11 installed.

User Interface and Visualization Drivers

• Qt User Interface and Visualization
  • Qt4 (>=4.6) or Qt5 headers and libraries
• OpenGL or MesaGL headers and libraries.
  • X11 OpenGL Visualization (Linux and macOS)
• Open Inventor Visualization
• Coin3D libraries and headers with SoXt(SoWin) graphics binding on Linux/macOS(Windows), compiled against the C++11 standard.
• Motif User Interface and Visualization (Linux and macOS)

Standard GEANT4 Building and Installation

• Unpack the Geant4 source package geant4.10.05.tar.gz to a location of your choice: source directory /path/to/geant4.10.05
• create a directory in which to configure and run the build and store the build products /path/to/geant4.10.05-build
• $ cmake -DCMAKE_INSTALL_PREFIX=/path/to/geant4.10.05-install /path/to/geant4.10.05
• CMake Variable CMAKE_INSTALL_PREFIX is used to set the install directory, the directory under which the Geant4 libraries, headers and support files will be installed. It must be supplied as an absolute path. The second argument to CMake is the path to the source directory.
• Additional arguments may be passed to CMake to activate optional components of Geant4, such as visualization drivers, or tune the build and install parameters.
• If you run CMake and decide afterwards you want to activate additional options, simply rerun CMake in the build directory, passing it the extra options plus the build directory path.
• Datasets are not required to be present to build Geant4, but may be required to run your application, depending on the physics models you use. If you wish to download and install the datasets automatically as part of your build of Geant4, simply add the option-DGEANT4_INSTALL_DATA=ON to the arguments passed to CMake.
• After the configuration has run, CMake will have generated Unix Makefiles for building Geant4. To run the build, simply execute make in the build directory
• Once the build has completed, you can install Geant4 to the directory you specified earlier in CMAKE_INSTALL_PREFIX
• additional options can be passed to CMake to adjust the way Geant4 is built and installed and to enable optional components. Some options enable components of Geant4 which require external software.If these options are enabled, the required software will be searched for, and hence there are also options which control where CMake should look for these packages.These options may be set by passing their name and value to the cmake command via -D flags.

Standard Options

• CMAKE_INSTALL_PREFIX installation prefix for Geant4. (default Unix: /usr/local)
• CMAKE_BUILD_TYPE : (DEFAULT : Release) Sets the type of build. It defaults to Release which gives a fully optimized build without debugging symbols.
• GEANT4_BUILD_MULTITHREADED : (DEFAULT : OFF) If set to ON, build Geant4 libraries with support for multithreading. At present, this is only supported on Unix systems. (Requires: Compiler/C++ Standard Library with support for C++ Threading Support)
• GEANT4_INSTALL_DATA : (DEFAULT : OFF) to ON, download and install any Geant4 datasets missing from GEANT4_INSTALL_DATADIR.
• GEANT4_INSTALL_DATADIR : (DEFAULT : CMAKE_INSTALL_DATAROOTDIR) Installation directory for Geant4 datasets.
• GEANT4_USE_G3TOG4 : (DEFAULT : OFF) If set to ON, build the G3ToG4 library for reading ASCII call list files generated from Geant3 geometries.
• GEANT4_USE_GDML : (DEFAULT : OFF) If set to ON, build the G4persistency library with support for GDML.
• GEANT4_USE_QT (DEFAULT : OFF) If set to ON, build Qt4/5 User Interface and Visualization drivers. Requires: Qt4 or Qt5 and OpenGL libraries and headers. Qt5 will be searched for first. If it is not found Qt4 will be searched for. This behaviour can be adjusted through the advanced option GEANT4_FORCE_QT4.
• GEANT4_USE_OPENGL_X11 (DEFAULT : OFF, Unix Only)If set to ON, build the X11 OpenGL visualization driver.
• GEANT4_USE_RAYTRACER_X11 (DEFAULT : OFF, Unix only) If set to ON, build RayTracer visualization driver with X11 support.
• GEANT4_USE_INVENTOR (DEFAULT : OFF) If set to ON, build the OpenInventor visualization driver.
• GEANT4_USE_XM (DEFAULT : OFF, Unix Only) If set to ON, build Motif User Interface and Visualization drivers.
• GEANT4_USE_SYSTEM_CLHEP (DEFAULT : OFF) If set to ON, build Geant4 with an external install of CLHEP. You should not set this unless your usage of Geant4 mandates a specific external CLHEP installation (e.g. if your project’s software uses CLHEP in other tools and requires consistent use of the same CLHEP across the software stack).Requires: CLHEP libraries and headers. See the CLHEP_XXX and CMAKE_PREFIX_PATHAdvanced Options if CMake has trouble locating CLHEP.
• GEANT4_USE_SYSTEM_EXPAT (DEFAULT : ON) If set to ON, build Geant4 with an external install of Expat.
• GEANT4_USE_SYSTEM_ZLIB (DEFAULT : OFF) If set to ON, build Geant4 with an external install of zlib.

Advanced Options

• http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/installguide.html
• Among the advanced, I list only the one used in the easy build recipe file https://bitbucket.org/jairhul/accsoft-eb/src/5f285502686e28d656dfa4d3ce1408d2551ee5d1/Geant4-10.00.p04-GCC-4.9.3.eb?at=master&fileviewer=file-view-default
  • EXPAT_LIBRARY If CMake cannot locate your external EXPAT installation, set this to the full path to the EXPAT library, e.g. /usr/lib/libexpat.so
  • EXPAT_INCLUDE_DIR If CMake cannot locate your external EXPAT installation, set this to the directory containing the EXPAT headers (e.g. if you have /foobar/expat.h, then set this to /foobar).

EasyBuild

Reference

https://media.readthedocs.org/pdf/caylo-easybuild/docfix/caylo-easybuild.pdf EasyBuild is a flexible software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. Divert from the standard configure / make / make install with custom procedures. Allows for easily reproducing previous builds.

EasyBuild

EasyBuild consists of a collection of Python modules and packages that interact with each other, dynamically picking up additional Python modules as needed for building and installing a (stack of) software package(s) specified via simple specification files

In EasyBuild terminology: the EasyBuild framework leverages easyblocks to automatically build and install software using particular compiler toolchains, as specified by one or multiple easyconfig files.

The EasyBuild framework provides functionality commonly needed when installing scientific software on HPC systems. For example, it deals with downloading, unpacking and patching of sources, loading module files for dependencies, setting up the build environment, autonomously running (interactive) shell commands, creating module files that match the specification files, etc. The implementation of a particular software build and install procedure is done in a Python module, which is aptly referred to as an easyblock. For each software package being built, the EasyBuild framework will determine which easyblock should be used, based on the name of the software package or the value of the easyblock specification parameter (see Easyblock specification). Since EasyBuild v2.0, an easyblock must be specified in case no matching easyblock is found based on the software name (cfr. Automagic fallback to ConfigureMake). The specification files that are supplied to EasyBuild are referred to as easyconfig files (or simply easyconfigs .eb), which are basically plain text files containing (mostly) key-value assignments for build parameters supported by the framework, also referred to as easyconfig parameters (see Writing easyconfig files: the basics for more information). An easyconfig file serves as a build specification for EasyBuild. Easyconfigs typically follow a (fixed) strict naming scheme, i.e. <name>-<version>[-<toolchain>][<versionsuffix> A handful of easyconfig parameters are mandatory:

• name, version: specify what software (version) to build
• homepage, description: metadata (used for module help)
• toolchain: specifies name and version of compiler toolchain to use
• Using external modules as dependencies
• Configure/build/install command options:
  • configopts: options for configure command
  • preconfigopts: options used as prefix for configure command
  • buildopts, prebuildopts: options for build command
  • installopts, preinstallopts: options for install

EasyBuild allows to easily reproduce software builds/installations and install multiple versions.

Environment Modules

Reference

http://modules.sourceforge.net/man/module.html

Environment Modules

Typically users initialize their environment when they log in by setting environment information for every application they will reference during the session. The Environment Modules system is a tool to help users manage their Unix or Linux shell environment, by allowing groups of related environment-variable settings to be made or removed dynamically. The modules system is based on modulefiles,[6] which specify groups of environment settings that need to be made together. Modulefiles can be installed in a central location for general use, or in a user directory for personal use. Environment Modules modulefiles are written in the Tcl (Tool Command Language) and are interpreted by the modulecmd program via the module[7] user interface. The key advantage of Environment Modules is that it is shell independent and supports all major shells such as bash, ksh, zsh, sh, tcsh, and csh. The second key advantage is that it allows to use multiple versions of the program or package from the same account by just loading proper module. modulefiles are created on per application per version basis. They can be dynamically loaded, unloaded, or switched. Along with the capability of using multiple versions of the same software it also can be used to implement site policies regarding the access and use of applications. https://modules.readthedocs.io/en/latest/FAQ.html

New commands for centOS 7

1. Tell module tool where the modules are

module use /unix/pbt/software/eb/CentOS7/modules/*

1. Load Geant4
2. This loads all the necessary libraries like CLHEP, Qt5, etc.

module load Geant4/10.04.p02-GCC-4.9.3 source $EBROOTGEANT4/bin/geant4.sh module load CMake/3.5.2-GCC-4.9.3

what does this mean

• module use [-a|--append] directory…

Prepend one or more directories to the MODULEPATH environment variable.

• load modulefile...

Load modulefile(s) into the shell environment.

when Geant4 is installed as a module To see which versions are available, execute: module avail geant4 To then use a particular version, execute for example module load geant4/10.02.p03 After you load the module, you can see where the software is installed by executing echo $EBROOTGEANT4

Set GEANT environment variables source $EBROOTGEANT4/bin/geant4.sh