Software/EasyBuild

From PBTWiki
Jump to navigation Jump to search

Instructions for using software with EasyBuild.

Compile BDSIM

James Chappell's question

Dear Laurie,

I’m trying to learn how to use the version of easy build that you setup to run some BDSIM simulations for AWAKE. I have a few questions for you.

1) The version of bdsim that you wrote the shell script for within /unix/pbt/software/eb/bdsim.sh does not have the AWAKE module compiled. Do you know if there is another version that I can link to which does for testing purposes for now?

2) Having looked through your eb files in /unix/pbt/software/eb/accsoft-eb-nevay, I can see that you define sources using a url and I’m assuming that in the process of compiling all the software, this source code is downloaded and then compiled. Is there a way to define the source from within the local system directories and then compile this version? Furthermore, to compile the AWAKE module, normally I use “ccmake .” to edit the compilation options - is the equivalent using the “configopts” command within an *.eb script? I’m assuming the command will be something like: “-USE_AWAKE=on”?

3) For our simulations, we edit the source code of the AWAKE module to make changes to the spectrometer and then recompile BDSIM before using it. How will this setup need to change when using easy build? Will it require recompiling the whole system of dependencies every time (I can see from your email to Simon previously that this took 5 hours)? Is it the case that we can compile everything up to BDSIM in one place (i.e. Geant4 and all its dependencies), and then just load our own version of BDSIM (or even just the AWAKE module) on top of this?

Sorry for all the questions, I’m just trying to get my head around how it all works! Thanks for your help, James.

The answer

Dear James,

1) The version of bdsim that you wrote the shell script for within /unix/pbt/software/eb/bdsim.sh does not have the AWAKE module compiled. Do you know if there is another version that I can link to which does for testing purposes for now?

My bad -> I'll recompile it with awake on. I need a time slot when you won't be using it as it'll disappear during the rebuilding - pretty bad if someone's running jobs when I do it! Let me know when I can do this - only takes about 15 mins.

Is there a way to define the source from within the local system directories and then compile this version? It looks like the system really only is designed to download things from what I can read online... Despite the url, it caches them in a sources directory, so if it was already there it would use that (a fudge though).

Furthermore, to compile the AWAKE module, normally I use “ccmake .” to edit the compilation options - is the equivalent using the “configopts” command within an *.eb script? I’m assuming the command will be something like: “-USE_AWAKE=on”?

We have so far in the Bdsim-1.1-GCC-4.9.4.eb easy build file:

configopts = ' -DCMAKE_CXX_FLAGS="-std=c++11"'

We can add to this:

configopts += '-DUSE_AWAKE=ON'

See ROOT-v6.08.06-GCC-4.9.3-Python-2.7.12.eb for lots of examples.

3) For our simulations, we edit the source code of the AWAKE module to make changes to the spectrometer and then recompile BDSIM before using it. How will this setup need to change when using easy build? Will it require recompiling the whole system of dependencies every time (I can see from your email to Simon previously that this took 5 hours)? Is it the case that we can compile everything up to BDSIM in one place (i.e. Geant4 and all its dependencies), and then just load our own version of BDSIM (or even just the AWAKE module) on top of this?

If you're modifying the C++ often I'd make your own build of BDSIM + AWAKE separately from the EasyBuild system and not bother trying to package it into a module. The module system is really for things that won't change.

You can get all the dependencies from EasyBuild and then make your own independent build of whatever you want. I think this was done for Gate for example and we do this just now for our nightly testing of BDSIM. I load the BDSIM module, then unload it - this leaves all the dependencies loaded.

module purge module load Bdsim module unload Bdsim module list

  1. still has all libraries loaded including Geant4 etc..
  2. somewhere else, make a build dir and build from your source.

When you load Geant4 and ROOT, note you should really source their setup scripts:

source $EBROOTGEANT4/bin/geant4.sh source $EBROOTROOT/bin/thisroot.sh

--->>> You should make a copy of the bdsim.sh script that does most of this and just add the unload to the end.

then...

mkdir build cd build cmake ../relative/path/to/bdsim -DCMAKE_CXX_COMPILER=/home/accsoft/SL68/software/GCC/4.9.3/bin/g++ 

-DCMAKE_C_COMPILER=/home/accsoft/SL68/software/GCC/4.9.3/bin/gcc
-DCLHEP_LIBRARY_DIR=/home/accsoft/SL68/software/CLHEP/2.3.1.1-GCC-4.9.3/lib/

make -j10

  • paths are example on my system at RHUL

Normally you wouldn't have to specify these options as CMake would find these on its own. However, practice shows it often finds a system compiler before it finds the one in easybuild (this is required to be the same for all the software to be consistent), so we explicitly specify them.

Note once a build is make and it finds a compiler, the compiler can never be changed in CMake. You must wipe the build directory contents to try again.

How do you use this new build? Simple, you must have the same environment loaded as when it was built (so hence the modified bdsim.sh) then execute from that directory. This could easily be done in your shell script for the simulation or via an alias (fictional paths here as an example)...

alias bdsim-awake-devel="/home/jchappell/my-builds/build123/bdsim" bdsim-awake-devel --help

Note if you haven't changed the output or the analysis you can just use the regular bdsim from easybuild if you do any analysis later on the output. If you have changed it, you'll need to use your build and modify your bdsim.sh - pretty clear where to change for the analysis stuff.

Ok, hopefully that should help you both get the build working and help you understand.

Cheers, Laurie

Additional info

Hello,

Ok, I've updated the general BDSIM build in easy build now. It should have the AWAKE module turned on. Note, this is the code of v1.1 and not the latest develop version.

I made an example environment only script in:

/unix/pbt/software/eb/bdsim-own-build.sh

I tried making my own build and it worked just fine. I only needed to do also:

module load CMake

to get a more up to date version. From my home dir:

mkdir build-custom-build cd build-custom-build cmake ../bdsim

  1. check compiler is GCC4.9.3 form easy build, and ROOT and Geant4 and CLHEP -> all ok

ccmake . turn on USE_AWAKE <c> <g> make -j15 works!

It's in /home/lnevay/bdsim-custom-build

Cheers, Laurie

Use modules

As Simon says, I've made modules ('environment modules for unix') on the system using EasyBuild. I've installed the latest Geant4 for BDSIM (our application) but I've also now put on the latest patch version of all recent Geant4 versions (was quite easy). You can access them from a CentOS7 node (I'm presuming Simon's here):

module use /unix/pbt/software/eb/CentOS7/modules/* module avail Geant4

.. this prints out the following 

------------------------------------------------ /unix/pbt/software/eb/CentOS7/modules/all -------------------------------------------------
Geant4/10.00.p04-GCC-4.9.3 Geant4/10.01.p03-GCC-4.9.3 Geant4/10.02.p03-GCC-4.9.3 Geant4/10.03.p03-GCC-4.9.3 Geant4/10.04.p02-GCC-4.9.3

module load Geant4/10.04.p02-GCC-4.9.3 source $EBROOTGEANT4/bin/geant4.sh

The last line sources the required environmental variable for Geant4 to be found. This will load the corresponding CLHEP version required (multiple available).

To see what's loaded:

module list

Currently Loaded Modulefiles: 

1) GCC/4.9.3                            9) bzip2/1.0.6-GCC-4.9.3               17) Mesa/11.2.1-GCC-4.9.3
2) expat/2.1.1-GCC-4.9.3               10) freetype/2.6.3-GCC-4.9.3            18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1
3) CLHEP/2.3.3.0-GCC-4.9.3             11) fontconfig/2.11.95-GCC-4.9.3        19) libffi/3.2.1-GCC-4.9.3
4) cURL/7.44.0-GCC-4.9.3               12) X11/20160819-GCC-4.9.3              20) gettext/0.19.6-GCC-4.9.3
5) Xerces-C++/3.1.2-GCC-4.9.3          13) libdrm/2.4.68-GCC-4.9.3             21) PCRE/8.38-GCC-4.9.3
6) zlib/1.2.8-GCC-4.9.3                14) ncurses/6.0-GCC-4.9.3               22) GLib/2.49.5-GCC-4.9.3
7) libxml2/2.9.3-GCC-4.9.3             15) LLVM/3.8.0-GCC-4.9.3                23) Qt5/5.7.0-GCC-4.9.3
8) libpng/1.6.21-GCC-4.9.3             16) eudev/3.1.5-GCC-4.9.3               24) Geant4/10.04.p02-GCC-4.9.3

To get rid of these loaded modules:

module purge

To see all modules:

module avail

There's the latest ROOT 6 on there too as well as a few versions of CLHEP (note Geant4 versions use different ones).

The module use and module load should be done in each new environment / terminal.

I tried making an easybuild recipe for GATE, but it fails at the end (I even tried a patch to remove their hard-coded make files for gjm) but still no success.


Install modules

Installing modules

Hello all,

- BDSIM has to use the single threaded version -> "high throughput" only vs "high performance" - BDSIM requires external CLHEP so Geant4's internal one isn't mixed in -> otherwise, whilst BDSIM would work, the random number generator found is unclear and the results are not strongly reproducible.

However, if you just want a multithreaded Geant4 build for your own application... yes you can make a separate easybuild file. In fact the full command I've used to construct a module is (for example):

eb Bdsim-1.2.1-GCC-4.9.3.eb --prefix=/unix/pbt/software/eb/CentOS7 -robot --robot-path=./: --modules-tool=EnvironmentModulesC --module-syntax=Tcl --allow-modules-tool-mismatch --optarch=GENERIC --buildpath=/tmp/easybuild/

The trick to get it to not replace a current version is to rename the module in the easybuild file. When you make one, easybuild checks for ones of the same name and version and if one with the same name and recipe is built it won't build it. If the options are updated, I believe it will update the build. Anyway, if you change the name it will create a new one. We used the name also as part of the source URL, so you should just hard code that. I've made an example here:

/unix/pbt/software/eb/accsoft-eb-nevay/Geant4-10.04.p02-mt-GCC-4.9.3.eb (copied from the regular geant4 one, turned multithreading on, changed version to have "-mt" and hardcoded source url)

When you build a module, you should do the module use command so it can find the others already available (it checks the requirements of the one you're building) and also the EasyBuild module itself. You should have a clean environment with no other modules loaded.

/unix/pbt/software/eb/easybuildonly.sh

seems to provide this.

I tried my example and found you should edit the version rather than the name for it to work. I've done this and it's setup now. You can import it as.

module load Geant4/10.04.p02-mt-GCC-4.9.3

Cheers, Laurie

PS we keep a note of the big incantation. You could get around most of these with some environmental variables, but we got with this so far.

PPS, remember you should source the geant4.sh from that installation before use. source $EBROOTGEANT4/bin/geant4.sh

PS we keep a note of the big incantation. You could get around most of these with some environmental variables, but we got with this so far.

PPS, remember you should source the geant4.sh from that installation before use. source $EBROOTGEANT4/bin/geant4.sh

The question

Dear Laurie,

I'm trying to compile a multi-threaded version of Geant4 using EasyBuild but would like to make sure I don't break everything.

Looking at the easyconfig file for the most recent Geant4 version installed on our system and the directory structure of the `eb/CentOS7/software` directory, I can see that the location of the installation depends on the name and the version of the package and on the toolchain used to compile it. This leads me to believe that if I were to set `GEANT4_BUILD_MULTITHREADED=ON` in the easyconfig file and run the following command,

eb Geant4-10.04.p02-GCC-4.9.3.eb --robot

it would overwrite the current (non-mutli-threaded) version of Geant4. I have two questions relating to this:

1) Am I correct in thinking it would overwrite it? 2) If so, would you mind showing me how I can compile it alongside the existing version?

The best solution I could come up with would be to install it in a different location using the command line option `--installpath-software=INSTALLPATH-SOFTWARE` but I'm unsure if that breaks compatibility with the modules tool and I'd also like to avoid overwriting the existing easyconfig file, which I don't know how to ensure.

Note on how to install Geant4 on Mac

Hello,

Ah we tried it briefly on mac, but it was a little frustrating so we stuck with our usual way, which is macports. For your mac, I'd recommend that. I know it's different than EasyBuild, but it's generally easier for mac and for scientific software.

https://www.macports.org

Before getting it you should xcode + the command line tools (from app store) installed (test is you can run 'make' on the terminal). You should also have XQuartz (https://www.xquartz.org) installed for X11 support. Then download and install the macports software (small, little mac installer). Gives you a command on the terminal 'port'. Everything is by default installed in /opt/local so you can nuke that directory if you don't want it anymore. It works by adding a line to your profile that adds that directory to your paths.

port search packagename

sudo port install py27-matplotlib

I'd install everything up to CLHEP from macports, then make my own geant4 installation. I think there is a geant4 port on there, but I haven't used it. For the geant4 installation you download hte source somewhere, make a build directory (outside the source) and then an install directory. Usual CMake pattern.

I'd start with a high level package in macports and it'll work out all the requirements. Like...

sudo port install clhep qt5 py27-matplotlib

I do everything apart from geant4 and bdsim through macports.

Hope that helps, Cheers, Laurie

Summary from Raffy about EasyBuild, Environment Modules and GEANT4

Geant4

Reference

http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/gettingstarted.html http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/installguide.html

GEANT4 OS/Software Prerequisites

the following source/software must be present to build Geant4:

  • Geant4 Toolkit Source Code.
  • C++ Compiler and Standard Library supporting the C++11 Standard
  • Linux (Scientific Linux CERN 6 and Linux CentOS 7 ): GNU Compiler Collection 4.8.5 or higher ( recommended to use the GCC compiler supplied by the package management system of your distribution unless this does not meet the minimum version requirement. The minimum required versions of GCC may be installed on SLC6 systems via the freeDeveloper Toolset)
  • CMake 3.3 or higher (as provided through the package management system of your distribution, unless it does not meet the minimum version requirement.)

The main requirement is that the system has a GCC of sufficient version to support C++11 installed.

User Interface and Visualization Drivers

  • Qt User Interface and Visualization
    • Qt4 (>=4.6) or Qt5 headers and libraries
  • OpenGL or MesaGL headers and libraries.
    • X11 OpenGL Visualization (Linux and macOS)
  • Open Inventor Visualization
  • Coin3D libraries and headers with SoXt(SoWin) graphics binding on Linux/macOS(Windows), compiled against the C++11 standard.
  • Motif User Interface and Visualization (Linux and macOS)

Standard GEANT4 Building and Installation

  • Unpack the Geant4 source package geant4.10.05.tar.gz to a location of your choice: source directory /path/to/geant4.10.05
  • create a directory in which to configure and run the build and store the build products /path/to/geant4.10.05-build
  • $ cmake -DCMAKE_INSTALL_PREFIX=/path/to/geant4.10.05-install /path/to/geant4.10.05
   * CMake Variable CMAKE_INSTALL_PREFIX is used to set the install directory, the directory under which the Geant4 libraries, headers and support files will be installed. It must be supplied as an absolute path. The second argument to CMake is the path to the source directory.
   * Additional arguments may be passed to CMake to activate optional components of Geant4, such as visualization drivers, or tune the build and install parameters.
   *  If you run CMake and decide afterwards you want to activate additional options, simply rerun CMake in the build directory, passing it the extra options plus the build directory path.
  • Datasets are not required to be present to build Geant4, but may be required to run your application, depending on the physics models you use. If you wish to download and install the datasets automatically as part of your build of Geant4, simply add the option-DGEANT4_INSTALL_DATA=ON to the arguments passed to CMake.
  • After the configuration has run, CMake will have generated Unix Makefiles for building Geant4. To run the build, simply execute make in the build directory
  • Once the build has completed, you can install Geant4 to the directory you specified earlier in CMAKE_INSTALL_PREFIX
  • additional options can be passed to CMake to adjust the way Geant4 is built and installed and to enable optional components. Some options enable components of Geant4 which require external software.If these options are enabled, the required software will be searched for, and hence there are also options which control where CMake should look for these packages.These options may be set by passing their name and value to the cmake command via -D flags.

Standard Options

  • CMAKE_INSTALL_PREFIX installation prefix for Geant4. (default Unix: /usr/local)
  • CMAKE_BUILD_TYPE : (DEFAULT : Release) Sets the type of build. It defaults to Release which gives a fully optimized build without debugging symbols.
  • GEANT4_BUILD_MULTITHREADED : (DEFAULT : OFF) If set to ON, build Geant4 libraries with support for multithreading. At present, this is only supported on Unix systems. (Requires: Compiler/C++ Standard Library with support for C++ Threading Support)
  • GEANT4_INSTALL_DATA : (DEFAULT : OFF) to ON, download and install any Geant4 datasets missing from GEANT4_INSTALL_DATADIR.
  • GEANT4_INSTALL_DATADIR : (DEFAULT : CMAKE_INSTALL_DATAROOTDIR) Installation directory for Geant4 datasets.
  • GEANT4_USE_G3TOG4 : (DEFAULT : OFF) If set to ON, build the G3ToG4 library for reading ASCII call list files generated from Geant3 geometries.
  • GEANT4_USE_GDML : (DEFAULT : OFF) If set to ON, build the G4persistency library with support for GDML.
  • GEANT4_USE_QT (DEFAULT : OFF) If set to ON, build Qt4/5 User Interface and Visualization drivers. Requires: Qt4 or Qt5 and OpenGL libraries and headers. Qt5 will be searched for first. If it is not found Qt4 will be searched for. This behaviour can be adjusted through the advanced option GEANT4_FORCE_QT4.
  • GEANT4_USE_OPENGL_X11 (DEFAULT : OFF, Unix Only)If set to ON, build the X11 OpenGL visualization driver.
  • GEANT4_USE_RAYTRACER_X11 (DEFAULT : OFF, Unix only) If set to ON, build RayTracer visualization driver with X11 support.
  • GEANT4_USE_INVENTOR (DEFAULT : OFF) If set to ON, build the OpenInventor visualization driver.
  • GEANT4_USE_XM (DEFAULT : OFF, Unix Only) If set to ON, build Motif User Interface and Visualization drivers.
  • GEANT4_USE_SYSTEM_CLHEP (DEFAULT : OFF) If set to ON, build Geant4 with an external install of CLHEP. You should not set this unless your usage of Geant4 mandates a specific external CLHEP installation (e.g. if your project’s software uses CLHEP in other tools and requires consistent use of the same CLHEP across the software stack).Requires: CLHEP libraries and headers. See the CLHEP_XXX and CMAKE_PREFIX_PATHAdvanced Options if CMake has trouble locating CLHEP.
  • GEANT4_USE_SYSTEM_EXPAT (DEFAULT : ON) If set to ON, build Geant4 with an external install of Expat.
  • GEANT4_USE_SYSTEM_ZLIB (DEFAULT : OFF) If set to ON, build Geant4 with an external install of zlib.


Advanced Options

Retrieved from "https://www.hep.ucl.ac.uk/pbt/pbtWiki/index.php?title=Software/EasyBuild&oldid=2283"