Raven user guide - quick start
Raven is the new University Supercomputer service, replacing the original Merlin system (NOTE: all user accounts should have been migrated onto the new service - if you have trouble accessing Raven please contact arcca: arcca<at>cardiff.ac.uk).
The Linux cluster consists of 2048 cores Intel Sandy Bridge processors (2.6GHz / 4GB per core / 8 cores per processor) as the main parallel MPI partition (including a SMP section), with an additional 864 cores Intel Westmere (2.8GHz / 3GB per core / 6 cores per processor) as a serial/high throughput subsystem. Raven is configured with 8+TB of total memory across the entire cluster, with a 50 TB global parallel file storage managed by the Lustre file system and 100TB NFS /home partition for longer term data store. Nodes are connected with InfiniBand QDR technology (40Gbps / 1.2μsec latency).
Quick start guide to using Raven
Below is a very quick guide for those experienced users who just want to understand the basics to get started on the cluster. If you do experience any problems using the cluster, please refer to the more detailed User Guide. If this does not resolve the issue, then please inform the team (arcca<at>cardiff.ac.uk) who will respond to your query.
A powerpoint presentation, including worked examples is also available:
This guide covers the following topics:
- Logging on to Raven
- Setting environment variables
- Compilers & mpi wrappers
- Job Scheduler: PBS Pro (simple script & job submission)
1. Logging on to Raven
From a linux environment:
ssh -X <username> ravenlogin.arcca.cf.ac.uk
Enter password when prompted.
The system is configured to use the University's LDAP authentication method - i.e. your standard university login credentials will form your username/password combination.
From a windows environment:
You will need to install Xming and putty on your PC in order to access Raven (available on the Networked Applications under the Departmental Software → ARCCA directory). Double click on the Xming icon prior to launching putty (Xming only needs to be started once during any session). In the putty window complete the following:
hostname window type ravenlogin.arcca.cf.ac.uk
Port window 22 (should be the default setting)
Under the windows category select the colours folder, select the "Use systems colour" box (normally unchecked) - this will provide a window with black text on a white background (default is white text on a black background).
Under the SSH category, X11 section, select the "Enable X11 forwarding" (ensure can bring back additional windows from your terminal window).
The default log in shell is bash. Your .bashrc file will contain the default settings - please do not modify this file. For user defined variables/commands please use the .myenv file which is also present in your home directory.
A modular environment is now available on the cluster - by default this will use the Intel MPI / Intel compiler combination. If you wish to change this to use the Bullxmpi environment you will have to unload the Intel MPI environment prior to adding the bullxmpi.
module list this command will list the modules currently installed on Raven.
module avail will list the modules loaded in your session
module unload <module-name> will remove a specific module from your current session.
module load <module-name> this command will load a new module into your account.
Unless these module commands are saved in your .myenv file, then any changes will only be "live" during your current log-in session and will revert to the default settings the next time you log onto the cluster.
Example to use Intel MPI / Intel compiler environment:
1. module load intel/compiler
2. module unload bullxmpi
3. module load intel/mpi
Once additional software (e.g. Portland Group compilers) is available on the cluster, users will be able to use modules to make this software available in their environment.
NOTE: Intel and gnu compilers are available on the cluster. The Portland Group Compilers will be available on the cluster soon.
Raven uses the Intel® C++ and Fortran compilers as default.
|icc||C||.c||icc [compiler_options] prog.c|
|icc||C++||.C .cc .cpp .cxx||icc [compiler_options] prog.cpp|
|ifort||F77||.f .for .ftn||ifort [compiler_options] prog.f|
|ifort||F90||.f90 .fpp||ifort [compiler_options] prog.f90|
Example C Program.
Create a simple program, for example:
Save as hello.c. and compile:
Then run: ./a.out
More compiler options and information can be found using either icc -help or man icc
Example Fortran (to follow):
Compiling with MPI wrappers
The mpiicc, mpiiCC, mpiifort compiler scripts (wrappers) compile MPI code and automatically link start up and message passing libraries to the executable.To determine which libraries are automatically included using the mpi wrapper scripts:
|mpicc||C||.c||mpicc [compiler_options] prog.c|
|mpiCC||C++||.C .cc .cpp .cxx||mpiCC [compiler_options] prog.cpp|
|mpif77||F77||.f .for .ftn||mpif77 [compiler_options] prog.f|
|mpif90||F90||.f90 .fpp||mpif90 [compiler_options] prog.f90|
mpif90 -show or mpicc -show
|mpiicc -cc=icc||C||.c||mpicc [compiler_options] prog.c|
|mpiiCC||C++||.C .cc .cpp .cxx||mpiCC [compiler_options] prog.cpp|
|mpiifort||F77||.f .for .ftn||mpif77 [compiler_options] prog.f|
|mpiifort||F90||.f90 .fpp||mpif90 [compiler_options] prog.f90|
mpiifort-show or mpiicc -cc=icc -show
When the Portland Group compilers are available on the system, for the MPI implementations use -f90=pgf90 or -cc=pgcc to enable the Portland compilers to be used rather than the default Intel Compilers.
Additional options for these wrappers can be found using the man pages.
The MPI compiler wrappers use the same compilers as for serial compilation (icc, icpc, ifort) and therefore any of the compiler flags used here can be used with the mpi wrappers. Common options include:
|-O3||Performs some compilation time & memory intensive optimizations in addition to those executed with "-O2". NOTE: it may not improve the performance of all codes|
|-xT||Flag for the Xeon chipset - includes specialized code for the SSE4 instruction set.|
|-ipo||Inter Procedural Optimization|
|-g||Debugging information produced during compilation|
All jobs run on Raven must be submitted via the batch scheduler, PBS Pro.
A working job submission script takes the following form:
#PBS -l select=2:ncpus=16:mpiprocs=16
#PBS -l place=scatter:excl
#PBS -o <output-file.txt>
#PBS -e <error-file.txt>
mpirun -np 32 <code> <inputfiles>
Where the line "-l select=2:ncpus=16:mpiprocs=16" is the number of processors required for the mpi job. "select" specifies the number of nodes required; "ncpus" indicates the number of CPUs per node required; and "mpiprocs" represents the number of mpi processes to run per node (normally ncpus=mpiprocs).
As this is not the most intuitive command, the following table is provided as guidance as to how this command works:
|2||16||16||32 Processor job, using 2 nodes and 16 processors per node|
|4||8||8||32 Processor job, using 4 nodes and 8 processors per node|
|16||1||1||16 Processor job, using 16 nodes running 1 mpi process per processor and utilising 1 processor per node|
|8||16||16||128 Processor job, using 8 nodes and 16 processors per node (each running an mpi process)|
"qstat" command displays the status of the PBS scheduler and queues. Using the flags "-Qa" shows the queue partitions available. By default, if no queue is defined, it will use the workq.
To select a different queue use:
#PBS -q queue_512
Once your job script is prepared, it can be submitted to PBS using the command "qsub":
To check on progress use the "qstat" command.
If for some reason the job is not running as expected and it needs to be cancelled, use the command qdel together with the job number.
For further information on the system, please refer to the User Guide.