Our supercomputer

Raven is our High-Performance Computing (HPC) cluster that can be accessed from anywhere with an internet connection.

Our Linux cluster consists of 2048 cores Intel Sandy Bridge processors (2.6GHz / 4GB per core / 8 cores per processor) as the main parallel MPI partition (including a SMP section), with an additional 864 cores Intel Westmere (2.8GHz / 3GB per core / 6 cores per processor) as a serial/high throughput subsystem.

Raven is configured with 8+TB of total memory across the entire cluster, with a 50 TB global parallel file storage managed by the Lustre file system and 100TB NFS /home partition for longer term data store. Nodes are connected with InfiniBand QDR technology (40Gbps / 1.2╬╝sec latency).

System details

Core MPI Partition

128 compute blades are Bullx B510 which accommodate two servers in a blade. Each compute blade has 2 sockets, containing:

  • Intel Xeon E5-2670 (Sandy Bridge) 2.60GHz processors
  • 8 cores/socket (2,60GHz, 8,00GT/s, Turbo+, 115W) giving 16 cores per node
  • 4GB RAM per core (DDR3-1600Mhz ECC SDR)
  • 128GB SATA2 Flash SSD disk
  • Single port Connect2-X 4x QDR PCIe Gen2-x8 Infiniband Interface.

Serial / High Throughput Partition

72 dual-socket compute blades using Bullx500, containing

  • 864 Intel Xeon (Westmere / X5660) 2.80GHz cores (12MB / 6.4GT/s)
  • 4GB RAM per core (48GB memory 1333MHz)
  • 128GB SATA SSD disk
  • Single Infiniband 4x QDR/PCIe Gen2-x8 interface embedded in the motherboard.

8 SMP compute nodes

8 SMP compute nodes (64 core subset of the MPI Nodes) are Bull B510 blade servers.

  • Intel Xeon E5-2670 (Sandy Bridge) 2.60GHz processors
  • 8 cores/socket (2,60GHz, 8,00GT/s, Turbo+, 115W) giving 16 cores per node
  • 8GB RAM per core (DDR3-1600Mhz ECC SDR)
  • 128GB SATA2 Flash SSD disk
  • Single port Connect2-X 4x QDR PCIe Gen2-x8 Infiniband Interface.

The High Speed, Low Latency (HS,LL), high performance interconnect is provided by an InfiniBand 8x QDR network using Mellanox MIS5030Q InfiniScale IV 36-Port QSFP 40Gb/s non-blocking switches.

The full non-blocking topology enables collision-less switching of both MPI and I/O traffic. A Connectx-2 single port 4x QDR PCIe gen2-x8 HCA (host card adaptor) is provided in each compute node.

In addition the management network is provided by Gigabit fabric using Cisco Catalyst 48 Port switches with 4 SFP prots.

Key features of the network:

  • Links: 4x QDR InfiniBand (40Gbps)
  • Ports: 36 per switch
  • Switching performance: 3.2GB/s per port
  • Switching capacity: 2.88TB/s
  • Latency: 100ns port-to-port

There are two main storage sub-systems:

A fast 50TB cluster file system based on Cluster File System (CFS) including Lustre software (RAID-6), and a redundant Network File System (NFS) of 100TB useable RAID-6 disk.

Cluster File System

Cluster File System is the scalable Parallel File System, Lustre from CFS and utilises two cross connected OSS arrays containing 60x1TB 3.5" 6Gb SAS 3.5" PI, FDE disks.

This provides high performance, fault tolerant, IO servers for data storage and meta-data storage. Bulls' PFS (Parallel File System) solution is designed and balanced so that the sustained bandwidth is very close to the theoretical peak. Sustained write performance of 6.2GB/s and Sustained read of 7.8GB/s.

The Lustre File System scales linearly when more OSS (Object Storage Servers) and more storage is added to the cluster.  The OSS arrays each contain 60x1TB 3.5" 6Gb SAS 3.5" PI FDE disks. Each IO cell (1xNetAPP and 2xOSS) provides 3.2GB/s write and 4GB/s read performance.

Network File System

The NFS Storage is a no-single-point-of-failure system running NFS over IB.

The array has dual controllers which are cross connected to two servers. The disks will be configured in RAID6 for protection against up to 2 disks concurrently failing.

The solution will commit to 450MB/s on sustained write and 600MB/s for sustained read performance.

  • Operating System: Bullxlinux 6.0 (based on RHEL6)
  • Job Scheduler: PBS Pro & PBS Analytics
  • Cluster Management Tools: Bull MCM (Monitoring and Control Management tools)
  • Cluster File System: Lustre
  • Compilers: Intel® Cluster Studio (Floating Academic 5 Seat Pack (ESD)) Intel C / C++ / Fortran
  • Libraries:
    • Intel Math Kernel Library - Cluster Edition Medium Cluster License for Linux
    • FFTW
    • HDF5
    • netCDF
    • gsl
  • Analysers, Profilers and Debuggers:
    • Intel® VTune™ Performance Analyzer for Linux - Floating Academic 1 Seat Pack (ESD)
    • Intel® Trace Analyzer & Collector (ITA / ITC), Large Cluster System License, Single Cluster, unlimited Developers, Academic
    • Bullx Supercomputer MCM (Monitoring and Control Management Tools)
    • Allinea: DDT (Distributed Debugging Toolkit) - 64 processor license
    • Allinea OPT (Optimization toolkit) - 64 processor license

Schematics

Name Type Last updated
Interconnect network schematic PDF 14/09/2016
Raven system schematic PDF 14/09/2016