Wilson HPC Computing Facility

Hardware Information

The Accelerator Simulations cluster consists of the following main components:

  1. Head Node
  2. Worker Nodes
  3. File Server
  4. Network Switches

NOTE: All node hostnames appear in brackets, bold italics and green (example)

Batch Submission Information


Head node: use SSH to tev.fnal.gov
All CPU hosts uses the SLURM workload manager. Please refer to the SLURM how-to page here.


Hardware Information

1. Head Node: tev.fnal.gov

The cluster head node hardware is configured as follows:

Component Type
CPU AMD dual-socket four-core (total 8 cores) 64-bit Opteron 6134 2.3 GHz
Memory 32GB DDR3 1333MHz ECC
Motherboard Supermicro H8SGL 
Network Three Gigabit Ethernet and one Double Data Rate (20 Gbps) Infiniband interface

The cluster head node has the following network interfaces:

Component Network Description
eth1 service interfaces with eth1 on all worker nodes (tevp)
eth1:0 ipmi interfaces with eth0 on all worker nodes
eth0 public (tev.fnal.gov)
ib0 infiniband interfaces with ib0 on all worker nodes (ibtev)

The cluster head node consists of a (mirrored) 250GB SATA disk formatted as follows:

Partition Size Description
sda1 100 MB DOS
sda2 965 MB boot
sda3 95 GB root
sda4 4 GB swap
sda6 122 GB user home area (NFS exported to all worker nodes)

2. Worker Nodes


There are two types of worker nodes on the Wilson cluster, "intel12" and "amd32".

intel12:

Component Type
CPU Intel dual-socket six-core (total 12 cores) Xeon 64-bit X5650 "Westmere" 2.67 GHz
Chipset Intel X5520 "Tylersburg"
Memory 12GB DDR3 1333MHz ECC
Motherboard Supermicro X8DT3
Network Two Gigabit Ethernet and one Single Data Rate (10 Gbps) Infiniband interface

These nodes are named tev0101 through tev0213. The network interfaces are configures as follows:

Component Network Description
eth0 ipmi access to worker node BMC (ipmitevXXYY)
eth1 service tevXXYY
ib0 infiniband NFS mounts from this network (ibtevXXYY)

The intel12 nodes consist of a single 250GB SATA disk formatted as follows:

Partition Size Description
sda1 100 MB DOS
sda2 1 GB boot
sda3 14 GB root
sda5 4 GB swap
sda6 212 GB scratch space

The following graphic provides an abstraction of the hierarchical topology of the CPUs installed in the intel12 worker nodes. The graphic was generated using The Portable Hardware Locality (hwloc) software package. The graphic lists NUMA memory nodes, sockets, shared caches and cores.

amd32:

Component Type
CPU AMD 6128 HE Opteron 2GHz quad-socket eight-core (32 cores total)
Memory 64GB DDR2 400 SDRAM
Motherboard Supermicro H8QGL
Network Two Gigabit Ethernet and one Double Data Rate (20 Gbps) Infiniband interface

These nodes are named tev0301 through tev0510. There are 12 nodes each in racks 3 and 4, and 10 nodes in rack 5. The Ethernet interfaces are configures as follows:

Component Network Description
eth0 ipmi access to worker node BMC (ipmitev0301)
eth1 service (tev0301)
ib0 infiniband NFS mounts from this network (ibtev0301)

The amd32 nodes consist of a single 2 TB SATA disk formatted as follows:

Partition Size Description
sda1 100 MB DOS
sda2 1.9 GB boot
sda3 97 GB root
sda5 4 GB swap
sda6 1.7 TB scratch

The following graphic provides an abstraction of the hierarchical topology of the CPUs installed in the amd32 worker nodes. The graphic was generated using The Portable Hardware Locality (hwloc) software package. The graphic lists NUMA memory nodes, sockets, shared caches and cores.


The intel12 and amd32 worker nodes NFS mount disks from the head node (tev) and the file server (tevnfs) using ip over ib on the Infiniband network as follows:

Mount Description Data Type
/usr/local NFS mounted from head node (ibtev) Common user applications, compilers and system tools.
Backup: YES Daily incremental backups to TiBs.
/home NFS mounted from head node (ibtev) User home area with 6GB quota limit per user.
Backup: YES Daily incremental backups to TiBs.
/data NFS mounted from file server (ibtevnfs) User application data area with 30GB quota limit per user.
Backup: NO
/fast NFS mounted from file server (ibtev0213) User application high throughput scratch space area with 30GB quota limit per project.
Backup: NO
/fnal/ups NFS mounted from head node (ibtev) Shared Fermilab UPS products area
Backup: YES Daily incremental backups to TiBs.

3.  File Server

The cluster file server tevnfs hardware configuration is as follows:

Component Type
CPU AMD dual-socket four-core (total 8 cores) 64-bit Opteron 6134 2.3 GHz
Memory 32GB DDR3 1333MHz ECC
Motherboard Supermicro H8SGL 
Network Three Gigabit Ethernet and one Double Data Rate (20 Gbps) Infiniband interface

The Ethernet interfaces on this node are configured as follows:

Component Network Description
eth0 public
 (tevnfs.fnal.gov)
eth1 service   private network (tevnfsp)
ib0 infiniband NFS exports from this network (ibtevnfs)

4.  Network Switches

The cluster consists of Ethernet and Infiniband switches. There are two Netgear FSM 700S 48-port Fast Ethernet (ports 49 and 50 are Gigabit Ethernet) switches connected in a master-slave mode.

There are six 24-port Mellanox Flextronics Infiniband switches with one leaf switch per rack (total 5 switches) and one spine switch. The two older racks have Single Data Rate (10Gbps) leaf switches. The three newer racks have Double Data Rate (20Gbps) leaf switches. The single spine switch supports Double Data Rate (20Gbps). All of these switches are unmanaged. The Infinband subnet manager runs on the NFS file server (tevnfs).


Contact: Amitoj Singh
Last modified: Nov 6, 2017