SCC
The operation of local HPC resources and especially of the Scientific Compute Cluster (SCC) at the GWDG is achieved by the transparent integration of different systems into a joint operating concept for the basic supply of the Max-Planck Institutes and the university. This includes a uniform software management, a shared batch management environment, cross-system monitoring and accounting, and cross-system file systems. Thus, synergies are achieved through the integration of different system generations and special-purpose systems (e.g. GPU clusters). Users will find a uniform environment on all HPC systems, while at the same time individual application environments are supported.
The SCC is distributed across the headquarter at Faßberg as well as in a modular data center:
This sketch shows the structure of the cluster and a schematic representation of the available resources. The spatial seperation has a direct influence on the use of the system, especially the use of storage. There are different compute nodes with different properties:
Host | # Nodes | CPU | Cores/CPU | Memory per Node [GB] | GPU | Partition |
---|---|---|---|---|---|---|
ampxxx | 96 | 2x Xeon Platinum 9242 | 48 | 384 | medium | |
dmpxxx | 82 | 2x Broadwell Xeon E5-2650 v4 | 12 | 128 | medium | |
Total | 178 | 11112 | 47360 | |||
dfaxxx | 15 | 2x Broadwell Xeon E5-2650 v4 | 12 | 512 | fat | |
dsuxxx | 5 | 4x Haswell Xeon E5-4620 v3 | 10 | 1536 | fat+ | |
gwdexxx | 1 | 4x Haswell Xeon E7-4809 v3 | 8 | 2048 | fat+ | |
Total | 21 | 592 | 17408 | |||
dgexxx | 7 | 2x Broadwell Xeon E5-2650 v4 | 12 | 128 | 2x 1080 | gpu |
dgexxx | 8 | 2x Broadwell Xeon E5-2650 v4 | 12 | 128 | 4x 980 | gpu |
dtexxx | 10 | 2x Broadwell Xeon E5-2650 v4 | 12 | 128 | 2x Tesla K40m | gpu |
agtxxx | 2 | 2x Xeon Gold 6252 | 24 | 384 | 8x Tesla V100/32 | gpu |
agqxxx | 14 | 2x Xeon Gold 6242 | 16 | 192 | 4x Quadro RTX5000 | gpu |
Total | 41 | 1144 | 6400 | 138 |
Storage Structure of the SCC
The SCC has various storage options. Each user has their own $HOME
directory which is accessible from anywhere within the SCC and has a backup. This is the general Unix/Linux file service operated by the GWDG. However, this file system is comparatively slow. The personal storage space can be increased upon request. To do so, please write an informal mail to support@gwdg.de with the request for a quota increase of your $HOME
directory.
The local storage of a compute node can be accessed with /local
. This consist of very fast SSD storage for the compute nodes located at Fassberg. For the compute nodes within the modular data center it resides in RAM. A directory /local/jobs/
is created automatically. This path is also exported as environment variable $TMP_LOCAL
. The local storage has an automatic file deletion feature to ensure a clean setup for every new batch job.
Via the frontend and compute nodes of the modular data center (amp, agq, agt, login-mdc.hpc.gwdg.de
) one has access to the shared filesystem /scratch
and via the compute nodes at the Faßberg (dfa, dge, dmp, dsu, dte, login-fas.hpc.gwdg.de
) one has access to the shared file system /scratch2
. The shared file systems /scratch
and /scratch2
are parallel BeeGFS file systems with high bandwidth. During program execution on the batch system, they are usually the best choice for temporary data. Neither of these file systems will automatically delete data or create a backup. To back up important results, the scc_backup
mechanism is available to back up the data at night to the home directory ${HOME}/scc_backup
. To do this, the data to be copied must be located in scratch/scratch2
in the folder /scratch(2)/${USER}/scc_backup
. To ensure access to /scratch
or /scratch2
you should use the -C scratch
or /scratch2
option in Slurm correspondingly. Since scratch/scratch2
is shared among all users, we strongly recommend that you store data that is not immediately needed elsewhere. You can use the archive, which can be reached with the environment variable $AHOME
. When using the archive you should always make sure to compress folders (as tar file) before mov
Transfer Data
In order to copy data to or from the $HOME
directory or to and from SCRATCH
/SCRATCH2
you can choose between several possibilities, depending on where you want to transfer data from.
The transfer node transfer-fas.hpc.gwdg.de
is available within the GÖNET. This node has access to the $HOME
directory as well as to /scratch2
.
The transfer node transfer-mdc.hpc.gwdg.de
is available within the GÖNET. This node has access to the $HOME
directory as well as to /scratch
.
The transfer node transfer.gwdg.de
is available worldwide, but only has access to the $HOME
directories. More information and command line examples can be found in the documentation.