Compute Details
As explained in the Overview, the compute partition is divided into 3 different categories : CPU nodes, GPU nodes, and specialized nodes.
CPU Nodes
The CPU nodes are housed in 75 HPE Apollo n2600 chassis, each supporting 4x HPE ProLiant XL225n compute nodes, offering:
- Total Performance: 1137 TFLOPS Rmax (LINPACK) / 1505 TFLOPS Rpeak (theoretical).
- Total CPU Cores: 38,400 cores.
| Node Details | Standard | Medium | Common Characteristics |
|---|---|---|---|
| Num. of Nodes | 270 | 30 | |
| Node Model | → | → | HPE XL225n |
| Processors | → | → | 2x AMD EPYC 7763 64-core |
| Processor Frequency | → | → | 2.45GHz (boost up to 3.5GHz) |
| Processor L3 Cache | → | → | 256MB |
| Cores per Node | → | → | 128 |
| Hyperthreading | → | → | Disabled |
| Memory | 256GB | 512GB | DDR4-3200 |
| User-Available Memory | 240GB | 492GB | |
| Ethernet | → | → | 2x 10Gbps |
| Fast Interconnect | → | → | 1x Infiniband HDR-100 |
| Local Disk | → | → | SATA SSD 480GB (system) |
| Node Hostnames | cns[001-270] |
cnm[001-030] |
Tip
For applications requiring more memory, consider using Large Memory Nodes.
GPU Nodes
The GPU nodes are built with HPE Apollo d6500 chassis, each supporting 2x HPE ProLiant XL645d nodes, with 4x Nvidia A100 GPUs per node. They provide:
- Total Performance: 2717 TFLOPS Rmax (LINPACK) / 3900 TFLOPS Rpeak (theoretical).
- Total GPUs: 200 Nvidia A100.
- TOP500 Ranking: Ranked 245th on the Nov. 2022 TOP500 List.
| Node Details | GPU Nodes |
|---|---|
| Num. of Nodes | 50 |
| Node Model | HPE XL645d |
| Processors | 1x AMD EPYC 7513 32-core |
| Processor Frequency | 2.6GHz (boost up to 3.65GHz) |
| Processor L3 Cache | 128MB |
| Cores per Node | 32 |
| Hyperthreading | Disabled |
| Memory | 256GB DDR4-3200 |
| User-Available Memory | 240GB |
| Accelerators | 4x Nvidia A100 40GB |
| Ethernet | 2x 10Gbps |
| Fast Interconnect | 2x Infiniband HDR-200 |
| Local Disk | SATA SSD 480GB (system) |
| Node Hostnames | cna[001-050] |
Tip
For AI workloads, consider using AI Nodes, which feature twice the GPU capacity.
Specialized Nodes
This partition contains nodes tailored for specific high-memory, visualization, and AI workloads.
Large Memory Nodes
These nodes are optimized for memory-intensive applications, offering up to 4TB of memory.
| Node Details | Large Memory | Extra Large Memory | Common Characteristics |
|---|---|---|---|
| Num. of Nodes | 7 | 1 | |
| Node Model | → | → | HPE ProLiant DL385 |
| Processors | → | → | 2x AMD EPYC 7513 32-core |
| Processor Frequency | → | → | 2.6GHz (boost up to 3.65GHz) |
| Processor L3 Cache | → | → | 128MB |
| Cores per Node | → | → | 64 |
| Hyperthreading | → | → | Disabled |
| Memory | 2048GB | 4096GB | DDR4-3200 |
| User-Available Memory | 2000GB | 4000GB | |
| Accelerators | → | → | No |
| Graphics Card | → | → | No |
| Ethernet | → | → | 2x 10Gbps |
| Fast Interconnect | → | → | 1x Infiniband HDR-100 |
| Local Disk | → | → | SATA SSD 480GB (system) |
| Node Hostnames | cnl[001-007] |
cnx001 |
CPU nodes vs Large Memory nodes comparison
Large Memory Nodes provide significantly more memory --4 times more for large memory nodes and 8 times more for extra large memory nodes-- compared to medium CPU nodes. Note that they have fewer cores (half the number of CPU nodes) but retain the same cache per core.
AI Nodes
Optimized for deep learning and machine learning workloads, featuring Nvidia A100 GPUs with a total of 8 GPUs per node.
| Node Details | AI |
|---|---|
| Node Model | HPE XL645d |
| Num. of Nodes | 2 |
| Processors | 1x AMD EPYC 7513 32-core |
| Processor Frequency | 2.6GHz (boost up to 3.65GHz) |
| Processor L3 Cache | 128MB |
| Cores per Node | 32 |
| Hyperthreading | Disabled |
| Memory | 2048GB DDR4-3200 |
| User-Available Memory | 2000GB |
| Accelerators | 8x Nvidia A100 SXM4 80GB |
| Graphics Card | No |
| Ethernet | 2x 10Gbps |
| Fast Interconnect | 2x Infiniband HDR-200 |
| Local Disk | 2x SATA SSD 480GB (system) + 1x NVMe SSD 6.4TB (local scratch) |
| Node Hostnames | cni[001-002] |
GPU nodes and AI nodes comparison
While both types feature the same processor, AI nodes offer substantial upgrades in several areas. They have 8 times more memory than GPU nodes, double the number of A100 GPUs, and the GPUs in AI nodes have twice the memory capacity of those in GPU nodes. Additionally, AI nodes come with a dedicated SSD for local scratch storage, which is not available in GPU nodes.
Visualization Nodes
Equipped with Nvidia T4 GPUs for rendering and visualization tasks.
| Node Details | Visualization |
|---|---|
| Node Model | HPE ProLiant DL385 |
| Num. of Nodes | 4 |
| Processors | 2x AMD EPYC 7313 16-core |
| Processor Frequency | 3GHz (boost up to 3.7GHz) |
| Processor L3 Cache | 128MB |
| Cores per Node | 32 |
| Hyperthreading | Disabled |
| Memory | 512GB DDR4-3200 |
| User-Available Memory | 492GB |
| Accelerators | No |
| Graphics Card | 4x Nvidia T4 16GB |
| Ethernet | 2x 10Gbps |
| Fast Interconnect | 1x Infiniband HDR-100 |
| Local Disk | 2x SATA SSD 480GB (system) |
| Node Hostnames | cng[001-004] |