Skip to content

GH200 Early Access

The GH200 nodes are now available for early access testing. They represent a significant architectural departure from the rest of the BMRC cluster — this page explains what is different, what that means for your software, and how to get started.

What makes the GH200 different

Each GH200 node contains two NVIDIA GH200 Grace Hopper Superchips. Unlike every other GPU node on the cluster, the GH200 is not simply a GPU card plugged into an x86 server — the CPU and GPU are a single unified superchip connected by NVLink-C2C, a high-speed coherent interconnect that lets both processors share memory transparently.

Feature Standard cluster nodes GH200 nodes
CPU architecture x86_64 (Intel/AMD) aarch64 (Arm Neoverse V2)
Operating system Rocky Linux 8 Rocky Linux 9
GPU A100 / other NVIDIA GH200 144GB
GPU memory 40–80 GB HBM2e 144 GB HBM3e per GPU
CPU–GPU link PCIe (~64 GB/s) NVLink-C2C (900 GB/s)
GPUs per node varies 2 per node
CPU memory DDR4/DDR5 ~480 GB LPDDR5X per superchip

The key figures for GPU-intensive work: 144 GB of HBM3e per GPU and 288 GB total per node, with ~4 TB/s memory bandwidth per GPU.

Why your existing software will not work

There are two independent reasons why software built for the rest of the cluster will fail on the GH200 nodes.

1. CPU architecture mismatch

The Grace CPU uses the Arm (aarch64) instruction set. Every binary compiled for x86_64 — Python, R, compiled extensions, conda environments, container layers — is the wrong architecture and will not execute. When you try to run an x86_64 binary on an aarch64 node you will typically see one of these errors:

Exec format error

/bin/conda: line 6: syntax error near unexpected token `sys.argv'

The second error, which looks like a Python syntax error in a shell script, is actually the shell trying to interpret an x86_64 binary as a shell script because it cannot execute it natively.

This affects everything

Conda environments, pip-installed packages with compiled extensions, pre-built binaries in your home directory, and any software loaded via module load from the standard module tree are all x86_64 and will not work on the GH200 nodes.

2. Operating system mismatch

The GH200 nodes run Rocky Linux 9 (EL9). The rest of the cluster runs Rocky Linux 8 (EL8). Packages and modules built against EL8 system libraries are not guaranteed to work on EL9 and vice versa. The aarch64 module tree described below is built specifically against EL9.

Software options

We have started building an aarch64/EL9 EasyBuild software stack at /apps/eb/el9/. The available toolchain generations with aarch64 builds are:

Toolchain Path
2023a /apps/eb/el9/2023a/aarch64/
2024a /apps/eb/el9/2024a/aarch64/
2025a /apps/eb/el9/2025a/aarch64/

This tree is not loaded by default. Add it to your module path with:

module use /apps/eb/el9/2025a/aarch64/modules/all

Then use module avail and module load as normal. We recommend adding this to your SLURM scripts rather than your ~/.bashrc to avoid affecting jobs on other partitions.

Coverage is growing

The aarch64 stack is under active development. If a module you need is missing, contact KIR Research Computing — we can prioritise building it.

Option B — uv for Python environments

For Python work, uv is the most practical option during the early access period. An aarch64-native build of uv is available at:

export PATH=/apps/kir/eb/hpc-utils/aarch64:$PATH

Add this to the top of your SLURM scripts before any Python or pip commands. This ensures the aarch64 uv binary is used rather than any x86_64 version that may be on your PATH.

Create a virtual environment and install packages as normal:

export PATH=/apps/kir/eb/hpc-utils/aarch64:$PATH

uv venv .venv --python 3.11
source .venv/bin/activate
uv pip install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu126
uv pip install transformers accelerate

CUDA module

Always load a CUDA module before activating your environment. Use CUDA/12.6.0 for GH200 — it is the minimum version with full support for the GH200's compute architecture (see below).

module purge
module load CUDA/12.6.0
source .venv/bin/activate

Option C — contact KIR Research Computing

During the early access period, KIR Research Computing can build and maintain a shared environment for your group in a shared path, saving you the effort of managing your own aarch64 environment. This is the recommended route for groups who want to get started quickly.

Requesting GH200 resources

Use the following partition in your SLURM scripts:

#SBATCH --partition  gpu_gh200_144gb

A few important points before you submit:

Start with one GPU

Begin with a single GPU while you are getting familiar with the nodes. Only request both if your workflow genuinely requires it.

#SBATCH --gres  gpu:1   # single GPU (recommended to start)
#SBATCH --gres  gpu:2   # both GPUs (only when needed)

Mandatory Slurm directives

#SBATCH --account       
#SBATCH --cpus-per-task
#SBATCH --partition
#SBATCH --gres           gpu:1
#SBATCH --time

Always set --cpus-per-task If you do not specify CPU cores, SLURM will allocate all 72 Grace CPU cores on the superchip to your job by default. This blocks other users from running even if they need only a handful of cores. Always be explicit:

#SBATCH --cpus-per-task  8   # adjust to what your job actually needs

Always set --time

The default time limit is 5 days. Please always set an explicit wall time — it helps the scheduler and frees the node promptly when your job finishes. During early access we recommend keeping jobs short while you are iterating on your workflow.

#SBATCH --time  01:00:00   # HH:MM:SS

Be mindful with --mem

Each node has approximately 1 TB of CPU memory. You can request up to that, but please only do so if your workflow genuinely requires it. Idle over-allocations prevent others from running.

#SBATCH --mem  32G    # for most GPU jobs this is sufficient
#SBATCH --mem  500G   # only if you are explicitly using large CPU memory

GPU compute architecture

The GH200 uses the Hopper GPU architecture. If you compile GPU code (CUDA C++, custom PyTorch extensions, or any package that builds against CUDA), you need to target the correct compute capability.

GPU Architecture Compute capability
A100 Ampere sm_80
H100 Hopper sm_90
GH200 Hopper sm_90a

The a suffix in sm_90a is specific to the Grace Hopper superchip. When compiling CUDA code, set:

-gencode arch=compute_90a,code=sm_90a

For PyTorch, this is handled automatically when you install a CUDA 12.x build of torch — it ships with pre-compiled kernels for sm_90a. No manual flag is needed for standard PyTorch workflows.

Pre-compiled binaries for sm_80 will run but slowly

CUDA includes a JIT fallback: binaries compiled for sm_80 will execute on a GH200 via PTX recompilation, but you lose all Hopper-specific optimisations (flash attention v3, FP8 tensor cores, improved WGMMA instructions). Always recompile for sm_90a if performance matters.

Available LLM models

A shared mirror of large language models is available at /well/kir/mirror/LLM/. No download or HuggingFace token is needed to use these.

HuggingFace format (for use with transformers, FSDP, vLLM, etc.)

Model Path
BERT base uncased /well/kir/mirror/LLM/huggingface/google-bert-bert-base-uncased/
Qwen2.5-72B /well/kir/mirror/LLM/huggingface/Qwen-Qwen2.5-72B

Load directly in Python:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "/well/kir/mirror/LLM/huggingface/Qwen-Qwen2.5-72B",
    torch_dtype="bfloat16",
    device_map="auto",
)

Ollama models

models which can be used with ollama serve
Model Tag
Llama 3.1 llama3.1
Llama 3.2 llama3.2
Llama 4 llama4
Gemma 4 gemma4
Mistral mistral
Qwen3 Coder qwen3-coder-next
Nomic Embed Text nomic-embed-text

Set the model directory before starting Ollama:

export OLLAMA_MODELS=/well/kir/mirror/LLM/ollama/models
ollama serve &
ollama run llama3.1

Need a model that isn't listed?

Contact KIR Research Computing and we can download it to the shared path. Do not download large models to your home directory or scratch space without checking storage quotas first.

Getting help

The GH200 nodes are in early access. We expect rough edges, missing modules, and workflows that need adaptation. Highly recommend contacting KIR Research Computing Manager for

  • Building aarch64 modules or shared environments
  • Requesting additional models in the shared mirror
  • Reporting node issues or unexpected behaviour
  • Help porting existing workflows to the GH200