Skip to content

GH200 node

KIR has been granted access to an exclusive GH200 node through a dedicated Slurm partition. This is a powerful but strictly limited resource .i.e. a single node shared across the whole of KIR. We would appreciate you taking a moment to read this page and adopt the recommended methods, so that the node is utilised correctly and remains available to every researcher who needs it

This node carries the highest scheduling weight

Although the node is exclusive to KIR, Slurm still accounts for its usage, and GH200 time carries the highest weight of any GPU on the cluster. For that reason:

  • Use other GPU types for standard workflows. If your job fits comfortably on an A100 or other cluster GPU, please run it there.
  • The node is fair game when it would otherwise sit idle. If the GH200 is free and other GPU partitions have long queues, you are always welcome migrate jobs rather than waiting. Just be ready to keep such jobs brief so the node stays available for the large-memory work it is meant for.

What makes the GH200 different

A GH200 node contains two NVIDIA GH200 Grace Hopper Superchips. Unlike every other GPU node on the cluster, the GH200 is not simply a GPU card plugged into an x86 server. Instead, the CPU and GPU are a single unified superchip connected by NVLink-C2C, a high-speed coherent interconnect that lets both processors share memory transparently.

Feature Standard cluster nodes GH200 nodes
CPU architecture x86_64 (Intel/AMD) aarch64 (Arm Neoverse V2)
Operating system Rocky Linux 8 Rocky Linux 9
GPU A100 / other NVIDIA GH200 144GB
GPU memory 40–80 GB HBM2e 144 GB HBM3e per GPU
CPU–GPU link PCIe (~64 GB/s) NVLink-C2C (900 GB/s)
GPUs per node varies 2 per node
CPU memory DDR4/DDR5 ~480 GB LPDDR5X per superchip

The key figures for GPU-intensive work: 144 GB of HBM3e per GPU and 288 GB total per node, with ~4 TB/s memory bandwidth per GPU.

Why your existing software✴️ will not work

✴️ = Environemnt modules, python virtual and conda environments compiled for existing CPU architecture

There are two independent reasons why software built for the rest of the cluster will fail on the GH200 nodes.

1. CPU architecture mismatch

The Grace CPU uses the Arm (aarch64) instruction set. Every binary compiled for x86_64 — Python, R, compiled extensions, conda environments, container layers — is the wrong architecture and will not execute. When you try to run an x86_64 binary on an aarch64 node you will typically see one of these errors:

Exec format error

/bin/conda: line 6: syntax error near unexpected token `sys.argv'

The second error, which looks like a Python syntax error in a shell script, is actually the shell trying to interpret an x86_64 binary as a shell script because it cannot execute it natively.

This affects everything

Conda environments, pip-installed packages with compiled extensions, pre-built binaries in your home directory, and any software loaded via module load from the standard module tree are all x86_64 and will not work on the GH200 nodes.

2. Operating system mismatch

The GH200 nodes run Rocky Linux 9 (EL9). The rest of the cluster runs Rocky Linux 8 (EL8). Packages and modules built against EL8 system libraries are not guaranteed to work on EL9 and vice versa. The aarch64 module tree described below is built specifically against EL9.

Software options

Be economical with compile time on this node

We have only one GH200 node, and it is used both for running jobs and for installing/building software. Unfortunately, we cannot afford a second node just for builds. Time spent compiling software on the node is GPU time no one else can use. Please be as economical as possible: prefer pre-built shared environments and containers, do heavy development elsewhere (see Develop on Isambard-AI below), and talk to Research Computing before starting any long compilation.

The best route for most groups is to let KIR Research Computing Team build and maintain a shared environment for these GPU architectures for you, saving you the effort and the node's compile time. Shared environments live under:

/well/kir/mirror/gh200_environments/

We also strongly encourage building containers wherever possible. A container is, without a doubt, the best way to maintain a reproducible software stack for these nodes and, as explained below, the same image can run on both our GH200 node and Isambard-AI without reinstalling anything. The Research Computing Manager can assist you with building and testing container images.

Option B — aarch64 module tree

We have started building an aarch64/EL9 EasyBuild software stack at /apps/eb/el9/. The available toolchain generations with aarch64 builds are:

Toolchain Path
2023a /apps/eb/el9/2023a/aarch64/
2024a /apps/eb/el9/2024a/aarch64/
2025a /apps/eb/el9/2025a/aarch64/

This tree is not loaded by default. Add it to your module path with:

module use /apps/eb/el9/2025a/aarch64/modules/all

Then use module avail and module load as normal. We recommend adding this to your Slurm scripts rather than your ~/.bashrc to avoid affecting jobs on other partitions.

Coverage is growing

The aarch64 stack is under active development. If a module you need is missing, contact KIR Research Computing — we can prioritise building it.

Option C — uv for Python environments

For self-managed Python work, uv is a practical option. An aarch64-native build of uv is available at:

export PATH=/apps/kir/eb/hpc-utils/aarch64:$PATH

Add this to the top of your Slurm scripts before any Python or pip commands. This ensures the aarch64 uv binary is used rather than any x86_64 version that may be on your PATH.

Create a virtual environment and install packages as normal:

export PATH=/apps/kir/eb/hpc-utils/aarch64:$PATH

uv venv .venv --python 3.11
source .venv/bin/activate
uv pip install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu126
uv pip install transformers accelerate

CUDA module

Always load a CUDA module before activating your environment. Use CUDA/12.6.0 for GH200 — it is the minimum version with full support for the GH200's compute architecture (see below). 

module purge
module load CUDA/12.6.0
source .venv/bin/activate

GPU compute architecture for software Compilation

The GH200 uses the Hopper GPU architecture. If you compile GPU code (CUDA C++, custom PyTorch extensions, or any package that builds against CUDA), you need to target the correct compute capability.

GPU Architecture Compute capability
A100 Ampere sm_80
H100 Hopper sm_90
GH200 Hopper sm_90a

The a suffix in sm_90a is specific to the Grace Hopper superchip. When compiling CUDA code, set:

-gencode arch=compute_90a,code=sm_90a

For PyTorch, this is handled automatically when you install a CUDA 12.x build of torch — it ships with pre-compiled kernels for sm_90a. No manual flag is needed for standard PyTorch workflows.

Pre-compiled binaries for sm_80 will run but slowly

CUDA includes a JIT fallback: binaries compiled for sm_80 will execute on a GH200 via PTX recompilation, but you lose all Hopper-specific optimisations (flash attention v3, FP8 tensor cores, improved WGMMA instructions). Always recompile for sm_90a if performance matters.

Use Isamabrd-AI for your development cycle(recommended) and BMRC GH200 for Production

To keep the single GH200 node free for production runs, we encourage doing all of your development work on Isambard-AI. Isambard-AI has an abundance of GPU nodes held on reservation for interactive development, so it is a far better place to iterate on code, test builds, and prototype workflows than our one shared node.

  • Easy transfer between the two. You can connect to Isambard-AI directly from BMRC and move content back and forth with minimal effort.

  • Containers make this seamless. This is another reason to prefer containers: because both our GH200 node and Isambard-AI share the same GPU architecture, you can build a container image once and run the same image on both — no need to reinstall anything when you move a workflow across.

Develop and build on Isambard-AI, then bring the finished container (or environment) over to the GH200 node to run your production jobs.

Requesting GH200 resources

Use the following partition in your SLURM scripts:

#SBATCH --partition gpu_kir

A few important points before you submit:

Do not run large job arrays without checking first

Please do not submit job arrays with hundreds of tasks to this node without discussing it with the Research Computing Manager first. A single large array can monopolise the node and block everyone else at KIR for weeks. If you have a high-throughput workload in mind, talk to us and we will help you find an approach that does not lock out other researchers.

Start with one GPU

Begin with a single GPU while you are getting familiar with the nodes. Only request both if your workflow genuinely requires it.

#SBATCH --gres  gpu:1   # single GPU (recommended to start)

Mandatory Slurm directives

#SBATCH --account       gpu_kir.prj
#SBATCH --cpus-per-task 4             # GPU jobs in general will not need more than few. 
#SBATCH --partition     gpu_kit
#SBATCH --gres          gpu:1
#SBATCH --time          1-00:00:00    # Make sure to add the time variable as the default time is quite a generous number

Always set --cpus-per-task If you do not specify CPU cores, Slurm will allocate all 72 Grace CPU cores on the superchip to your job by default. This blocks other users from running even if they need only a handful of cores. Always be explicit:

#SBATCH --cpus-per-task  8   # adjust to what your job actually needs

Always set --time

The default time limit is 5 days. Please always set an explicit wall time — it helps the scheduler and frees the node promptly when your job finishes. During early access we recommend keeping jobs short while you are iterating on your workflow.

#SBATCH --time  01:00:00   # HH:MM:SS

Be mindful with --mem

Each node has approximately 1 TB of CPU memory. You can request up to that, but please only do so if your workflow genuinely requires it. Idle over-allocations prevent others from running.

#SBATCH --mem  32G    # for most GPU jobs this is sufficient
#SBATCH --mem  500G   # only if you are explicitly using large CPU memory

Interactive sessions vs batch jobs

Please choose sbatch for work on the GH200 wherever possible. A batch job releases the node the moment your computation finishes, returning it to the queue for the next pending job.

Interactive sessions (srun --pty, or salloc) behave differently: they hold the allocation until you exit or the wall time is reached, not when your computation completes. For example, if you request a long wall time and your process finishes overnight, the session will continue to occupy the node until you manually terminate it or the wall time expires. On a single shared node, this can leave the GPU idle for hours while other jobs wait in the queue.

If you do need an interactive session:

  • Exit it as soon as your computation is done, rather than leaving it running to the wall time.
  • Request a wall time close to what you actually need.

Use Isambard-AI for interactive and exploratory work

Isambard-AI has GPU nodes reserved for interactive development. For debugging, prototyping, and any exploratory work, please use Isambard-AI where possible and reserve the GH200 for production batch jobs. See Develop on Isambard-AI.

Being a good citizen

The GH200 is a single node shared across all of KIR, and it is a limited resource. A few habits keep it accessible to everyone who needs it:

  • Right-size your jobs — use other GPU types for standard workflows and reserve the GH200 for genuinely large-memory work.
  • Keep jobs short and set an explicit --time so the node frees up promptly.
  • If possible, do development on Isambard-AI, not on the BMRC GH200.
  • Check whether a prebuilt environment or container already exists for your software before building — contact the Research Computing Manager if unsure.
  • Avoid launching large job arrays on this partition

Thank you for helping make sure every researcher at KIR who needs this resource can get to it without prolonged waits.

Getting help

Please contact the KIR Research Computing Manager for:

  • Building aarch64 modules, shared environments, or containers
  • Requesting additional Large Language Models in the shared mirror
  • Reporting node issues or unexpected behaviour
  • Help porting existing workflows to the GH200
  • Planning any high-throughput or long-running work on the node