Skip to content

Getting started

Install PantheonGPU and run a first validation

PantheonGPU automatically detects CUDA, ROCm/HIP, or mock mode. Run the pantheon command directly after installation.

Install

Install the basic build tools:

sudo apt-get update
sudo apt-get install -y make g++

Install the compiler for your platform. You only need one:

sudo apt-get install -y nvidia-cuda-toolkit

sudo apt-get install -y hipcc

Pantheon is open source. Building from source lets you read exactly what will run on your hardware; a package is quicker if you just want to run it.

pipx install pantheon-gpu

Installs a pantheon command on your PATH. pipx keeps it in its own environment, which is what recent Ubuntu and Debian releases require of anything installed outside the system package manager; pip install --user pantheon-gpu works too where that restriction does not apply. Add the reports extra (pipx install "pantheon-gpu[reports]") for spreadsheet export.

The wheel carries the kernel sources rather than prebuilt binaries, so the first run compiles the workloads for your GPU into a per-user cache. That takes a minute or so once, and needs make, a C++ compiler, and the CUDA or ROCm toolkit from the tabs above.

conda install -c conda-forge pantheon-gpu

From conda-forge, so mamba and micromamba install it the same way. It installs both pantheon and pantheon-gpu; they are the same program.

The package carries the kernel sources rather than prebuilt binaries, so the first run compiles the workloads for your GPU into a per-user cache. That takes a minute or so once, and needs make, a C++ compiler, and the CUDA or ROCm toolkit from the tabs above.

A release reaches conda-forge a little after it reaches PyPI. conda search -c conda-forge pantheon-gpu shows which versions are there.

docker run --rm -t --gpus all -v "$PWD:/reports" \
  ghcr.io/pantheongpu/pantheon:latest --test tensor_virus --duration 60

The image carries the CUDA 12.8 toolchain, so nothing is installed on the host beyond the NVIDIA driver and the NVIDIA Container Toolkit. Kernels still compile for the GPU actually present on first run; mount /root/.cache as well to keep the compiled cache across containers. Reports land in the directory mounted at /reports.

Tags: latest, a version (1.2.4), or a version pinned to its toolchain (1.2.4-cuda12.8).

On AMD hardware use the ROCm variant, which carries the ROCm 6.4 toolchain instead and needs the device nodes rather than --gpus:

docker run --rm -t --device=/dev/kfd --device=/dev/dri --group-add video \
  -v "$PWD:/reports" \
  ghcr.io/pantheongpu/pantheon:latest-rocm --test tensor_virus --duration 60

If --gpus all is refused

failed to discover GPU vendor from CDI: no known GPU vendor found (newer Docker) and could not select device driver "" with capabilities: [[gpu]] (older Docker) both mean the same thing: the host has no NVIDIA Container Toolkit registered, so Docker cannot hand a GPU to any container. If nvidia-smi works on the host, fix it with:

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

If the package is not found, add NVIDIA's repository first: see the NVIDIA Container Toolkit install guide. A stray ERROR: init ... result=9 line at container start is harmless noise from the base image's GPU hooks on hosts without passthrough; Pantheon's own output follows it.

sudo dnf install 'dnf-command(copr)'
sudo dnf copr enable saqibkhanpantheongpu/pantheon-gpu
sudo dnf install pantheon-gpu

Built for Fedora 43/44 and EPEL 10 (RHEL, Rocky and Alma 10). EPEL 9 is not built: its Python stack cannot satisfy the build's setuptools floor, so use pipx or the Docker image there. Add the CUDA or ROCm toolkit separately for real hardware.

git clone https://github.com/pantheongpu/pantheon.git
cd pantheon
pip install -r requirements.txt
make PLATFORM=CUDA -j$(nproc)     # or PLATFORM=HIP for AMD

There is no build-time auto-detection, so pass PLATFORM= explicitly. Use PLATFORM=MOCK to build a CPU backend that needs no GPU at all, which is useful for trying the tooling before committing hardware to it.

Run it from the checkout with python3 pantheon.py in place of the pantheon command used below.

curl -fsSL https://pantheongpu.com/apt/pantheon-archive-keyring.asc \
  | sudo tee /usr/share/keyrings/pantheon-archive-keyring.asc > /dev/null
echo "deb [signed-by=/usr/share/keyrings/pantheon-archive-keyring.asc] https://pantheongpu.com/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/pantheon.list
sudo apt update && sudo apt install pantheon-gpu

Installs both pantheon and pantheon-gpu; they are the same program. Telemetry and spreadsheet export come from Recommends, so apt pulls them by default and they can be left out on a minimal install; the CUDA toolkit is only a Suggests.

The workloads compile on first run, which is why the package depends on g++ and make. Add the CUDA or ROCm toolkit separately for real hardware.

VERSION=1.2.4
BASE="https://github.com/pantheongpu/pantheongpu_website/releases/download/v${VERSION}"
wget "${BASE}/pantheon_gpu-${VERSION}-py3-none-any.whl"
wget "${BASE}/SHA256SUMS" && sha256sum --ignore-missing -c SHA256SUMS
pipx install "./pantheon_gpu-${VERSION}-py3-none-any.whl"

Installs a pantheon command on your PATH. pipx keeps it in its own environment, which is what recent Ubuntu and Debian releases require of anything installed outside the system package manager; pip install --user works too where that restriction does not apply.

The wheel carries the kernel sources rather than prebuilt binaries, so the first run compiles the workloads for your GPU into a per-user cache. That takes a minute or so once, and needs make, a C++ compiler, and the CUDA or ROCm toolkit from the tabs above.

Run a first test

Run a short inventory test:

pantheon --test baseline_metrics --duration 10

Then run a targeted stress test on GPU 0:

pantheon --test fp64_virus --duration 30 --gpu 0

Read the verdict

Every run ends with one word per GPU and the evidence behind it:

VERDICT GPU 0, NVIDIA H100 80GB HBM3: WATCH  (thermally throttled on 2; correctable errors on 8)
  ! thermal: memory_read thermally throttled, GPU at 95 C, memory_read_agg thermally throttled, GPU at 94 C
  ! correctable errors on 8 workload(s): pcie.bad_tlp +9, pcie.lcrc +6
  note: PCIe link recovery events on 8 workload(s): link power-state cycling, not a fault

HEALTHY means every workload completed, no error counter that matters moved, and nothing ran hot. WATCH names what deserves a second look: a thermal limit, a GPU at 90 C or memory at 95 C, correctable ECC or PCIe errors, or a workload that did not complete. FAULT means a memory diagnostic failed or an uncorrectable error was counted. The verdict is written into the report as well.

pantheon --test quick is the ten-minute version: idle baseline, memory read, a march test, a retention check and one power-limited compute load.

Check a runner's GPUs in GitHub Actions

On a self-hosted runner with a GPU, one step tests the cards before the job uses them, and fails the job if a card is faulty:

jobs:
  train:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: pantheongpu/gpu-health-check@v0
      - uses: actions/checkout@v4
      - run: python train.py

The action installs Pantheon, runs a short set of workloads on every GPU of the runner, writes each card's result into the job summary and keeps the full reports as a workflow artifact. A check that could not run fails the job too, so a green tick means a card was tested. The runner needs the same things as any other install: the GPU's driver and compiler, make, a C++ compiler and Python.

Pantheon GPU Health Check on the GitHub Marketplace lists its options. GitHub's own hosted runners have no GPU, so this is for runners you host or rent.

For Slurm, NHC and ReFrame, see Integrations.

Profiling and reports

Verification of workload output is on by default; pass --skip_verify to turn it off. Use --profile to collect performance counters, traces, and a per-workload HTML summary:

sudo pantheon --test llm_decode --duration 60 --gpu 0 --verify --profile

Three things commonly stop a profiled run before it starts:

  • It needs root. NVIDIA drivers ship with profiling restricted to administrators, so an ordinary user sees ERR_NVGPUCTRPERM and almost no counters.
  • Both tools must be installed. On NVIDIA that is ncu and nsys; Pantheon refuses the run rather than producing half a profile. On AMD it is rocprofv3.
  • Nsight Compute must match the driver. A build newer than the driver installs cleanly and then fails every capture with stub libcuda.so. Install the package matching the CUDA version nvidia-smi reports, such as cuda-nsight-compute-12-8.

Profiling also needs host memory comparable to the GPU memory in use, because the profiler stages device memory in host RAM while replaying kernels. On a machine whose RAM is close to its VRAM, run without --profile rather than lowering --mem.

See Test Documentation for workload guidance, Releases and Downloads for other installation methods, and Reports for output details.

Uninstall

Remove the package the way it was installed:

pipx uninstall pantheon-gpu                    # PyPI or a downloaded wheel
conda remove pantheon-gpu                      # conda-forge
sudo apt-get remove pantheon-gpu               # apt repository
sudo dnf remove pantheon-gpu                   # Fedora COPR
docker rmi ghcr.io/pantheongpu/pantheon:latest # container image

To drop the apt source as well, remove /etc/apt/sources.list.d/pantheon.list and /usr/share/keyrings/pantheon-archive-keyring.asc; on Fedora, sudo dnf copr disable saqibkhanpantheongpu/pantheon-gpu.

Releases up to v1.0.19 shipped as a Debian package named pantheongpu. Remove one of those with:

sudo apt-get remove pantheongpu

The workloads compile into ~/.cache/pantheongpu on first run and stay there after the package is gone. To remove that cache, the apt source, any runtime files under /opt/pantheongpu, and whichever package is installed, in one go:

curl -fsSL https://pantheongpu.com/uninstall.sh | sudo sh

This leaves CUDA, ROCm, system compilers, and benchmark reports outside PantheonGPU's installation and cache directories untouched.