Everything I know about running SOTA LLMs locally

Note: nothing in this README aside from the tables was written by AI.

Have $2k burning a hole in your pocket and want some local, state-of-the-art machine
intelligence? How about $40k?

If Dario and Altman are giving you heartburn (they should be), read on to figure out
how to run this new kind of computing locally.

My setup

I was lucky/dumb enough to buy 4x RTX Pro 6000s back when they were cheaper. Because
RAM is now so expensive, I opted to build a last-gen DDR4 system to host these cards,
the parts for which I got off eBay. This allowed me to keep base system cost reasonable
while still getting a lot of VRAM.

Another somewhat unusual thing I did was to use PCIe4 switches (from c-payne.com). This
allows the GPUs to communicate to one another "directly" at wire speeds during the
allreduce step in tensor parallelism, rather than having to send all data through the
PCI root complex. The upshot of this is reduced latency between the cards.

Consequently, I'm spending money on VRAM (where it counts) rather than on a PCIe5/DDR5
base system, which is terrifically expensive as of July 2026.

My particular BOM is detailed below.

How much are you willing to spend?

~$2k

A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run
Qwen3.6-27B, which is an awesome model.

You can also run SOTA speed-to-text with
whisper-large-v3, which I find very
useful. That's the model - you'd then access it with my cross-platform stt
harness.

~$40k

At this price level, you get the next step up in model intelligence. Something pretty
close to Claude Opus.

You'd buy 4x RTX 6000 Pros for a total of 384GB of VRAM.

The current best model in my experience that you can run here is a quantized, REAP'd
version of GLM-5.2:
GLM-5.2-Int8Mix-NVFP4-REAP-594B

Note: these are my recommendations, but there are other completely valid ways to spend
your money. For example, there's probably also some regime where rather than getting 4
rtx6kpros, you allocate most of your money to building out a linked 4x DGX Spark
cluster for a total of 512GB VRAM
and use that as the slow, big brain to drive Qwen3.7-27b to do the rote tasks quickly.

Hardware

Here's the hardware I wound up purchasing for the 4x RTX 6000 pro machine.

Base system

A modest, last-gen EPYC system purchased in parts almost entirely from eBay.

Total: $5,587

GPUs

Total: $46,000 (though this is definitely subject to change.)

c-payne PCIe Gen4 Switch Sub-BOM (c-payne.com)

Switch subtotal: ~€1,220 (~$1,330 USD)

GPU enclosure

I had to custom fabricate a wood enclosure for the PCI switch and GPUs, which took
about a day.

I found the PCI switch's builtin fan very loud and seemingly useless, so I simply
unplugged that from the board.

Getting the PCI switches to work properly

There was a lot of fiddling with the BIOS in order to make sure the motherboard wasn't
downregulating the PCI switch speeds.

BIOS Configuration (ROMED8-2T)

Kernel / GRUB Parameters

# /etc/default/grub
GRUB_CMDLINE_LINUX="iommu=off amd_iommu=off nomodeset"
sudo update-grub

# nvidia_uvm P2P fix
echo 'options nvidia_uvm uvm_disable_hmm=1' | sudo tee /etc/modprobe.d/uvm.conf
sudo update-initramfs -u

Without iommu=off, NCCL hangs on multi-GPU P2P.

ACS Disable (critical for switch P2P)

With ACS enabled (default), P2P traffic gets bounced through the CPU root port
instead of staying inside the switch fabric — negating the switch entirely.
pcie_acs_override requires a patched kernel, so we disable via setpci at runtime.

# /usr/local/bin/disable-acs.sh
#!/bin/bash
if [ "$EUID" -ne 0 ]; then
  echo "ERROR: must be run as root"
  exit 1
fi

for BDF in $(lspci -d "*:*:*" | awk '{print $1}'); do
  setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
  if [ $? -ne 0 ]; then
    continue
  fi
  echo "Disabling ACS on $(lspci -s ${BDF})"
  setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done

Run on every boot via systemd oneshot:

# /etc/systemd/system/disable-acs.service
[Unit]
Description=Disable PCIe ACS for GPU P2P
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disable-acs.sh

[Install]
WantedBy=multi-user.target

Verify: lspci -vvv | grep ACSCtl should show all minus signs, and
nvidia-smi topo -m should show PIX between all four GPUs (not PHB/NODE).

GPU Power Limiting

Persistence mode + power cap applied at boot via systemd
(install-gpu-power-limit.sh):

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 350    # 350W per GPU (default 600W)

350W/GPU = 1,400W GPU load, sized for the PSU budget. During the interim
single-1700W-PSU phase (before the 240V circuit), cards ran at ~260W
(4×260 = 1,040W GPUs + ~280W system ≈ 1,320W total).

Verify: nvidia-smi --query-gpu=index,power.limit,power.draw --format=csv

NCCL Environment

export NCCL_P2P_LEVEL=PHB
export NCCL_IB_DISABLE=1
export NCCL_MIN_NCHANNELS=8
export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1
export OMP_NUM_THREADS=8
export SAFETENSORS_FAST_GPU=1

Result

Upstream: Gen4 x16 (~30 GB/s to CPU). P2P through switch: 27.5 GB/s
unidirectional / 50.4 GB/s bidirectional, 0.37–0.45 µs latency — Gen4 line
rate. Note: lspci may still show downstream GPU links as "2.5GT/s (downgraded)"
at idle if ASPM is active anywhere; this is cosmetic — links retrain to Gen4
under load.

Operating the GPUs

In order to avoid installing a 220V circuit, I (probably unwisely) run this rig on a
single 110V circuit, but I power regulate the cards.

I have a script that automatically runs sudo nvidia-smi -pl 350 on boot.

In order

Notes

• Software: Debian 13 Trixie, NVIDIA 595.58.03 open kernel modules, vLLM/SGLang TP4
• Measured P2P through switch: ~27.5 GB/s unidirectional / ~50 GB/s bidirectional (Gen4 max), 0.37–0.45 µs latency
• BIOS gotchas: set slot bifurcation to x16 (not x8/x8) for switch upstream; disable ASPM L1 (causes idle links to report Gen1 2.5GT/s); enable Re-Size BAR; disable SR-IOV
• RTX PRO 6000 Blackwell has no NVLink: all GPU-to-GPU traffic rides the PCIe switch
• Blackwell cards are Gen5 devices negotiating down to Gen4 on this platform

Resources

• A frequently updated repo on getting the most out of 4, 6, or 8 RTX 6000 Pro cards: https://github.com/local-inference-lab/rtx6kpro
• Indie PCI switches that I use: https://c-payne.com