contact

LLM serving that works on the GPUs you actually ship.

When vLLM or SGLang breaks or runs slow on PCIe RTX PRO 6000 servers, we find out why, fix it upstream, and measure the speedup on the multi-GPU box. RTX 5090 next.

get your stack tested see the results

Same GPUs. Same model. Different stack.

Two runs on rented RTX PRO 6000s. The left lane is the stack before one change, the right lane is the same stack after it. Each lane streams at the speed we measured.

users
before #12--enforce-eager, prefix caching off
this user-
after #12CUDA graphs + prefix caching on
this user-
$ vllm bench serve --max-concurrency 1 --random-input-len 1024 --random-output-len 512before 17.4 tok/s total after 125.0 tok/s total 7.2x

Independent test of a public repo: Puget Systems' Team LLM pack v1.1.0, menu entry 7, run exactly as installed. Puget merged this change as puget-docker-app-packs#12. 2x RTX PRO 6000 Workstation, Nemotron-3-Super-120B-A12B-NVFP4, TP2, vLLM nightly e68efc68. GSM8K: 99/100 before vs 98/100 after, every reply answered. Each lane streams at the measured output rate per user; the text is illustrative. Full report.

What serves, what breaks.

Every row is one run on rented hardware with a fixed budget. Each one links to its full report: image, flags and commands.

GPUs Model Engine Outcome Number Run
2x RTX PRO 6000 WS cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit vLLM nightly e68efc68 serves Puget pack v1.1.0 entry 1 as installed; its eager flags cost 6.3x to 7.1x at 1 user across two hosts until Puget merged #15, which turned graphs on 245.6 tok/s at 1 user (34.5 before #15) puget-team-llm-e1
2x RTX PRO 6000 WS cyankiwi/Qwen3.6-27B-AWQ-INT4 vLLM nightly e68efc68 serves Puget pack v1.1.0 entry 2 as installed; its eager flags cost 3.5x at 1 user until Puget merged #15, which turned graphs on 112.2 tok/s at 1 user (32.4 before #15) puget-team-llm-e2
2x RTX PRO 6000 WS cyankiwi/Qwen3.5-35B-A3B-AWQ-4bit vLLM nightly e68efc68 serves Puget pack v1.1.0 entry 3 as installed; its eager flags cost 6.4x at 1 user until Puget merged #15, which turned graphs on 221.7 tok/s at 1 user (34.6 before #15) puget-team-llm-e3
2x RTX PRO 6000 WS nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 vLLM nightly e68efc68 serves Puget pack v1.1.0 entry 6 as installed; its eager flags cost 9.1x at 1 user until Puget merged #15, which turned graphs on 324.3 tok/s at 1 user (35.6 before #15) puget-team-llm-e6
2x RTX PRO 6000 WS nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 vLLM nightly e68efc68 serves Puget pack v1.1.0 entry 7 as installed; its eager flags cost 7.2x at 1 user until Puget merged #12, which turned graphs on 125.0 tok/s at 1 user (17.4 before #12) puget-team-llm-e7
2x RTX PRO 6000 WS cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit vLLM nightly e68efc68 serves Puget pack v1.1.0 entry 8 as installed; its eager flags cost 4.8x at 1 user until Puget merged #15, which turned graphs on 227.3 tok/s at 1 user (47.6 before #15) puget-team-llm-e8
1x RTX PRO 6000 WS nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8 vLLM 0.26.0 serves Wowza Video Intelligence Framework's engine config; its --enforce-eager costs 2.8x at 1 user (text only, the video path was not measured) 108.8 tok/s at 1 user (38.9 shipped) wowza-vif
1x RTX 5090 deepseek-ai/DeepSeek-R1-Distill-Llama-8B RunPod worker-vllm, vLLM 0.32.0 serves Hub defaults; the worker's out-of-memory relaunch to eager costs 2.0x at 1 request. MAX_NUM_SEQS=32 keeps graphs and the full context, and boots in 80 s instead of 251 s 97.7 tok/s at 1 request (49.2 after relaunch) runpod-worker-vllm-r1d8b
1x RTX PRO 6000 SE RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic RunPod worker-vllm, vLLM 0.32.0 serves Hub defaults; the same relaunch costs 0-3% on this model. MAX_NUM_SEQS=32 boots in 90 s instead of 210 s 19.2 tok/s at 1 request (19.1 after relaunch) runpod-worker-vllm-l70b
1x RTX PRO 6000 WS nvidia/Qwen3.6-35B-A3B-NVFP4 vLLM 0.28.0 and nightly serves 25 min saturated, graphs on, 0 failed 2.8-2.9K tok/s at 70 users vllm-54331
1x RTX PRO 6000 SE Ornith-1.5-35B-A3B-NVFP4, fp8 KV, video vLLM 0.28.0 serves 25 min saturated with video, 0 failed 2,856 tok/s at 70 users vllm-54331
4x RTX PRO 6000 SE DeepSeek-V4-Flash SGLang v0.5.22 serves 1.10x vLLM on 9 x 90K prompts; the reported gap did not show 8,983 prefill tok/s sglang-33422
4x RTX PRO 6000 SE DeepSeek-V4-Flash, TP4 + EP4 SGLang v0.5.22 serves one socket; NCCL's default ran over SHM, not P2P. NCCL_P2P_LEVEL=SYS switched it to P2P and cut TTFT 25-30%, median ITL 2-3% 7,048 ms TTFT on 9 x 90K at 1 user (9,910 default) sglang-33422
4x RTX PRO 6000 SE DeepSeek-V4-Flash vLLM 0.25.1 recipe serves same 9 x 90K prompts 8,171 prefill tok/s sglang-33422-vllm
4x RTX PRO 6000 SE DeepSeek-V4.1-Flash + DSpark vLLM nightly, FlashInfer 0.7.2, vllm#60762 serves 10.7K and 106K prompts with graphs on; crashed on FlashInfer 0.6.18 101.5 decode tok/s vllm-56771
8x RTX PRO 6000 SE DeepSeek-V4.1-Flash, TP8 vLLM 0.33.0 + vllm#60762 backport serves dual socket; FlashInfer PCIe IPC all-reduce alone cut median ITL 20% with 32K TTFT unchanged. Adding NCCL_P2P_LEVEL=SYS gave no more decode and made 32K TTFT 18-19% slower on this host. Repeated on a second rental 118.5 tok/s at 1 user (95.3 default), GSM8K 49/50 vllm-59203
4x RTX PRO 6000 Max-Q nvidia/DeepSeek-V4.1-Flash-1017-NVFP4, TP4 vLLM 4fab5a07 + vllm#61530 serves with vllm#61530, merged 11-11; FlashInfer PCIe IPC cut median ITL 9.9%. One TP4 engine holds 116 GiB of host RAM it cannot give back, mostly Engram tables, so two do not fit a 125 GiB host 118.0 tok/s at 1 user (130.6 with FlashInfer PCIe IPC), GSM8K 49/50 vllm-61530
2x RTX PRO 6000 SE nvidia/Qwen3.8-Flash-Next-NVFP4, EP2 + NEXTN SGLang nightly 2026-10-21 (c1c287a4) serves no patches (fix sglang#44480 merged); TP2 still needs #42566, custom all-reduce with expandable_segments still needs #44448 187.5 / 820.3 / 1,477.8 tok/s at 1 / 16 / 64 users, GSM8K 0.995 sglang-44291-stack
1x RTX PRO 6000 SE Apertus-70B FP8 vLLM 0.25.1 serves the reported garbage output did not reproduce GSM8K-200 0.775 vllm-61100
1x RTX PRO 6000 SE DeepSeek-V2-Lite-Chat-FP8 vLLM 0.31.0 crashes at startup when routed to CUTLASS; the default config is fine - vllm-47990
1x RTX PRO 6000 nvidia/Qwen3.6-35B-A3B-NVFP4 vLLM, FlashInfer 0.7.0.post1 fixed CUDA-graph crash at startup, gone with FlashInfer 0.7.1 - vllm-49476

21 runs. Updated 2026-11-12. All kits: github.com/adi-suresh01/sm120-kit

How a run works.

  1. 01

    Real hardware

    Rented RTX PRO 6000 Workstation and Server Edition boxes, one card or several over PCIe, the same parts integrators ship. One RTX 5090 run so far; more are next.

  2. 02

    A fixed budget

    Every run gets a GPU budget before it starts, and the pod shuts down when the run ends.

  3. 03

    A reproducible report

    Image digest, flags, driver and commands. Rerun it on your own box and you should get the same numbers.

  4. 04

    Fixes go upstream

    When vLLM, SGLang or FlashInfer is the problem, the fix goes to the project as a PR, not into a fork you have to maintain.

Start free. Pay when it's useful.

Pilot Free

Your current pack on one GPU config, tested in public. You get the report, and any fixes as PRs.

start here
Acceptance test from $500 per deployment

Your customer's exact model, prompt lengths and multi-GPU box, tested before it ships. White-labeled report.

Fix sprint from $2.5K fixed

One broken or slow deployment, fixed to a target we agree on up front.

Stack Care from $1.5K / month

Up to 3 multi-GPU configs x 5 models. When an engine release breaks or slows one, you get the root cause, an upstream fix and the measured speedup. Known-good image tags and flags, 2-business-day escalations.

for integrators

Every one starts with the same email: get your stack tested.

I'm Aditya, an ML engineer who most recently served models on PCIe RTX PRO 6000 clusters at Kos AI, now founder of Provedex.

Every number on this page comes from sm120-kit, the public harness that rents the GPUs, runs the matrix and writes the report. More on GitHub.

Get your stack tested.

Tell us the GPUs, model and engine, and what breaks or feels slow.

email hello@provedex.io opens a short template