need analysis there need analysis there need analysis there ...

If that matches, check the server load log for one line:

Parameter lm_head.weight_scale not found in params_dict

That warning plus degenerate repetition is a specific, known failure with a specific fix. This post is the short version of an A/B verification I ran on a single RTX PRO 6000 Blackwell (SM120, 96 GB).

What is actually broken

Mixed-precision compressed-tensors checkpoints such as unsloth/Qwen3.8-27B-NVFP4 quantize most of the network to NVFP4 but keep a few parts, including lm_head, in FP8 W8A8 with per-channel weight scales. The quant config names the head explicitly with a re:.*lm_head target.

SGLang's CompressedTensorsConfig.get_quant_method() used to dispatch only LinearBase and FusedMoE. ParallelLMHead fell through, so the head loaded via the unquantized embedding path and the FP8 weight_scale tensor was never applied. Raw FP8 weights (values up to around 448) were consumed as if they were bf16. With per-channel scales averaging 1.55e-4 and a 12x row-to-row spread, the logit ranking is corrupted badly enough that greedy decoding locks into a repetition loop immediately.

The same checkpoint works on vLLM because vLLM fixed the identical gap earlier (vllm-project/vllm#37291).

The fix, and where it is not

SGLang fixed this on main on 2026-08-19: PR #35228 added a dedicated get_lm_head_scheme resolver, so a head named by the quant config now routes through the compressed-tensors linear method and its scale loads.

Two things worth knowing as of late August 2026:

• The fix is in no released version. v0.5.18, the latest release, was cut from a branch point that predates the fix, so pip-installed SGLang still has the bug. You need main at or past commit 5375babb.

• Separately, sglang[all]==0.5.18 currently does not resolve from PyPI at all (a cuda-tile==1.6.0rc5 pin with no matching distribution), so you could not stay on the release even if you wanted to.

The A/B, one variable

Same machine, same venv, same launch flags, TP=1, --attention-backend flashinfer, temperature 0. The only difference is the git checkout.

Full table and logs are in the issue thread: sgl-project/sglang#34895. The before and after rows also live in a public Blackwell serving matrix I maintain of verified model x engine x GPU cells.

Practical guidance

• On SGLang releases up to v0.5.18 with an FP8-lm_head checkpoint: you will hit this. Either build from main past 5375babb or serve that checkpoint on vLLM until an SGLang release ships the fix.

• The failure is silent at startup. The server boots cleanly and the health endpoint is green; only generation is garbage. If you automate deployments, grep load logs for weight_scale not found and fail the deploy on it.

• To identify your GPU and stack cell quickly, blackwell-doctor prints a stable environment key you can match against the serving matrix: github.com/jahnclawdmonet/blackwell-doctor.

I keep an index of exact Blackwell serving error signatures mapped to reproductions here.

```plaintext

triton.runtime.errors.OutOfResources: out of resource: shared memory,

Required: 102400, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

```

We hit this, measured exactly where the 1,024 missing bytes come from, and posted the measurements on the upstream issue (vllm-project/vllm#53748). This post is the diagnostic method, because it applies to any Triton kernel on this hardware class, not just this one bug.

## The hardware fact underneath

Datacenter Blackwell parts advertise a larger per-block shared-memory budget than the workstation and consumer silicon. On an RTX PRO 6000 Blackwell (sm_120), `torch.cuda.get_device_properties` reports:

| property | bytes |

|---|---:|

| `shared_memory_per_block_optin` | 101,376 |

| `shared_memory_per_multiprocessor` | 102,400 |

GB10 (DGX Spark) reports the same 101,376-byte per-block opt-in limit. So a kernel tuned to fit 102,400 bytes fits the SM but not the block, and the whole workstation Blackwell class fails together. If your kernel config was validated on H100 or B200 numbers, this is the cliff you fall off when someone runs it on a desk machine.

## The five-minute probe

Do not debug this inside the serving engine. Launch the suspect kernel directly with the exact tile configuration the engine would pin, sweep `num_stages`, and read the compiled kernel's shared-memory size. For vLLM's MLA decode kernel (`_fwd_grouped_kernel_stage1`) with the DeepSeek tile shape (`BLOCK_DMODEL=512, BLOCK_DPE=64, BLOCK_DV=512, BLOCK_N=32, BLOCK_H=16, num_warps=4`):

| num_stages | KV bf16 | KV fp8e4m3 |

|---|---|---|

| 1 | 59,392 (launches) | 83,968 (launches) |

| 2 | 63,488 (launches) | 102,400 (OutOfResources) |

| 3 | 63,616 (launches) | 102,656 (OutOfResources) |

| 4 | 63,616 (launches) | 102,656 (OutOfResources) |

The table says everything. With bf16 KV, an extra pipeline stage costs about 4 KB and even four stages sit far under the limit. With fp8 KV, a stage costs about 18 KB, and stage two lands on 102,400 bytes: exactly the full per-SM budget, and 1,024 bytes over the per-block cap. The crash is specific to the fp8 KV path; MLA with bf16 KV is fine at these tile sizes.

The probe is ordinary Triton: import the kernel function, build dummy tensors for the tile shape, launch once per (dtype, num_stages) combination, and catch `OutOfResources`. The compiled size is on `kernel.metadata` after a successful launch. No server, no model download, under a minute per configuration.

## Reproducing end to end

DeepSeek-V2-Lite-Chat is the smallest convenient MLA model with the same `Lk=576` head layout (512 latent plus 64 rope):

```shell

vllm serve deepseek-ai/DeepSeek-V2-Lite-Chat --trust-remote-code \

--enforce-eager --kv-cache-dtype fp8 --max-model-len 4096

```

On sm_120 the backend selector reports `TRITON_MLA` as the only MLA decode option, so there is no fallback path: the engine dies during the startup dummy run (vLLM 0.27.1, torch 2.13.0+cu130, triton 3.7.1, CUDA 13.1).

## What a durable fix looks like

The upstream patch direction pins `num_stages=1` for MLA kernels on non-HIP devices, which makes the server start and generate correctly (we validated greedy output on sm_120 after applying it). But the sweep shows why a blanket `is_mla` condition costs something: bf16 KV at two stages fits comfortably (63,488 < 101,376), and the second stage exists for throughput. The robust rule is to compute the tile's actual shared-memory requirement and compare it against `shared_memory_per_block_optin` for the device you are actually on. That handles GB10, RTX PRO 6000, and whatever ships next, without a special case per chip.

The general lesson for anyone shipping Triton kernels: your shared-memory budget is a per-block, per-device property, not a compile-time constant, and the workstation cards your users actually own report smaller numbers than the datacenter parts your CI runs on. Probe first; it takes five minutes.

This came out of inference engineering work on Blackwell workstations. The benchmark that surrounds it, with raw data and reproduction scripts, is here: https://conatus.jahn.ai/ai-engineering/sample-report