An inference runtime for Qwen3.8 27B was constructed using an NVIDIA RTX PRO 4000 Blackwell SFF (24 GB), achieving an average throughput of 50.44 tok/s.
Practical 256K context operation was also verified, including an input of 261,500 tokens and a generation of 256 tokens. The system maintained 12.61 tok/s even at the end of the cache, and no out-of-memory errors occurred.
The model features a 64-layer dense configuration. It repeats a pattern of three Gated DeltaNet layers and one full-attention layer, meaning only 16 layers require a KV cache. This reduces the memory load for the 256K context.
Q4_0 quantization was adopted because it offered a superior balance of speed and quality compared to IQ4_XS and Q4_K_M. Speculative decoding via MTP was also effective, with a confirmed 2.81x throughput increase compared to using the target model alone.
A custom llama.cpp build is 21.97% faster than the standard master branch. This figure represents the performance of the integrated system, rather than a simple sum of individual optimization elements.
Source: Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU（HN 38pt・36コメント） (HN Search (backfill), 2026-08-17)