Qwen3.8-27B on One RTX 3090 vs Two: +20% Decode, +14% Cold Prefill, and 3x on Cached Prompts

TL;DR: Qwen3.8-27B — the dense 27.8B that dropped on August 14 — fits on a single RTX 3090 at W4A16 and decodes at ~125–155 tokens/sec. Splitting it across two cards with tensor parallelism buys roughly 20% more decode (noisy: +13% in one run, +22% in the next) and 14% faster prefill on average on a cold prompt (23% at the top of the ladder: an 8,600-token prompt takes 8.5 s to first token on one card and 6.9 s on two). On a prompt the server has already seen, the pair answers 2–3x faster — but that is the prefix cache, not the cards, and the script below measures the two separately. At the API list price of $3.00 per million output tokens, one million tokens cost 0.21–0.34 BGN of electricity, about $0.12–0.20, on the box that serves them. the box vader is the machine I run this on: three RTX 3090s in an ASUS Z10PE-D8 WS, dual Xeon E5-2660 v4 (56 threads), 60 GB of RAM, Ubuntu 26.04. It has been my inference box since the Whisper experiments, and since July it has also carried the power-limiting work (the meters post is where those numbers live). Two of the three cards are in this test; the third sits idle. The model under test is the one that is currently loaded on it: single-card TP2 container qwen38-27b-single-1 qwen38-27b-tp2-single-1 image ghcr.io/syv-ai/qwen38-27b-rtx3090 same, :latest weights Qwen3.8-27B-W4A16-AutoRound-fast same cards 1x 3090 (22.6 GB VRAM) 2x 3090 (21.7 GB each) max context 131,072 262,144 port 18021 18022 Both are vLLM, both have been up two days, both report healthy, and both run with prefix caching on — which is why every number below comes in two flavours, cold and warm. The W4A16 quantization is what puts a 27.8B dense model inside one 24 GB card — 22.6 GB of it, with the KV cache budget for the rest of the 128k window coming out of what is left. the method No benchmark harness, no vllm bench — a 150-line script (bench_qwen38_v2.py) that sends a real chat-completion request with a prompt of a given size, streams the answer, and times it. For each prompt size it runs three times and takes the median: prompt sizes: calibrated so the server reports ~600, ~2,400, ~4,700, ~9,300 prompt tokens (the script probes the tokenizer first, then scales the prompt to hit the target) output: max_tokens=256, temperature=0 measured: time to first SSE event, total time, and the server-reported token counts from stream_options.include_usage — not my own chunk counting cold vs warm: every cold request carries a unique first line, so no cached prefix can match it; the warm pass repeats one identical request after a warm-up call That last bullet is the one that decides whether a benchmark means anything. With prefix caching on — and it is on by default in vLLM — timing the same prompt twice does not measure prefill the second time, it measures a cache lookup. The tell is arithmetic: 8,600 tokens "prefilled" in 0.56 s is 15,000 tokens/sec, roughly ten times what two 3090s can actually push through a 27B model. If your TTFT numbers look like that, you are timing the cache. So every cold request here is salted with a unique first line, and the warm pass is run deliberately and reported in its own column. One caveat: this is a single-request benchmark. vLLM's continuous batching is where production throughput lives, and I did not hammer it with concurrent requests. What I measured is the latency you feel as the one person using the box — which, for a homelab, is most of the time. decode: the second card buys 15–30%, noisily The single card holds 123–154 tokens/sec across the ladder; the TP2 pair runs 146–193. A 27B dense model at W4A16 on one 3090 is memory-bandwidth-bound in a way that does not care much about prompt size, because the KV cache for these sizes is small next to the weights. The pair is faster at every size but one (at 4,700 tokens the two came out level), and the average gain was +22% in this run against +13% in a run an hour earlier. Single-stream decode on these cards moves by about 10% between runs an hour apart, so I am not going to print a decimal: call it 15–30%, and expect the lower end. That is a real number and a small one. For a 27B model, tensor parallelism across two cards is not the 2x you get from adding compute — the all-reduce between the cards eats most of it. prefill: cold is the cards, warm is the cache Two columns, two different questions. Cold — a prompt the server has never seen — is prefill, and prefill is compute-bound: the single card does 1,100–1,220 tokens/sec of it, the pair 1,230–1,360. Time to first token climbs from 0.53 s to 8.5 s on one card as the prompt goes from 600 to 9,300 tokens, and from 0.50 s to 6.9 s on two. The gap grows with the prompt: 6% at 600 tokens, 23% at 9,300, 14% on average. That is what the second card buys you on a fresh long prompt. Warm — the same prompt again — is the prefix cache doing its job, and here the pair really is 2–3x faster: 0.5–0.75 s flat across the ladder against 0.5–1.7 s on the single card, whose smaller KV pool and slower re-read of the uncached tail show at every size above 600 tokens. It is a cache number rather than a prefill number, and it is also the one you feel most often, because every follow-up turn in a chat is a warm request. The routing rule that falls out: a long fresh prompt goes to the pair and saves about a fifth of its wait; a conversation stays on whichever lane it started on, so its prefix stays cached. For prompts under about 2,000 tokens the single card is within noise of the pair on both counts and draws one card's power. what it costs The box draws 620 W at the wall with both containers loaded and serving — 247 W on one TP2 card, 328 W on the other, 45 W on the single-card 3090 sitting at P8 with its weights resident. Those are the monitor's readings from the day the containers went up, not a measurement taken during these runs; the governor from the meters post is power-capping it, holding 79°C. At the API's list price — $3.00 per million output tokens, DeepInfra's rate for Qwen3.8-27B as listed on llm-stats — one million generated tokens takes about 6,667 seconds at 150 tokens/sec. At 620 W that is 1.15 kWh. On the Bulgarian dual-rate tariff that is 0.34 BGN by day (0.30 BGN/kWh) or 0.21 BGN by night (0.18 BGN/kWh) — roughly $0.20 or $0.12. The API price is 15x to 25x the electricity, depending on when the box runs. I am not going to pretend that makes the API a bad deal. The API does not cost you 620 W of heat in your room, it does not need a 60 GB machine, it does not need you to keep a W4A16 quant on a 24 GB card, and its $3.00 buys you their datacenter's power, cooling and idle capacity, not yours. But for a box that is already on and already paying its standby tax, every token it serves is almost pure margin. The electricity argument for local inference is not "it is cheaper than the API" in any total-cost sense — it is "the marginal cost of a token on hardware you already run is an order of magnitude below the list price." the verdict Qwen3.8-27B is the first 27B-class dense model I have run that I would put in front of real work on this box, and the numbers say why: It fits. W4A16 on one 3090, 22.6 GB, with room for a 128k context window. The TP2 variant doubles the context to 262k by splitting the weights. It is fast enough to feel like an API. 125–190 tokens/sec decode is faster than most hosted endpoints feel in a chat UI. The part you notice is a cold long prompt: several seconds on either lane. The two-card setup is not 2x. It is roughly +20% decode and +14% cold prefill, and 2–3x on warm follow-ups because the pair keeps a bigger cache. Buy the second card for the context window and the cache, not for raw speed. The marginal cost is 0.21–0.34 BGN, about $0.12–0.20, per million tokens against a $3.00 list price, on hardware that is already drawing its standby power. Benchmark with the cache in mind. If a prefill number looks like 15,000 tokens/sec on consumer cards, it is the cache. Salt every request, or turn prefix caching off, and report which one you did. The script, the raw JSON and the charts are in the repo: qwen38-bench. If you run a different quant or a different card, I want your numbers — the interesting part of this post is the shape of the curve, and the shape changes with the hardware. vader: 3x RTX 3090 (two used here), dual Xeon E5-2660 v4, 60 GB RAM, Ubuntu 26.04, vLLM, W4A16-AutoRound-fast quantization.