Slow per stream. Linear in aggregate.
Every number on this page comes from a dated run on our own ten-PC fleet, with the failures left in. One stream on a 106B model decodes at about one token per second. Add streams and the fleet's total output grows almost one for one, until the machines run out of RAM. That boundary is also published here.
A 106B model, ten ordinary PCs, eight people at once.
GLM-4.5-Air (106B parameters, Q5_K_M, 83.5 GB) pooled across ten consumer machines. No single machine holds the model. Three waves of two, four and eight concurrent graded tasks: factual questions, exact arithmetic, and tool calls that had to return schema-exact JSON.
Method: ten Windows and Linux desktops and laptops, 8 to 16 GB of RAM each, on a home LAN. Tasks were short-answer by design, so this run proves admission and correctness under concurrency, not sustained long-form throughput. That stress is listed under "Not yet measured" below.
Aggregate scales with streams. RAM sets the ceiling.
The same 106B model at two quantizations. Per-stream speed barely moves with load. Aggregate throughput grows linearly until the fleet's memory runs out, which is a hardware limit, and the reason the next step is more machines.
| Date | Quant · fleet | Streams | Per stream | Aggregate | Result |
|---|---|---|---|---|---|
| 13 Jul | IQ4_XS · 8 PCs | 1 | 1.33 tok/s | 1.33 tok/s | 24 of 24 jobs completed in a 35-minute unattended soak |
| 15 Jul | IQ4_XS · 6 PCs | 1 | 1.09 tok/s | 1.09 tok/s | Single-stream floor |
| 15 Jul | IQ4_XS · 6 PCs | 4 | 1.12 tok/s | 4.49 tok/s | 4.1× · linear |
| 15 Jul | IQ4_XS · 6 PCs | 8 | — | — | RAM wall three boxes under 4 GB free, thinnest box froze |
| 15 Jul | Q5_K_M · 10 PCs | 4 | 0.55 tok/s | 1.8 tok/s | All four long jobs completed, zero freezes. Q5 is 2.5× slower than IQ4. |
| 15 Jul | Q5_K_M · 10 PCs | 8 | short tasks | — | 16 / 16 pass the collapse did not recur on ten machines |
| 16 Jul | Q5_K_M · 10 PCs + speculation | 1 | 0.95 tok/s | 0.95 tok/s | n-gram drafts, 41% accepted, 1.56× fewer ring passes, output unchanged |
Per-stream is one request's decode rate. Aggregate is every active stream summed. IQ4_XS and Q5_K_M are 4-bit and 5-bit weight formats of the same model; Q5 is higher quality and 38% larger, which is what pushed the fleet to its memory limit. Prefill (reading the prompt) is excluded; a long-context prompt of about 800 tokens took around 185 s to read on this fleet.
IQ4_XS aggregate by concurrent streams.
Where the 83.5 GB lived.
The model's 47 layers were split across ten machines by available memory. The largest share was 14 GB; the smallest was under 1 GB on the head PC, which also runs the coordinator. Every machine was a consumer desktop or laptop already in use for other work.
| Machine | OS | Layers | Shard on disk |
|---|---|---|---|
| PC 01 · head and driver | Windows | 1 | 0.6 GB |
| PC 02 · laptop | Windows | 8 | 14.2 GB |
| PC 03 · desktop | Windows | 5 | 8.9 GB |
| PC 04 · mini PC | Windows | 5 | 8.9 GB |
| PC 05 · mini PC | Linux | 8 | 13.9 GB |
| PC 06 · mini PC | Windows | 7 | 12.1 GB |
| PC 07 · mini PC | Windows | 2 | 3.6 GB |
| PC 08 · mini PC | Windows | 7 | 12.6 GB |
| PC 09 · mini PC | Windows | 2 | 3.8 GB |
| PC 10 · mini PC | Windows | 2 | 5.2 GB |
Each machine downloaded only the byte ranges for its own layers, so no machine ever held the whole model on disk or in memory. Sizes are the on-disk shards after the final placement; machine identities are withheld, the hardware class is not.
Three 24-prompt batteries, same fleet, same day.
Questions, reasoning, code, strict output formats, long context and tool calls. The first battery exposed the real problem: the model often had the right answer inside its thinking and hit the token cap before saying it. Turning thinking off fixed most of it.
| Battery | Answered cleanly | Cut off at cap | Never left thinking | Repetitive | Decode tok/s |
|---|---|---|---|---|---|
| Greedy baseline | 13 / 24 | 11 / 24 | 5 / 24 | 2 / 24 | 1.33 |
| Sampled, t=0.6 | 11 / 24 | 13 / 24 | 6 / 24 | 6 / 24 | 1.04 |
| Sampled, thinking off | 21 / 24 | 3 / 24 | 1 / 24 | 1 / 24 | 1.27 |
Where it was already right
Strict-format prompts (JSON only, single word, CSV) were 5 of 5 exact on the first battery, and both tool-call prompts emitted well-formed JSON calls. Facts that escaped the thinking phase were correct.
What the fix was
Not a model change. Thinking is suppressed at the right position in the prompt template and the output cap was raised from 768 to 2048 tokens. The 16 of 16 headline run two days later was the first with both in place.
Raw runs: greedy-baseline-20260713-213015.jsonl · sampled-t0.6-20260713-223308.jsonl · sampled-nothink-20260713-233231.jsonl · 24 jobs each, unattended, ring never dropped.
The three levers, measured separately before they were combined.
Overlap · ~3×
With a stock pipeline, machines take turns and only one computes at a time: four users shared 0.73 tok/s on a 70B model. Keeping every stage busy at once gave 3.2 → 25.8 tok/s on a small 3B test model across three machines.
Batching · 4 to 5.7×
Each machine was benchmarked alone: pushing several sequences through one pass of the weights multiplied decode output 5.2× on one box and 5.7× on another at 16 sequences, flattening after that.
Combined · 16×
Same three machines, same 3B model, both levers on: 3.2 tok/s single stream to 51.0 tok/s aggregate at 32 streams, 10 July 2026. The 106B curve above is these levers on the real model.
The 70B reference was Llama 3.3 70B split across three machines, 0.73 tok/s single stream and stable for over eight hours. The small-model figures are a testbed, not a product claim; they are here because they show where the aggregate scaling comes from.
What we have not run, so we do not claim it.
Match the tool to the workload.
Fits well
Fleets of agents working in parallel, overnight batch processing of documents, bulk extraction and classification, and pipelines where total throughput matters more than any one answer's latency.
Fits poorly
A single user waiting on a fast interactive chat. One isolated request runs at about one token per second on a model this size; that is the trade for running many at once on hardware you already own.
How to size it
Plan around concurrent streams and memory per machine, not per-stream speed. Tell us your desktop count and model size and we will show you the curve it supports.