Qwen3.6 35B A3B

Unsloth's UD-Q4_K_XL is 22.4 GB, which leaves almost nothing for context on a 24 GB card. Plan on 32 GB of VRAM or unified memory, or offload some experts to system RAM.

Running it

32 GB GPU or 32 GB Mac at 4-bit. On 24 GB, offload some experts to system RAM and accept slower generation. See the hardware sizing tables for how that maps to specific chips and cards, and the quantisation guide for what you give up at each bit width.

Which cloud instance actually runs this, and what it costs

At Q4_K_M the weights are about 22.4 GB, so 14 catalogued instances fit with room for context, of which the 8 best value are shown. It is a mixture-of-experts model: all 22.4 GB must sit in memory, but each token reads only about 1.7 GB of active experts. The speeds below assume every weight is read, so treat them as a conservative floor and the costs as a ceiling. Measured single-user speeds for this kind of model run several times higher: about 100 tok/s for gpt-oss-120b on one A100 under vLLM, and around 240 tok/s for Qwen3.6 35B-A3B with multi-token prediction on an RTX 6000 GPU in Unsloth's tests. Cheapest per token is RunPod A100 PCIe (~$3.83 per million output tokens at an estimated 86 tok/s). Fastest on a single card is Azure NCads H100 v5 at roughly 150 tok/s for $12.96 per million tokens, which is the speed-versus-cost trade in one line.

Token generation on a single request is memory-bandwidth-bound, not compute-bound: the GPU must read every active weight from memory for each token it emits. So the ceiling is roughly GPU memory bandwidth divided by the size of the active weights. A 16 GB model on a 300 GB/s L4 tops out near 19 tokens/sec; the same model on a 2039 GB/s A100 tops out near 127. Figures on model pages are that arithmetic, computed from the bandwidth and weight-size columns here. They are a ceiling for one request at batch size 1, before serving overhead, so treat them as an upper bound for comparing instances rather than a benchmark. Batching raises total throughput well above this and does not raise per-request speed.

For identical silicon, the hyperscalers charge roughly 2 to 5 times what the specialist GPU clouds charge. An A100 80GB is about $3.43/GPU-hr on AWS, $3.67 on Azure and $5.03 on Google Cloud, against $1.19-1.39 on RunPod and $1.99 on Lambda. That spread is the single largest cost lever in a self-hosted inference budget, and it is larger than any saving from picking a smaller model.

Three costs that are absent from every headline rate: egress, storage for the weights, and idle time. A 70B model at Q4 is tens of gigabytes that must be pulled to the instance on every cold start, and an endpoint billed hourly is billed while it sits idle waiting for a request. Rates read 2026-09-06; check each provider's live pricing on the directory hub.

Jurisdiction: China

This is the crucial split most write-ups miss: using a Chinese lab's HOSTED API sends your data to Chinese infrastructure and is a real procurement question. Downloading their OPEN WEIGHTS and running them on your own hardware, or on a Western host, sends nothing anywhere. DeepSeek and Z.ai publish under MIT; Alibaba publishes most of Qwen3 under Apache 2.0. Those are among the most permissive licences on this page.

Watch for: Several US states and a number of government bodies restrict Chinese-hosted AI services on official devices. That restriction is about the hosted service, not about the weights running in your own VPC — but expect to have to explain the difference to a security reviewer.

Frequently asked

Similar models