Speculative decoding is one of those ideas that sounds too good to be true: use a small, fast draft model to guess several tokens ahead, then let the big model verify them all in one pass. When the guesses are right — and they often are — you get the big model’s quality at a fraction of the latency. The catch that keeps practitioners up at night is subtle nondeterminism: most implementations can produce output that differs from what the model would have generated token-by-token, which makes debugging, testing, and compliance-minded deployments miserable.
TensorFold, the MIT-licensed open-source inference engine from developer Ash Hart, has decided that catch is unacceptable. Its headline feature in the v0.6.x releases (v0.6.0 and v0.6.1 landed around October 1, after a rapid run from v0.4.0 through v0.6.1 in the space of about a week spanning late September) is exact speculative decoding — drafted output that is byte-identical to plain serial decoding. Row-exact kernels, keyed sampling, and a per-reply token_sha (plus min_rows) let you verify the guarantee rather than take it on faith. For anyone who has ever diffed two “identical” inference runs and found mystery divergences, that is a genuinely appealing promise.
How exact speculative decoding works
The idea is easiest to grasp with an analogy. Imagine a fast junior engineer drafting replies to customer emails, and a senior engineer reviewing each draft: accepting it wholesale when it is right, correcting it when it is not. The senior never does less work per email, but the pair clears the queue faster because most drafts are fine. In speculative decoding, the “junior” is a small, cheap draft model proposing several tokens at once, and the “senior” is the full model verifying them in a single parallel pass.
The subtlety TensorFold obsesses over is what happens at the boundary. Most implementations sample the draft and the verifier slightly differently — different random seeds, different numerical paths — so the accepted output can diverge from what the big model would have produced alone. Usually the difference is harmless. But “usually” is not a property you can put in a compliance document. TensorFold’s row-exact kernels and keyed sampling are designed to make the draft path and the serial path bit-for-bit identical, with token_sha giving you a fingerprint to prove it. It is the difference between “trust us, it’s the same” and “here’s the hash.”
Local-first, both flavors of silicon
TensorFold positions itself as the inference engine for people who own their hardware. It targets Apple Silicon via Metal/MLX and NVIDIA GPUs via CUDA, and speaks the OpenAI-compatible API dialect (/v1/chat/completions), so existing tooling mostly just works. The project’s documentation even ships runbooks written for AI agents — a nice touch that acknowledges how much inference infrastructure is now provisioned, configured, and debugged by other models.
The performance claims
Hart reports 1.6× to 3.1× speedups versus vLLM with multi-token prediction (MTP=3) on NVIDIA DGX Spark hardware: Qwen3.8-27B with DFlash2 at 2.70–3.05× on a single Spark, and GLM-5.3-Flash at 1.78–2.06× on two. On the Apple side, a Mac with M5 Max and 128GB of unified memory reportedly serves Nemotron 3.5 Lightning 30B at 188–206 tokens per second, against 138 for mlx_lm.
Now the asterisks, because there are several. These are the developer’s own benchmarks, not independent measurements. Model coverage is narrow — about four families (Nemotron 3.5 Lightning 30B-A3B, Qwen3.8 Flash Next, GLM-5.3-Flash, Qwen3.5-2B) — and quantized GPTQ/AWQ formats are explicitly refused rather than supported. A 3× speedup on one model family is a data point, not a law of nature. Treat the numbers as a reason to try it, not a reason to migrate.
Who is this for
The target user is easy to picture: the developer with a Mac Studio or a DGX Spark on their desk who is tired of choosing between vLLM’s breadth and MLX’s Apple Silicon polish. TensorFold is narrower than vLLM — four model families versus hundreds — but depth has its own appeal. If your workload is one of the supported models and you value reproducibility, the pitch is compelling. If you live in quantized GGUFs or need forty architectures, it is not your tool yet. The honest framing is that TensorFold is a specialist instrument, not a Swiss Army knife — and specialists are allowed to be opinionated. Refusing GPTQ/AWQ rather than supporting them badly is, in its way, a sign of engineering seriousness.
Why “exact” matters more than “fast”
Here is the thing the benchmark tables undersell. In regulated or high-stakes deployments — finance, legal, medical — reproducibility is a feature, not a nicety. If your speculative decoder can silently produce different tokens than serial decoding, every output becomes probabilistically suspect, and your eval harness has to account for sampler noise on top of model noise. TensorFold’s byte-identical guarantee collapses that problem: the fast path and the slow path are the same path, observably. That is arguably a bigger deal than the tokens-per-second figure, and it pairs naturally with the broader efficiency conversation happening in research right now — see, for example, the recent “Don’t Drop Dropout” paper on squeezing more out of every training and inference FLOP.
Try it in five minutes
Installation is a single pip command against the GitHub repo:
pip install git+https://github.com/ashhart/TensorFold.git
tensorfold serve <model> --context 65536
curl localhost:8080/v1/chat/completions -d '{...}'The 64k context window in that example is representative, not maximal — check the repo for current limits per model. The project is young, the release cadence is blistering (four minor versions in a week suggests either admirable velocity or a test suite playing catch-up), and the exactness guarantee is the kind of claim the community will verify quickly and loudly. That verifiability is precisely the point. In a corner of the ecosystem where benchmarks are too often vibes with error bars, TensorFold has made a falsifiable promise. Now we get to watch people try to falsify it.
One more thing worth watching: the release cadence. Four minor versions in about a week is either exhilarating or alarming depending on your temperament. For a young systems project it usually means the author is dogfooding hard and fixing what breaks — the AI-agent runbooks in the repo suggest exactly that workflow. The risk is API churn; the reward is a tool shaped by real use. Pin your install to a version, read the changelog, and enjoy the show.
Featured image: a bare NVIDIA GeForce GTX 780 circuit board, up close (GBPublic_PR, via Wikimedia Commons — CC BY 2.0).
