A new paper called DSpark came out recently (From the DeepSeek team,), and I wanted to understand what it adds beyond regular speculative decoding. The basic trick is to draft several tokens cheaply, then verify them with the big model in one pass. DSpark builds on that with two ideas: better long drafts, and a smarter way to decide how many drafted tokens to verify in production.

1. A full forward pass for every single token

An autoregressive language model produces text one token at a time. To generate token t+1 it needs a forward pass conditioned on everything up to token t.

To see whether that’s a problem, let’s follow one decoding step through the GPU and watch where the time goes. To generate one new token, the model has to produce its logits, the pre-softmax scores over the vocabulary. Getting them means every weight matrix in the model has to travel from the GPU’s main memory into the arithmetic units. That main memory is called HBM (High Bandwidth Memory), and it holds the model’s weights along with the KV cache, the stored attention state. For a 70B model in 16-bit, one token’s worth of logits costs about 140 GB of weight reads.

The arithmetic done with all that data is tiny by comparison: each weight participates in roughly one multiply-add per token. An H100 does on the order of a thousand trillion multiply-adds per second but reads only about 3 TB/s out of HBM. So the decoding step is memory-bound: its wall-clock time is set by how long the weight reads take, not by the math. The opposite regime, where the arithmetic itself is what you wait on, is called compute-bound. Decoding is nowhere near it, and that gap is the whole opportunity. In the memory-bound regime the arithmetic units are mostly idle: the multipliers finish their work instantly and then sit there while the next slab of weights streams in.

The loophole is that the weight reads are paid per forward pass, not per token. Push eight token positions through the same pass and the weights are read from HBM once, then multiplied against eight vectors instead of one. The memory traffic barely changes. You do eight times the arithmetic, but arithmetic was the idle resource anyway, so a pass over eight positions takes barely longer than a pass over one.

You already know this effect as the prefill/decode gap. Prefill is when the model ingests your prompt. All the prompt tokens are known up front, so they go through the model together, thousands of positions per pass, and the weight-read cost is split across all of them. Decoding generates one new token per pass and pays the full cost every single time.

Decoding can’t batch like prefill because the input at position t+1 is the token sampled at position t; it doesn’t exist until the previous pass finishes. But suppose someone handed you a guess for the next eight tokens. Now you do have all eight inputs, tentatively, and the big model can process them prefill-style in one pass. And the same pass gives the model its own next-token distribution at each of the eight positions. Those distributions are all it needs to go through the guess position by position and find where it stops being right. Verification is prefill-shaped. Generation is decode-shaped.

Who writes the guess? A smaller, cheaper model. That’s speculative decoding: everything in this post converts slow generation into fast verification plus a cheap guess.

2. Draft cheap, verify in parallel

Speculative decoding runs as a loop: draft a few tokens, verify them, keep the good prefix, repeat. The version everyone uses is due to Leviathan et al. and Chen et al. A lightweight draft model proposes a block of γ candidate tokens. The target model runs one forward pass over the whole block and gets its own next-token distribution at every position. Then it walks the block left to right and decides, position by position, whether it agrees with the draft.

Say the last confirmed token is D. Call it the anchor: the confirmed token this round’s draft hangs off. The draft proposes E F G H. The target might accept E and F and reject G. A rejection isn’t wasted work: the target already computed a distribution at G’s position, so it samples a replacement G* from a corrected version of it. G* is called the correction token. Everything after the first rejection (H) is thrown away, because it was conditioned on a token that never happened.

If every draft token is accepted, the round earns one extra token for free: the pass also computed the target’s distribution at the position after the last draft token, so the target samples from it and appends the bonus token.

The acceptance rule, and why the output is exact

The big claim of speculative decoding is that it’s lossless: the output tokens are distributed exactly as if the target model had generated them alone, temperature and all. This is a real guarantee, not an approximation, and it comes from a rejection-sampling rule. At position k, with draft distribution pᵈ and target distribution pᵗ, the drafted token xₖ is accepted with probability

In other words: when the target assigns the drafted token at least as much probability as the draft did, pᵗ(xₖ) ≥ pᵈ(xₖ), keep it always. When the draft over-sampled it, giving it more probability than the target does, keep it only in proportion pᵗ/pᵈ. On rejection, the replacement is sampled from the residual distribution norm(max(0, pᵗ − pᵈ)): max keeps the tokens where the target wanted more mass than the draft gave them, and norm rescales what’s left to sum to 1.

A nice consequence is that the per-position acceptance probability equals

which says the drafter’s entire job is to stay distributionally close to the target. DSpark reuses this exact quantity later as a free training label. (Why the rule is exact, and why this distance is total variation rather than KL: Appendix A1 and A2.)

3. The latency equation

How much time does it take to produce one token? Speculative decoding pays per round, one draft plus one verification, and gets a variable number of tokens back, so its cost is time-per-round divided by tokens-per-round. Let τ be the average number of tokens you get per round (accepted draft tokens plus the bonus/correction token), T_draft the time to write the draft, and T_verify the time for the target’s verification pass. Then the latency per generated token is

That leaves three ways to go faster. You can draft faster, shrinking T_draft. You can draft better, raising τ by getting the drafter to agree with the target more often and for longer. Or you can verify smarter, and stop spending T_verify on draft tokens that were never going to survive. Most of the literature picks one of the three.

DSpark is unusual in going after two terms at once: τ with an architecture change (Parts 5 and 6), and T_verify with a scheduler (Parts 7 to 9). The second attack only makes sense once you stop thinking about a single user and start thinking about a serving system with hundreds of concurrent requests. But first, the standard menu of drafter designs.

4. Two ways to build a drafter

Autoregressive drafters (EAGLE, DeepSeek’s MTP (multi-token prediction)) are small language models that generate the draft one token at a time, each conditioned on the last. They produce coherent drafts, but the drafting cost scales linearly with the block size, T_draft ∝ γ. To keep the draft cheap they have to stay shallow (EAGLE-style drafters are often a single transformer layer) and keep γ small, which caps both the block length and how much the drafter can know.

Parallel drafters (Medusa, and more recently DFlash) fill in all γ positions in a single forward pass. Feed in the anchor token plus a row of mask tokens, read out logits everywhere at once, diffusion-style. Now T_draft is nearly independent of γ, so you can afford a much deeper drafter (DFlash uses 5 layers where EAGLE uses 1) and much longer blocks (γ = 16 costs about what γ = 4 costs).

DFlash, the parallel drafter DSpark builds on, is worth describing in some detail because DSpark reuses its entire skeleton. The drafter is a small transformer, 5 layers in the paper. Its input for one drafting round is the embedding of the anchor token (the last token the target actually confirmed) followed by mask-token embeddings, placeholders for the positions to be filled in. All positions attend to each other bidirectionally, and one forward pass produces logits for every position in the block. The drafter doesn’t even own an embedding table or an output head. It borrows the target model’s, frozen.

Its central trick is how it sees the conversation, called KV injection. While the target processes the context, hidden states from a few of its layers are saved, projected down into the drafter’s width, and prepended to the keys and values of every drafter layer, so each draft position attends to the target’s own internal representation of everything said so far. All of it is computed once, at prefill, and reused every round. The drafter isn’t understanding the conversation with 5 layers of its own; it’s reading the big model’s notes, which is a big part of why it can imitate a model thousands of times larger. (Mechanics and a diagram: Appendix A3.)

So a parallel drafter is both faster to run and free to be deeper. That sounds like it should win outright, but it doesn’t…

5. Where parallel drafters break: the multi-modal collision

A parallel drafter predicts every position independently, in one shot. Position 3 can’t see what position 2 actually sampled, only the shared context. So when the context admits several equally good continuations, each position hedges across all of them.

Take the context “Sure, “. The target model is happy with either of course or no problem. A parallel drafter, sampling each position from its marginal, can happily emit of problem or no course: fragments of two valid answers stitched into an invalid one. This is the multi-modal collision problem, known since the non-autoregressive machine translation days (Gu et al., 2018). Each token is individually plausible, the combination is garbage, and the target rejects it.

The damage shows up as suffix decay: acceptance falls off rapidly with position in the block. The paper measures this with a nice metric, position-wise conditional acceptance: at position k, only count cases where positions 1…k−1 were all accepted, then ask how often k is accepted too. This strips out the compounding survival penalty and shows the drafter’s raw predictive quality at each depth.

Here are the curves (Qwen3-4B target, chat domain):

Why DFlash still gets the longer accepted length

The curves make Eagle3 look like the better drafter everywhere past position three. And yet, on accepted length, DFlash beats it on nearly every benchmark in the paper. The mismatch is explained by how verification scores a block.

Verification accepts a prefix and discards everything after the first rejection, so the expected haul per round compounds multiplicatively. Write the expected accepted count in terms of the conditional acceptance rates c₁, c₂, …:

Every term contains c₁, so raising c₁ by 10% grows the entire sum by 10%, while raising c₆ only moves the last couple of terms. A rejection at position 1 erases the whole block; a rejection at position 6 erases one or two tokens. Whatever a drafter is good at, being good at position 1 is worth several times more than being good anywhere else.

And position 1 is where the deep parallel drafter has its edge. No draft tokens exist yet at the first position, so both drafter types condition on the same thing, the verified context, and dependency modeling buys nothing there. What decides position 1 is raw capacity, and the latency budgets are lopsided: the autoregressive drafter runs once per position and must stay shallow (Eagle3 is one layer), while the parallel drafter runs once per block and affords five layers plus KV injection.

6. DSpark idea #1: bolt a bigram onto the parallel drafter

The obvious fix for suffix decay is to make the drafter autoregressive again, but then you’re back to T_draft ∝ γ. DSpark adds autoregression only where it costs almost nothing. The expensive part, the 5-layer KV-injected DFlash backbone, runs fully parallel as before: one pass, producing hidden states h₁…hᵧ and base logits U₁…Uᵧ for every position. After that, a small sequential head passes over the block left to right and adds a correction to each position’s logits, and the correction depends on the tokens already sampled before it:

In this formula, Uₖ is what the parallel backbone already gave us: the base logits for position k, one score per candidate token v over the vocabulary (V ≈ 10⁵ tokens), so Uₖ(v) is token v’s score. Bₖ(x₀, x<k, v) is the sequential head’s correction to that score, and it’s the only term allowed to look at the block’s history: the anchor x₀ and the draft tokens x<k already sampled at earlier positions.

The default B is a small Markov head. It throws away everything except the immediately preceding token: the correction becomes B(xₖ−₁, ·) ∈ ℝV, a full vector of logit corrections, one per candidate next token, selected by which token came just before. Written out as a table that’s a V×V matrix, a row per possible previous token and a column per possible next token. In other words, a learned bigram model, stored factorized at rank 256: per position the sequential loop costs one table lookup plus one small matrix-vector product, invisible next to the target’s verification pass. And it is learned with the rest of the drafter rather than counted from a corpus (dimensions and training: Appendix A4).

Back to the collision. The backbone, in parallel, gives position 1 as {of, no} and position 2 as {course, problem}. The sequential loop samples position 1 first and gets of. When it moves to position 2, the bigram bias for “of” boosts course and suppresses problem, so the drafter no longer stitches “of problem” together.

Does one token of memory suffice? The authors also try an RNN head, a small gated recurrence that carries the whole within-block prefix instead of just the last token. It helps only marginally, and only at long blocks. I think that near-null result is the most useful thing in the paper: suffix decay in parallel drafters is mostly a problem of adjacent-token incoherence, not of missing long-range information.

Through KV injection (Part 4), every draft position already sees the target’s representation of the whole conversation. The missing piece is what the neighboring positions sampled, and that doesn’t exist until after the forward pass is over. Only something that runs at sampling time can close that gap, and a bigram is the smallest such thing.

Across Qwen3-4B/8B/14B, DSpark’s macro-average accepted length comes out 27–31% above Eagle3 and 16–18% above DFlash, and a 2-layer DSpark already beats the 5-layer DFlash, so the gain comes from the dependency modeling and not from extra parameters. On the conditional-acceptance curves, DSpark opens where DFlash opens and stays flat where Eagle3 stays flat.

7. The serving problem: draft tokens compete for batch space

Everything so far was a single-user story: one request, one GPU, minimize that one user’s latency. In production the target model is shared. One copy serves hundreds of users at once, and each of its forward passes processes a batch. The batching is done by the serving engine, the software layer that owns the GPU and runs the model; vLLM and SGLang are the open-source examples, and DeepSeek runs its own. Each pass, the engine collects the pending tokens of many requests and pushes them through together: the amortize-the-weight-reads trick from Part 1, applied across users instead of across positions.

But the batch is a shared, finite resource. A small batch keeps the pass memory-bound, so extra tokens ride along at no cost (Part 1). Keep adding tokens though, and at some batch size the arithmetic units stop being the idle party; past that point, every additional token makes the pass measurably slower for everyone in it. This gives every engine a characteristic curve, SPS(B): steps per second as a function of how many tokens the batch holds. It stays flat while the pass is memory-bound and falls once the batch pushes it into compute-bound territory, the two regimes from Part 1.

Every draft token a request submits for verification takes up one token-slot in the batch, and a rejected token wastes its slot. On the flat part of the curve that waste is harmless, so even a 20% long shot is worth checking. On the steep part, the same token contributes 0.2 expected tokens while slowing down the several hundred others sharing the pass, and it isn’t worth the slot.

The right amount of verification also depends on the content. Drafters do well on structured text and badly on open-ended text: on Qwen3-4B the paper measures roughly 5.6 accepted tokens per round on math, 5.1 on code, 3.5 on chat. A single fixed length is therefore wrong in both directions at once, wasting slots on chat requests whose late tokens were never going to be reached and cutting short code requests that could have gone longer.

Before this paper, the DeepSeek’s production system drafted a single token per round, the MTP-1 setup, even though they had already built multi-token drafters (MTP-3, MTP-5). With a static 3-token draft, every request adds 3 tokens to every batch whether the load or the content justifies it, and at production concurrency many of those tokens are late-position chat tokens headed for rejection. The batches grow, every pass slows, and the slowdown across all users outweighs the extra accepted tokens: total throughput drops. Production stayed at the largest setting that never backfired, one token per round, at a price: at γ = 1 a round yields at most two tokens, the draft plus the bonus, so the ceiling is roughly a 2x speedup and everything Part 4 built for long parallel blocks goes unused.

The way out of that dilemma is to stop picking one length for everyone and instead decide per request, per step: given the load right now, how many of this request’s drafted tokens deserve a batch slot? The drafter has already produced its full γ-token block, so that cost is sunk; the only choice is which of those tokens get slots, and whatever is cut is dropped unseen. Deciding needs a per-token estimate of the chance a slot pays off, plus a rule turning those estimates and the SPS curve into lengths.

8. DSpark idea #2: choose verification lengths by expected throughput

The scheduler is the part of the engine (Part 7) that decides, before each forward pass of the target model, what goes into that pass: which requests participate, and (the new degree of freedom here) how many draft tokens each request contributes. It answers Part 7’s question once per step, for the whole batch at once.

For one verification step, there are R active requests, each with a freshly drafted γ-token block, and the scheduler has to pick a verification length ℓᵣ ∈ {0, …, γ} for every request r: verify the first ℓᵣ draft tokens and drop the rest. A choice of lengths is judged by the expected token throughput of the whole system for this step:

SPS(B) is the Part 7 cost curve; the engine measures it once at startup by timing passes at a range of batch sizes and keeps the results as a lookup table. τ is the expected number of output tokens the pass will produce for the whole batch, i.e. Part 3’s per-request τ summed across all R requests. The two +1s are both the anchor: even at ℓᵣ = 0 a request still sends one token through the pass and gets one token back, the bonus or correction from Part 2. Each admitted draft token at position j then adds aᵣ,ⱼ to τ: the probability that it survives verification and actually ends up in the output. Expected tokens per step times steps per second gives expected tokens per second. The one missing part is those survival probabilities (a_r,j)

The confidence head

What does it mean for a slot to pay off? A draft token survives verification if the left-to-right check actually reaches it (every earlier token in the block was accepted) and accepts it too. “Reaches it” is the price of batched verification: the whole block is checked in one pass, but acceptance is prefix-only, so a rejection at position 3 means positions 4 and onward are discarded without ever being checked, however good their tokens were.

To do that, they use a confidence head, which outputs one scalar per position, cₖ, the conditional piece of survival: the probability that position k is accepted given that the check reached it. This is the same conditional acceptance rate the Part 5 curves measured, now predicted per token instead of averaged over a benchmark. The head itself is small. It reads the backbone hidden state plus the Markov embedding of the previous token and runs them through a single linear projection and a sigmoid. Its training label is free: the exact acceptance probability from Part 2 is computable at every training step and used directly as a soft label (Appendix A4).

Because the cₖ are conditionals, the chain rule turns them into full survival probabilities for free:

These aₖ, computed per request, are exactly the aᵣ,ⱼ the objective consumes, with one requirement: the arithmetic only works if the scores are calibrated, meaning a token scored 0.8 really is accepted about 80% of the time. Raw heads are overconfident; per-position temperature scaling shrinks the predicted-versus-observed gap from 3–8% to about 1% (Appendix A5).

Picking all R lengths at once sounds combinatorial, but the survival probabilities have a property that makes it easy. Within a request, aᵣ,ⱼ = cᵣ,₁·cᵣ,₂⋯cᵣ,ⱼ is a product of numbers between 0 and 1, so tacking on another factor can only shrink it: aᵣ,₁ ≥ aᵣ,₂ ≥ … ≥ aᵣ,ᵧ. In other words, surviving to j+1 requires surviving to j first, so it’s the stricter condition.

That ordering is what licenses a greedy algorithm. Extending request r’s length from j−1 to j adds exactly aᵣ,ⱼ to τ and exactly one token to B, a gain and a cost both known in advance. So pour all candidate extensions from all requests into one pool, sort by aᵣ,ⱼ, and admit from the top, recomputing Θ after each admission (a table lookup). The within-request constraint (you can’t verify position 3 without positions 1 and 2) enforces itself: since each request’s survival probabilities are already in decreasing order, its position-2 candidate always sits above its position-3 candidate in the global sort.

When the batch is nearly empty, SPS(B) sits on its flat stretch, admitting another token raises τ almost for free, and the scheduler verifies long: 4–6 tokens per request in the paper’s numbers. As the batch fills, each admitted token drags SPS down, only tokens whose survival probability beats the drag get in, and the budgets shrink. Within one batch, a code request at 0.9 confidence keeps a long verification while a chat request whose confidence collapses by position 2 gets cut there, with no threshold set anywhere.

A version of this that breaks losslessness

Losslessness needs one more condition beyond the accept/reject rule of Part 2, and this one falls on the scheduler: whether token k gets admitted for verification must not depend on the value of token k. The proofs call this non-anticipation. Fixed verification length satisfies it trivially, but a length chosen by the greedy search above can violate it.

The violation runs through the confidence head, which reads the Markov embedding of the previous sampled token: c₂, and therefore a₂, is a function of which x₁ was drawn. The block is fully sampled before scheduling (Part 7), so a₂ is available; the question is whether the scheduler may consult it. Suppose it evaluates the whole admission path and takes the argmax: Θ at lengths 0, 1, 2. The length-2 value used a₂, which used x₁. So the comparison that decides whether x₁ is admitted at all already depends on which x₁ was drawn, and draft tokens that lead to confident continuations get admitted more often than ones that don’t.

For example, with a two-token vocabulary: the target wants {A: 0.7, B: 0.3}, the drafter samples from {A: 0.5, B: 0.5}, and suppose drawing A leads to a high c₂ while drawing B leads to a low one, so the argmax scheduler admits the draft after drawing A and skips it after drawing B. When A is drawn (probability 0.5), it is admitted and accepted with probability min(1, 0.7/0.5) = 1. When B is drawn, nothing is admitted and the target samples fresh from {0.7, 0.3}. Total: P(output = A) = 0.5 + 0.5 × 0.7 = 0.85 instead of 0.7.

The clean algorithm’s solution is an early-stopping break: walk the sorted admission path and stop at the first admission that lowers Θ. Then the decision about position k is finalized before anything computed from xₖ is ever consulted, and the proof goes through.

9. Making it run in a real engine

Inside DeepSeek’s actual engine, the Part 8 algorithm breaks in two places. The first is that the SPS curve isn’t smooth. Kernels are tuned for particular batch shapes, so pass time jumps at tile-size and dispatch boundaries rather than rising gradually. On a jagged curve the early-stopping break turns into a liability: Θ can dip at a cliff and recover just past it, and a search that quits at the first dip strands the system at a local optimum.

The second is about timing. A modern engine assembles the next pass while the current one runs, and with CUDA-graph replay the next batch size has to be fixed before the current pass finishes. The Part 8 scheduler can’t meet that deadline, since it needs the current step’s confidence scores, and those don’t exist until the step is done. Run it synchronously and the GPU stalls between every pair of steps.

The deployed version works around both problems by computing the batch’s capacity a couple of steps ahead of time. The scheduler fixes K, the number of tokens the next pass can afford to verify, using confidence scores from two steps back, so it’s ready before the pass launches. The tokens that actually fill those K slots are still chosen at the last moment from the current step’s real confidence scores, by sorting the live candidates as in Part 8 and keeping the top K.

This design also settles the losslessness question from Part 8, where the danger was that computing the cut-off consulted downstream token values. Here the cut-off is K, fixed from two-step-old data before any of this step’s tokens existed, and the fresh ranking that fills the slots consults only tokens earlier in the block, never token k itself or anything after it. That is exactly what non-anticipation demands, so it holds by construction, the early-stopping break can go, and the search over the jagged curve can run globally to the true maximum. The same design choice that fixed the pipeline stall also restored exactness; my guess is the pipeline constraint came first and the losslessness argument was noticed after the fact.

The live-traffic numbers

Deployed on DeepSeek-V4-Flash and V4-Pro under real user traffic, against the incumbent MTP-1:

The paper’s Figure 8 shows that at moderate concurrency the average verification budget sits at 4–6 tokens per request, and as concurrency rises the scheduler shrinks it, dropping low-confidence tokens before they occupy batch slots.

Next Steps

A few things the paper leaves open. The SPS table is profiled once and indexed by batch size alone, but real per-step cost also depends on the batch’s context-length mix, so a 1-D curve is an approximation. The confidence head is trained on the analytical acceptance labels under teacher-forced prefixes but deployed on sampled ones. That’s a train/inference mismatch. Calibration is fit on a held-out set, and live traffic drifts; whether the head stays calibrated online isn’t reported. And the drafter still burns a full γ-block forward on every request, even hopeless ones. The scheduler prunes verification, not drafting.

10. What to actually remember

The important thing about DSpark is not just that it makes speculative decoding faster. Earlier work mostly asked how to build a better drafter: make it cheaper, deeper, or more accurate. DSpark shows that this is only half the story. In a real serving system, the question is not “how many tokens can I draft?” but “which drafted tokens are worth spending target-model batch capacity on right now?” The paper’s two main ideas answer both sides of that question: a semi-autoregressive Markov head makes long parallel drafts much more coherent, and a throughput-aware scheduler decides how much of each draft to verify under the current load.

A. Appendix: the fine print

Details that back the main text but aren’t needed to follow it.

A1. Why the rejection rule is exact

Check any token v. It reaches the output two ways: drafted and kept, or rejected and resampled. If pᵗ(v) ≤ pᵈ(v), the kept route alone contributes pᵈ(v) · pᵗ(v)/pᵈ(v) = pᵗ(v) and the residual has zero mass on v. If pᵗ(v) > pᵈ(v), the kept route contributes pᵈ(v) and the resample supplies exactly the missing pᵗ(v) − pᵈ(v), because the overall rejection probability equals the residual’s normalizer. Either way: pᵗ(v). The same accounting shows why the replacement can’t simply be drawn from pᵗ: acceptance already gave every token min(pᵗ, pᵈ), so the rejection branch may supply only the shortfall. With pᵗ = {A: 0.7, B: 0.3} and pᵈ = {A: 0.5, B: 0.5}, resampling from the full pᵗ would give P(A) = 0.5 + 0.2 × 0.7 = 0.64 instead of 0.7, tilting the output toward the tokens the draft over-proposed.

A2. The distance is total variation, not KL

The distance in the acceptance formula is total variation, not KL. TV is symmetric, so there is no forward or reverse version of the statement, and Pinsker’s inequality (TV ≤ √(KL/2)) says pushing either KL down also pushes acceptance up. The paper’s distillation loss skips the surrogate entirely and minimizes the L1 term directly, so it is optimizing the acceptance rate itself.

A3. KV injection, in detail

Its central trick is how it sees the conversation, called KV injection. The drafter never reads the context as text, and it never runs its own forward pass over it. Instead it recycles work the target already did. When the target processes the context, every one of its layers produces a hidden state for every token: a vector encoding that token after the layer has mixed in information from everything before it. DFlash saves these hidden states from a handful of target layers; different depths carry different levels of abstraction, from surface features early to semantics late, so the drafter gets a cross-section rather than a single view. For each context token, the saved vectors are concatenated across the chosen layers and passed through a learned projection down into the drafter’s much smaller hidden size, giving one compact context vector per context token. All of this is computed once, during the target’s prefill, and reused on every drafting round after that.

These vectors enter the drafter through attention. Recall the split of roles there: queries ask, keys and values are what gets read. In every layer of the drafter, the context vectors are projected into keys and values and prepended to that layer’s own keys and values, so from a draft position’s point of view they behave like extra tokens it can attend to. The draft positions do the querying; the target’s representations do the answering. Each draft position therefore sees two things at once: the other positions in its block, and the target’s own internal summary of everything said so far, at several depths. The drafter isn’t trying to understand the conversation with 5 layers of its own. It’s reading the big model’s notes, and the notes were written by the very network it’s trying to imitate, which is a big part of why a 5-layer model can predict what a model thousands of times larger will say next.

One construction detail: in the original DFlash the anchor plus γ masks go in and only the mask positions are predicted; DSpark trims this to anchor plus γ−1 masks and predicts at the anchor slot too, which saves a little compute at the same quality.

A4. The Markov head and the confidence head: dimensions and training

At this vocabulary size the bigram table would have ~10¹⁰ entries, so it’s stored factorized: B = W₁W₂, with W₁ ∈ ℝ^{V×256} and W₂ ∈ ℝ^{256×V}. Row xₖ−₁ of W₁ is a 256-dimensional embedding of the previous token (the paper’s Markov embedding, which the confidence head in Part 8 also reads), and multiplying it by W₂ expands it back to a V-dimensional correction. Per position: one table lookup plus one 256-by-V matrix-vector product.

W₁ and W₂ are not counted from corpus statistics. They are learned by gradient descent along with the rest of the drafter, trained to imitate the frozen target’s outputs, with Part 2’s TV distance as part of the loss.

The confidence head’s training label is free. Part 2 showed that the acceptance probability at a position is exactly 1 − ½‖pᵈ − pᵗ‖₁. Training the drafter already computes both distributions: the frozen target runs on the training data to provide supervision, which yields pᵗ at every position, and pᵈ is the drafter’s own output. So the true acceptance probability is available in closed form at every training step, and the confidence head is supervised directly with it as a soft label, without ever simulating an accept/reject decision.

A5. Calibrating the confidence head

A confidence head is calibrated if its numbers mean what they say: of all the tokens it scores 0.8, about 80% should actually be accepted. Simpler schemes don’t need this. A threshold rule like “drop everything below 0.4” only compares scores to a cutoff, so it works as long as better tokens get higher scores, even if every score is inflated. DSpark’s scheduler does arithmetic with the scores. It multiplies them into survival probabilities and sums those into an expected token count for the whole batch, so if the head says 0.9 where the truth is 0.7, the count is wrong and the scheduler ends up maximizing a throughput that doesn’t exist. Neural confidence estimates are, reliably, overconfident in exactly this way.

The fix is temperature scaling with one twist, which the paper calls Sequential Temperature Scaling. Because the scheduler consumes cumulative products, aₖ = c₁⋯cₖ, the calibration targets the products rather than the raw scores: walking left to right, each position gets one temperature, fit on held-out data to shrink the gap between the predicted and the observed acceptance of the cumulative product, with earlier positions held fixed. Temperature scaling only rescales scores, it never reorders them, so the ranking the head learned survives. The average gap between predicted and observed acceptance drops from 3–8% to about 1%.

A6. The empty-system limit

In the limit of an empty system, the maximization says to verify the entire γ-token block, since an admitted token can only add to τ. Light-load speed is therefore capped by γ and by drafter quality rather than by the scheduler, which is part of why Part 6’s fight to keep acceptance high at long blocks matters. The one cost light load doesn’t forgive is T_{draft}, paid every round regardless of what gets verified.

Source: Cheng et al., DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation (DeepSeek-AI & Peking University, 2026). Figures in this post are stylized redraws of the paper’s Figures 1, 2, and 7; all numbers are from the paper. DeepSeek released the DSpark checkpoints for the V4 preview models along with DeepSpec, a training repo where Eagle3, DFlash, and DSpark are all trained on the same data, so the baselines in this comparison are held fixed for once. Background reading: Leviathan et al. and Chen et al. (2023) on speculative sampling; Gu et al. (2018) on non-autoregressive generation and multi-modal collision; Li et al. on the EAGLE series; Chen et al. (2026) on DFlash.

References and links

• Cheng et al. (2026), DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

• Chen et al. (2026), DFlash: Block Diffusion for Flash Speculative Decoding

• Leviathan et al. (2023), Fast Inference from Transformers via Speculative Decoding

• Chen et al. (2023), Accelerating Large Language Model Decoding with Speculative Sampling

• Li et al. (2024), EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

• Li et al. (2025), EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

• Cai et al. (2024), Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

• DeepSeek-AI (2024), DeepSeek-V3 Technical Report — includes the multi-token prediction (MTP) objective.

• Gu et al. (2018), Non-Autoregressive Neural Machine Translation

• Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM.

• Zheng et al. (2023), SGLang: Efficient Execution of Structured Language Model Programs

• DeepSeek-AI DeepSpec repository

A Transformer remembers by keeping everything. Every token it has read stays in the KV cache, and any later token can look back at any earlier one exactly. That is why attention is so good at recall, and why its memory and compute grow with the length of the context.

Linear attention makes the opposite bet. It throws the cache away and keeps a single fixed-size matrix: a running summary that every new token updates and every query reads. Memory stops growing and decoding gets cheap. But a fixed-size summary cannot hold an unbounded number of facts cleanly, so writing something new can disturb what is already stored. Almost all the recent progress here (DeltaNet, Gated DeltaNet, KDA, and now Gated DeltaNet-2) is about making that write more surgical.

This post builds the whole thing from the ground up. You do not need to know any of these models going in; just linear algebra and a rough sense of what attention does. Shape of the argument

1. A fixed-size state is an associative memory built by summing key–value outer products, and reading it is content-addressed lookup.

2. Because it is fixed-size, overlapping keys interfere. That is the one limitation everything else fights.

3. Giving an old key a new value is the hard case. Adding leaves the stale value behind; replacing the matrix destroys every other fact; the delta rule does the surgical thing.

4. The delta rule looks sequential but trains in parallel as one small triangular solve per chunk.

5. Decay, then per-channel decay (KDA), then decoupled erase/write gates (GDN-2) are three refinements that keep that solve intact.

Which memory we mean

“Memory” means three different things in a language model. This post is about one of them.

• The weights. The query/key/value projection matrices and the gates, learned during training and fixed afterward. Long-term knowledge, changed only by more training. Not this.

• The KV cache (softmax). The full list of past keys and values, so any query can look back exactly. Lossless, grows with context, reset each sequence. Linear attention removes this.

• The recurrent state (linear attention). One fixed-size matrix summarizing every token so far. Lossy, fixed size, reset each sequence. This is the memory we mean.

So this is in-context memory: holding the current input within a single forward pass, so token 5,000 can use what token 3 said. New prompt, empty state, nothing saved. It is not retrieval/RAG, not continual learning, not “remembering you across sessions”; it is the job plain attention does with its KV cache, just compressed into a fixed matrix instead of a growing list.

Where the state comes from

Each token has a representation, and three fixed learned matrices turn it into a query, a key, and a value:

The key is a token’s address, what it is about; the value is the content stored there; the query is what the current token is asking for, matched against the keys to decide what to pull out. A token writes itself in as a key–value pair and later reads with a query. The vectors depend on the input, but the three projection matrices are fixed weights, shared across every position and sequence.

Ordinary attention computes each output as a softmax-weighted sum over the past:

The exponential is what forces the cache. The score exp(query · key) does not split into a part that depends only on the query times a part that depends only on the key, so the weight on each value is tied to that specific key, and the normalizer sums over every past key. There is no running summary you can keep instead: you have to store every key–value pair and revisit them for each new query. That is the KV cache: memory grows linearly with sequence length, and producing all outputs scales quadratically with it.

Linear attention drops the softmax and uses a score that factorizes, in the simplest case just the dot product of query and key. Once it factorizes, the sum rearranges:

All of history collapses into one matrix of fixed size (key-dimension by value-dimension), and the query reads it in a single multiply. Memory no longer grows with context and the per-token cost is constant. This fixed state is the object the rest of this post is about; it exists precisely because the softmax is gone.

That fixed size is the appeal and the problem at once. Packing an unbounded history into one matrix is cheap, but it means many facts share the same finite space. The next section shows what that does to a read.

Why a fixed-size memory interferes

We have the fixed-size state. Before writing into it, look at what reading it gives you. Reading is applying a query to the memory:

Each stored value is weighted by how aligned its key is with the query, the dot product of the two. That is content-addressed recall: values whose keys match what you asked for, weighted by the match.

To expose the problem, take the cleanest possible query, one that exactly equals a key you already stored. This is the case that should return its value perfectly, so any mess is the memory’s fault, not a mismatched query. Splitting off that term:

If the stored keys were orthonormal, every cross term would be zero and the read would be clean. They are not. Each nonzero overlap leaks a fraction of some other value into the answer. And here is the structural reason they cannot all be orthogonal: the state is a single matrix, so the key space has only as many dimensions as the key vector is wide, and a space of that dimension holds at most that many mutually orthogonal directions. Store more associations than that and some keys must share directions; even below the limit, random unit keys have small but nonzero overlaps that add up.

As the context carries more associations, the term you want stays about the same size while the leakage is a sum over everything else, so it grows. Signal-to-noise falls with context length: a long document forces many distinct facts to share one fixed box and they smear together. That is why this whole family struggles on long, many-needle retrieval, and why the improvements below all aim at that pressure point.

Why softmax doesn’t have this problem

The leakage is not caused by folding the sum into the state matrix. The summed form and the matrix form are the same number; folding only fixes the size and the cost, not the value. The leakage is already there in the raw dot-product score.

Softmax runs those same dot products through an exponential and normalizes. The exponential sharpens them: the matching key saturates near one and the mismatched keys are crushed toward zero, so the wrong values effectively drop out of the read even when the keys overlap. Same overlaps, clean answer.

But that is exactly the property that cannot be summarized. The exponential of a dot product does not split into a query part times a key part, so there is nothing to precompute: you are forced to keep every key and recompute the exponential against each one, which is the growing cache. So it is an either/or: a sharp score reads cleanly but cannot be folded into a fixed state, while a foldable score gives the fixed state but leaks. Interference is not the cost of compressing; it is the cost of using a score weak enough to be compressible.

Updating a value when a key comes back

As the model reads a sequence, a later token sometimes produces a key close to one an earlier token already wrote, but carrying a different value. The state already holds a binding in that direction, and the new value should take its place. This is the update case, and it is the one ordinary outer-product memory gets wrong.

For example, a passage sets x = 5 and later sets x = 7. Both tokens produce nearly the same key (the direction standing for “the value of x”), but with different values. When a later token reads x, the answer should be 7. Plain addition cannot give that: it never removed the old binding, so the slot holds 5 and 7 at once and the read returns a blend. The same shape shows up whenever a key recurs with a new value: an entity whose state changes (”Alice is in Paris… now Tokyo”), a correction (”blue… actually green”), a form field revised.

Two clarifications, since “update” can mislead. The prompt itself is fixed; the forward pass only reads it left to right, and “update” means a later position’s binding should win over an earlier one. “x = 5” stays in the text; it just should not win the read. And keys are not matched by name: two tokens are “the same key” when their key vectors point in roughly the same direction, so their writes land on the same spot in the state. A repeated mention produces a nearby key, the later write hits that slot, and the read afterward should reflect the new value.

So every write is one of two cases. Add: the key points somewhere new, a fresh fact, which is most tokens; plain accumulation is fine, and that is what vanilla linear attention does. Overwrite: the key lands on a direction already in the state, and the slot has to be updated to the new value, not stacked on top of the old one.

Why not just replace the whole matrix?

Because the state is shared by every association at once. Three ways to write an update to one key, into a memory that also holds a second fact:

• Replace the whole matrix with the new key–value outer product: fixes the target key perfectly and deletes everyone else. Read the second key afterward and you get near zero. You wanted to change one slot and you erased the notebook.

• Add the new outer product: keeps the second fact, but leaves the old binding in place, so reading the target key returns old-plus-new, the stale value smeared into the fresh one.

• Delta: read what the key currently points to, subtract just that, then write the new value. Only the target slot changes; the other fact is untouched.

Add keeps the other fact but smears the target. Replace fixes the target but wipes the other fact. Only the third (read, subtract, write) gets both right. That third option is the delta rule.

The delta rule

The update that does this is the delta rule (Widrow & Hoff, 1960), used for linear attention in DeltaNet (Yang et al., 2024). It writes the new value relative to what is already stored, not absolutely. First read what the memory currently returns for the key:

This is whatever sits in that key’s direction right now. We never have to know in advance whether the key was used before; we just read it back. Then move the slot from that old value toward the target, by a fraction β (the write strength):

The bracket is the gap between the new value and the old one, and adding it back pushes the value stored at that key toward the target. One update covers both cases with no branching: if the key points somewhere new, the read is about zero, the gap is just the new value, and it reduces to a plain add; if the key lands on a direction that already holds a value, the read returns that old value, and the update subtracts it and writes the new value in its place. The memory tells the rule which case it is in.

Multiplying the correction out shows what it does to the whole state:

The second term writes the new value along the key. The first term removes a β fraction of whatever the state held along that key, and only along that key: the projection onto the key direction leaves everything orthogonal to it untouched. That is exactly why, in the previous widget, replacing the whole matrix wiped the bystander but the delta update did not; it only edits that one key’s line of the state.

Why β is not just 1

A write strength of 1 is a hard overwrite: erase the old binding completely, write the new value. So why not use it everywhere? β is produced per token by the model, and two things argue against pinning it to 1. Real keys are not exactly orthogonal, so erasing hard along one key also disturbs neighbors that partly share its direction, and a smaller β makes a gentler edit with less collateral damage. And not every write should fully replace: sometimes the right move is to nudge a value, accumulate evidence, or write weakly under uncertainty. So β between 0 and 1 is a dial: 1 overwrites, 0 leaves the slot alone, in between is a partial move. (In the online-learning view it is a per-step learning rate, and a rate of 1 everywhere is rarely what you want.)

One caveat the next widget makes concrete: the clean overwrite is exact only when the key is orthogonal to the others. When keys overlap, editing along one drags on whatever shares its direction, the same interference from before, now showing up in the write.

Training it in parallel

Training needs every output over the whole sequence at once, then a gradient. Plain linear attention gives them cheaply because the state is a running sum, so the outputs collapse into two matrix multiplies (with a causal mask zeroing the future):

Why this is fast: it is all dense matmuls, and a GPU runs a matmul as thousands of multiply-adds in parallel on its tensor cores, every output position at the same time. Nothing waits for anything else.

The delta rule breaks this. Its erase factor (the one from the operator form above) makes each state genuinely depend on the previous one, so you cannot write the answer as one sum of independent terms. Done literally you process tokens one at a time, each a tiny rank-one update that uses a sliver of the GPU while the rest sits idle.

DeltaNet’s contribution (Yang et al., 2024) was to recover the matmul form by working in chunks. A chunk is a contiguous block of C tokens; a length-L sequence is split into L/C of them. The expensive work happens inside a chunk, all as matmuls, and only a small summary state is passed from one chunk to the next.

The trick: solve for the values that were actually written

Every step adds a rank-one term whose left factor is a key, so the state is always the start state plus one such term per token:

The written value here is not the raw value, but the correction from the delta rule (target minus old value, scaled by β). The keys are known; these written values are the unknowns. The point is that if we can get all of them in a chunk at once, with a single matrix solve instead of a token-by-token walk, the whole chunk becomes parallel matmuls. So we solve for them jointly.

The written value at each step depends on the current read, and that read expands into known keys and earlier written values:

In words: reading a key against the state-so-far is the start-state read, plus every earlier written value weighted by how much its key overlaps the current one. Substituting gives a relation among the written values alone:

Each written value depends only on earlier ones, which makes this a triangular system. Stack the written values into a matrix, collect the pairwise key overlaps into a matrix T, and the whole set of equations becomes a single solve:

Because the matrix being inverted is unit lower-triangular, the inverse is one forward substitution on a small C-by-C matrix. Everything else is dense matmuls: build the overlap matrix from pairwise key dot products, then form the carried state and the outputs:

The sequential token loop is gone, replaced by matmuls plus one small triangular solve. Writing a product of rank-one factors as a single low-rank update this way is a classical move from numerical linear algebra, the WY representation (Bischof & Van Loan, 1985) and its UT-transform variant (Joffrain et al., 2006); DeltaNet borrows it to collapse the chunk into matrix operations.

Why the chunk size is small

The chunk size sets how often the state is handed off: there are L/C chunks, so that many sequential state updates. The two extremes make this concrete. A chunk of one token is the original fully sequential recurrence. A single chunk covering the whole sequence is one handoff, done in one parallel block. (This is the opposite of what it might sound like: a bigger chunk means fewer, larger steps, not more.)

So why not use one giant chunk and be fully parallel? Because the chunk builds and solves a C-by-C matrix, so its cost and memory grow quadratically in the chunk size. At the full length you are back to the quadratic cost of full attention, and the matrix no longer fits in the fast on-chip memory the matmul engine reads from. Too small, and you pay too many sequential steps and underfill each matmul. The kernels use 64.

Adding decay: Gated DeltaNet

Everything up to here is DeltaNet: a fixed-size associative memory, edited by the delta rule, trained in parallel. The last three sections are refinements, each adding expressive power with a small change that leaves the chunk algorithm intact.

The delta rule overwrites one slot at a time but cannot let old context fade on its own. Gated DeltaNet (Yang, Kautz & Hatamizadeh, 2025, arXiv:2412.06464) multiplies the state by a scalar decay before each edit:

Tracking the cumulative product of the decays, an earlier write contributes to a later read scaled by how much decay has accumulated in between. In the chunk algorithm this is just a per-row reweighting of the same matrices plus an extra factor in the causal mask; the triangular solve is unchanged. Decay is close to free to add. What it cannot do is forget different features at different rates, since it is one number.

Decay per channel: KDA

KDA, the linear-attention layer in Kimi Linear (Kimi Team, 2025, arXiv:2510.26692), replaces that single decay with a per-channel decay vector, a different forget rate for every key channel:

Now every channel is scaled differently at every step, which looks like it should break the chunk form. It does not, because of a change of variables: factor the cumulative per-channel decay out of the state, and it cancels from the recurrence, leaving a plain delta product in reweighted key and erase factors:

After this substitution the chunk equations have the same shape as before; only the entries carry the decay factors. KDA buys richer forgetting at no structural cost.

The per-channel rates are not hand-set hyperparameters; they are learned and data-dependent, produced from each token by a small projection (the Gated DeltaNet parameterization, a softplus of a learned linear map passed through an exponential). A per-head term and a per-channel bias set each channel’s baseline forget rate, and the per-token projection pushes that rate up or down, so the model learns both the typical decay profile and how to modulate it on the fly. The active edit, though, is still a single write-strength scalar, which scales both the erase and the write at once.

Splitting the edit: Gated DeltaNet-2

Erasing acts on the key side: which coordinates of the old read to remove. Writing acts on the value side: which coordinates of the new value to keep. These are different axes of the state, so GDN-2 (Hatamizadeh, Choi & Kautz, 2026, arXiv:2605.22791) gives each its own channel-wise gate, an erase gate on the key and a write gate on the value:

Compared with KDA, the write direction is unchanged (the left factor is still the key), but the read it subtracts is now channel-selected by the erase gate, and the value it writes is channel-selected by the write gate.

Forward: same machine

Run the same change of variables, now folding the erase gate into the reweighted factor, and the recurrence is again a plain (now asymmetric) delta product. The chunk pipeline keeps the same shape: the same overlap matrix, the same triangular inverse, the same state and output equations. The only difference is what fills them: the erase gate enters the key-side rows, the write gate the value-side rows, and the overlap matrix is now built from an asymmetric pair rather than a symmetric one.

Backward: one real difference

Training propagates a loss gradient back through the chunk. Write the solve as the triangular inverse applied to the written values. Backprop needs the gradient with respect to that inverse, which accumulates as a product of the incoming gradient with the written values:

The whole question is whether the gate can be pulled out of that inner product. In KDA the written value is a scalar times the value, so it slides straight out, and you can compute the gate-free products once as a matmul and scale afterward:

In GDN-2 the written value is a per-channel product, so the gate sits inside the sum over channels and there is nothing to pull out:

No single number multiplies the whole inner product; the gate reweights each channel before it is summed, so no row or column scaling recovers it from the gate-free version. The erase side has the same issue. The gate therefore has to be folded into the matmul itself, not applied as a scaling after. The forward pass is essentially KDA’s; the backward kernel is the part that must be rewritten to carry both gates inside its accumulation, and that gate-aware backward is the real implementation cost of the split.

Setting both gates to the same scalar recovers KDA exactly; tying the decay to a scalar as well gives Gated DeltaNet; dropping the decay gives the delta rule. Each model is the next with some gate held to a scalar.

Where this nets out

Step back and it is all one idea, taken in stages. Linear attention compresses an unbounded history into a fixed matrix, fast but lossy. The delta rule edits that matrix surgically instead of piling onto it. The chunked triangular solve makes the edit trainable at scale. Decay, per-channel decay, and decoupled erase/write gates each give the edit finer control over what to keep and what to remove, without giving up the fixed-size state or the parallel training. None of them recover the softmax cache's perfect recall; they make the compression smarter.

That is also where the measured gains land. In the Gated DeltaNet-2 paper the improvement over KDA is modest on language modeling but clear on long-context, multi-key retrieval, the regime where many associations are forced to share one fixed state and interference is worst. The ablation is honest about the split: a channel-wise erase gate with a scalar write recovers most of the gain, so the erase side is doing more work than the write side.

This is also why pure linear attention rarely replaces softmax outright. Exact recall is often worth the cost of the growing cache, so most production models stay softmax, and these layers show up where memory and throughput dominate: long context, high-throughput serving, constrained hardware. The common deployment is hybrid: interleave a few full or sliding-window attention layers for exact recall with many cheap linear layers. Recent open-weight models make this concrete. Qwen3-Next and Kimi Linear both stack three linear blocks (a Gated DeltaNet variant) per full-attention block, a 3:1 ratio, and MiniMax-01 mixes lightning (linear) and softmax attention in a similar pattern.

Sources:

• DeltaNet chunkwise algorithm (Yang, Wang, Zhang, Shen, Kim, NeurIPS 2024)

• Gated DeltaNet (Yang, Kautz, Hatamizadeh, ICLR 2025, arXiv:2412.06464);

• KDA / Kimi Linear (Kimi Team, 2025, arXiv:2510.26692);

• Gated DeltaNet-2 (Hatamizadeh, Choi, Kautz, 2026, arXiv:2605.22791).