EAGLE-3 Speculative Decoding in vLLM for Qwen3-30B-A3B-Instruct-2507

TL;DR

The old notebook version of this workflow is no longer the best way to explain EAGLE-3: it is tied to Colab, an older Qwen3-32B pairing, and a one-off benchmark run.
A cleaner current setup is Qwen/Qwen3-30B-A3B-Instruct-2507 plus RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3, served through vLLM with the model-specific eagle3 draft configuration.
The tokenizer and chat-template side still works cleanly in transformers==5.12.1, which makes it a good companion stack for client-side prompt rendering and sanity checks.

Abstract

Speculative decoding is one of the few inference optimizations that can make a large model feel meaningfully faster without changing its output distribution. The original draft behind this post had the right subject, but it had the wrong shape for 2026: a Colab export, a stale verifier-speculator pairing, and a benchmark section that mattered less than the serving path itself. This rewrite keeps the important idea and updates everything else. You will see what EAGLE-3 is doing, why Qwen/Qwen3-30B-A3B-Instruct-2507 is a better current example than the older Qwen3-32B setup, how to launch the matching Red Hat AI speculator in vLLM, and how to sanity-check the tokenizer flow with transformers==5.12.1.

Requirements

Use this setup if you want to follow the post as written:

Python 3.10+
transformers==5.12.1
requests
a current vllm build with EAGLE-3 speculative decoding support

You also need enough GPU memory for the verifier model you choose. Qwen3-30B-A3B-Instruct-2507 is still a serious model even though the speculator itself is small.

Prerequisites

You are comfortable running shell commands and short Python scripts.
You understand the difference between a verifier model and a draft or speculator model.
You have used an OpenAI-compatible endpoint before, or can follow a curl example.
You are testing an interactive or low-concurrency workload where speculative decoding can actually help.

Problem
Background
Approach
Example
Failure Modes & Caveats
Practical Guidance
Summary
Next Steps
Resources

Problem

Notebook exports age badly when the ecosystem around them moves fast.

That is exactly what happened to the original file. It still had the bones of a useful post, but it mixed blog prose with Colab boilerplate, installed everything in one cell, and centered the story on Qwen/Qwen3-32B plus a benchmark snippet instead of the real question a reader cares about:

How do you run a current EAGLE-3 speculator with vLLM in a way that is easy to understand, easy to verify, and still compatible with the modern transformers stack?

That is the problem this rewrite solves.

Background

Speculative decoding uses two models together:

A small draft model proposes several next tokens.
The large verifier model checks those tokens in parallel.
Correct draft tokens are accepted, and the final output stays consistent with normal verifier-only decoding.

The reason people care is simple: large models are usually slow because generation is sequential. If the draft model guesses well, the verifier can accept multiple tokens per pass instead of only one.

EAGLE-3 is the current high-water mark in the EAGLE family because it moves beyond earlier feature-prediction approaches and uses multi-layer fusion with direct token prediction. In practice, the important part is not the paper vocabulary. It is that EAGLE-3 has matured into a deployable path in vLLM, with published draft checkpoints that are specific to particular verifier models.

That model-specific coupling matters. An EAGLE-3 draft model is not a generic accelerator. It is trained to match one verifier family and often one exact verifier checkpoint.

Approach

For this rewrite, the best current compromise between freshness and practical clarity is:

Verifier: Qwen/Qwen3-30B-A3B-Instruct-2507
Speculator: RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3

This pairing is better than the older notebook example for three reasons:

It reflects a more current Qwen release than the original draft.
It has an official EAGLE-3 speculator model card with a direct vLLM serve recipe.
Its tokenizer and chat-template flow still behave cleanly in transformers==5.12.1, which makes the surrounding Python examples easier to trust.

The post keeps the serving path deliberately boring:

Verify the tokenizer and chat template in transformers.
Launch vLLM with the matching eagle3 speculator.
Send one OpenAI-compatible chat request.
Measure whether the workload is actually the kind that benefits from speculative decoding.

Example

1. Install a current stack

Keep the client-side dependencies lean:

python -m pip install "transformers==5.12.1" requests vllm

If you already manage vllm separately, the main thing to preserve from this post is the transformers==5.12.1 tokenizer path and the exact verifier-speculator pairing.

2. Sanity-check the tokenizer path in Transformers

This is a small check, but it is worth doing because it proves the tutorial is aligned with the current transformers API instead of only pasting model-card commands.

from __future__ import annotations

from typing import Final

from transformers import AutoTokenizer


MODEL_ID: Final[str] = "Qwen/Qwen3-30B-A3B-Instruct-2507"
PROMPT: Final[str] = "Explain speculative decoding in one sentence."


def main() -> None:
    """Render a Qwen chat prompt with the current Transformers stack."""
    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
    messages = [{"role": "user", "content": PROMPT}]
    rendered = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    print(type(tokenizer).__name__)
    print(rendered)


if __name__ == "__main__":
    main()

On a healthy setup, the rendered prompt should use the Qwen chat markers:

<|im_start|>user
your prompt text
<|im_start|>assistant

That is the exact behavior I wanted to preserve from the old Qwen-based post while moving the model choice forward.

3. Launch vLLM with the EAGLE-3 speculator

The current long-form command is:

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  -tp 1 \
  --speculative-config '{
    "model": "RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3",
    "num_speculative_tokens": 3,
    "method": "eagle3"
  }'

This example sticks to the published default of 3 speculative tokens because that is the safest place to start. The speculator is small, but the verifier is not, so adjust tensor parallelism to match your hardware rather than copying -tp 1 blindly.

If your vLLM build supports model-embedded speculators_config, there is also a shorter path:

vllm serve RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3

That shorthand is convenient, but the long-form command is better for learning because it makes the verifier-speculator relationship explicit.

4. Send a quick chat request

Once the server is up, test it with a predictable prompt rather than a creative one:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-Instruct-2507",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain why speculative decoding helps code generation."}
    ],
    "max_tokens": 192,
    "temperature": 0.2,
    "stream": false
  }'

You can also call the same endpoint from Python:

from __future__ import annotations

from typing import Final

import requests


API_URL: Final[str] = "http://localhost:8000/v1/chat/completions"
MODEL_NAME: Final[str] = "Qwen/Qwen3-30B-A3B-Instruct-2507"


def main() -> None:
    """Send one chat request to a local vLLM server."""
    payload = {
        "model": MODEL_NAME,
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {
                "role": "user",
                "content": "Explain why speculative decoding helps code generation.",
            },
        ],
        "max_tokens": 192,
        "temperature": 0.2,
        "stream": False,
    }
    response = requests.post(API_URL, json=payload, timeout=120)
    response.raise_for_status()
    body = response.json()
    print(body["choices"][0]["message"]["content"])


if __name__ == "__main__":
    main()

This client code is intentionally plain. The interesting part of the tutorial is the server-side acceleration, not the SDK wrapper.

Failure Modes & Caveats

Speculative decoding is useful, but it is not universal.

It works best when the workload is predictable

Code generation, structured outputs, templated responses, and low-temperature Q&A are usually good fits. Free-form creative writing is often a worse fit because the draft model has a harder time guessing the next token sequence well.

High concurrency can erase the benefit

Speculative decoding often shines in interactive or low-concurrency serving. If the GPU is already saturated with many concurrent requests, the extra draft-model work can stop helping.

The draft model must match the verifier

This is the mistake most new readers make. You cannot casually mix an EAGLE-3 speculator trained for one model with a different verifier. The pairing in this post is specific:

Qwen/Qwen3-30B-A3B-Instruct-2507
RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3

Faster does not mean cheaper if you measure the wrong thing

Time to first token can move differently from total throughput. Measure both. A setup that feels better for a long response may still have slightly higher startup overhead.

The verifier is still the expensive model

The speculator is only the helper. You do not get to ignore memory planning just because the draft checkpoint is small.

Practical Guidance

If you want this setup to be useful outside a toy demo, start here:

Keep num_speculative_tokens=3 for your first run, because that is the published default for this speculator.
Use a predictable prompt family first, such as code, extraction, JSON-like formatting, or constrained Q&A.
Compare speculative and non-speculative runs on the same prompt set before drawing conclusions.
Watch tokens per second, time to first token, and time per output token together.
If acceptance is consistently high, experiment with a slightly larger speculative token count.
If your real workload is mostly creative writing or large offline batches, do not assume speculative decoding will help.

There is also a broader 2026 lesson here. The newest published verifier-speculator pairing overall is not always the best tutorial choice. For example, Gemma 4 speculators are newer, but the updated Qwen path above is easier to explain cleanly and stays closer to the original Qwen-based draft while still being current.

Sanity check

Before you benchmark anything, verify that:

transformers.__version__ is 5.12.1
AutoTokenizer.from_pretrained("Qwen/Qwen3-30B-A3B-Instruct-2507") loads successfully
apply_chat_template(...) renders the Qwen chat markers
vllm serve ... --speculative-config ... starts without model-pairing errors
/v1/chat/completions returns a normal assistant message

Summary

The original notebook had a useful topic but an outdated delivery mechanism. A better 2026 version is a real post built around a current verifier-speculator pair, a current tokenizer check, and a serving example that makes the moving parts obvious.

If you want a practical EAGLE-3 starting point today, Qwen/Qwen3-30B-A3B-Instruct-2507 plus RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3 is a cleaner choice than the old Qwen3-32B example. It keeps the Qwen workflow readable, works cleanly with transformers==5.12.1 on the client side, and maps directly onto the current vLLM speculative decoding story.

Next Steps

Run the same prompt set once with standard vLLM and once with the EAGLE-3 speculator enabled.
Measure whether your actual workload looks more like code or structured output than creative generation.
Try a larger tensor-parallel setting if your hardware requires it for the verifier.