Compress Qwen3.5 with AutoRound in Transformers 5.12.1

TL;DR

Use transformers==5.12.1 and auto-round==0.13.1 for a current Qwen3.5 + AutoRound stack.
For W4A16, the current default recommendation is auto-round, not auto-round-best.
Start with Qwen/Qwen3.5-0.8B to validate the flow, then scale the same recipe to Qwen/Qwen3.5-4B or Qwen/Qwen3.5-35B-A3B.

Abstract

The original notebook version of this workflow had the right instinct but the wrong shape for 2026. It installed transformers and AutoRound directly from GitHub, used auto-round-best as the default path, and mixed a few examples that are no longer the cleanest way to explain Qwen3.5 quantization. In this post, you will get a tighter version: a stable install path, a W4A16 recipe that matches current AutoRound guidance, a small model you can use to validate the workflow first, and a plain transformers loading path for the quantized output.

Requirements

If you want the exact stack targeted by this post, use:

Python 3.10+
torch==2.12.1
transformers==5.12.1
auto-round==0.13.1

Optional speedups for quantization:

flash-linear-attention
causal-conv1d

Prerequisites

You are comfortable running shell commands and Python scripts.
You understand the difference between model quantization and model serving.
You have enough local disk and RAM for the checkpoint you choose.

Problem
Background
Approach
Example
Failure Modes & Caveats
Practical Guidance
Summary
Next Steps
Resources

Problem

Notebook-style quantization guides tend to age badly. They often pin a random Git commit, assume a fast-moving nightly package, and bury the one decision that actually matters: which combination of model, quantization scheme, and export format gives you a reproducible result today.

That is exactly what happened here. The old file still reflected a moment when installing transformers from main and using auto-round-best everywhere felt like the safest move. Today, that is more complexity than you need for a standard W4A16 Qwen3.5 workflow.

The real problem is not “How do I quantize the biggest checkpoint I can find?” It is “What is the smallest, most boring path that still proves the workflow is correct?” Once you answer that well, scaling up is mostly a resource problem.

Background

AutoRound is a post-training quantization toolkit that can export multiple formats for downstream runtimes. That distinction matters:

auto_round is the format to prefer when your next step is loading the quantized model back into transformers.
llm_compressor is the format AutoRound recommends for formats like NVFP4.
auto-round-best is the highest-accuracy recipe, but it is slower and not the default recommendation for routine W4A16 usage.

Qwen3.5 also has a mixed family shape. Some checkpoints are straightforward to use as a practical demo, while others deserve extra caution because architecture details can complicate quantization behavior. That is why this post uses Qwen/Qwen3.5-0.8B as the runnable starting point, then shows how to scale the same command to larger checkpoints once the path is stable.

Approach

The clean workflow is:

Install stable package versions instead of Git main.
Quantize a small Qwen3.5 checkpoint with auto-round and W4A16.
Load the exported model back with transformers.
Only then scale to 4B or 35B-A3B.

Two updates are doing most of the work here:

For W4A16, use auto-round as the default recipe.
Keep the first export in auto_round format if your goal is plain transformers inference.

Example

1. Install a current stack

uv pip install "torch==2.12.1" "transformers==5.12.1" "auto-round==0.13.1"

If you want the optional quantization speedups:

uv pip install flash-linear-attention causal-conv1d

2. Quantize a small Qwen3.5 checkpoint first

This is the boring path on purpose:

auto-round \
  --model Qwen/Qwen3.5-0.8B \
  --scheme "W4A16" \
  --format "auto_round" \
  --output_dir ./Qwen3.5-0.8B-W4A16

This command is a better default than the old notebook for two reasons:

it uses the stable package release instead of a Git install, and
it follows current AutoRound guidance for W4A16.

3. Scale the same recipe to larger checkpoints

Once the 0.8B path works, the same structure carries over:

auto-round \
  --model Qwen/Qwen3.5-4B \
  --scheme "W4A16" \
  --format "auto_round" \
  --output_dir ./Qwen3.5-4B-W4A16

auto-round \
  --model Qwen/Qwen3.5-35B-A3B \
  --scheme "W4A16" \
  --format "auto_round" \
  --output_dir ./Qwen3.5-35B-A3B-W4A16

The important thing to notice is what did not change. You are not switching recipes, inventing special flags, or reaching for a nightly transformers build just because the model got bigger.

4. Load the exported checkpoint with Transformers

Use the quantized output directory like a normal model path:

from __future__ import annotations

from typing import Final

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


MODEL_DIR: Final[str] = "./Qwen3.5-0.8B-W4A16"
PROMPT: Final[str] = "Explain weight quantization in one short paragraph."
MAX_NEW_TOKENS: Final[int] = 128


def build_prompt(tokenizer, user_text: str) -> str:
    """Render a chat-style prompt for Qwen3.5."""
    messages = [{"role": "user", "content": user_text}]
    return tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )


def main() -> None:
    """Load a quantized Qwen3.5 checkpoint and generate one reply."""
    tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_DIR,
        device_map="auto",
        torch_dtype="auto",
    )

    prompt = build_prompt(tokenizer, PROMPT)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        generated = model.generate(
            **inputs,
            max_new_tokens=MAX_NEW_TOKENS,
            do_sample=False,
        )

    new_tokens = generated[0, inputs.input_ids.shape[1] :]
    print(tokenizer.decode(new_tokens, skip_special_tokens=True))


if __name__ == "__main__":
    main()

If you want to load an already-published AutoRound checkpoint from the Hub, transformers 5.12.1 also exposes AutoRoundConfig:

from transformers import AutoModelForCausalLM, AutoRoundConfig, AutoTokenizer


MODEL_ID = "Intel/Qwen3.5-4B-int4-AutoRound"

quantization_config = AutoRoundConfig()
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="auto",
    torch_dtype="auto",
    quantization_config=quantization_config,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

That matters because it shows the current stack is not just “AutoRound as an external tool.” The quantized-model loading story now reaches directly into transformers as well.

Failure Modes & Caveats

This is the section that keeps the tutorial honest.

`auto-round-best` is not the default W4A16 path

The old notebook used auto-round-best for routine W4A16 examples. Current AutoRound guidance is more specific: use auto-round for standard W4A16 work, and reserve the more expensive recipes for cases where you actually need them.

`NVFP4` is not the same workflow as `auto_round`

The old file ended with an NVFP4 example:

auto-round-light \
  --model Qwen/Qwen3.5-9B \
  --scheme "NVFP4" \
  --format "llm_compressor" \
  --output_dir ./Qwen3.5-9B-NVFP4

That can still be a valid export path, but it is solving a different problem. If your goal is “quantize Qwen3.5 and reload it in plain transformers,” keep the main tutorial on W4A16 plus auto_round format.

Treat `Qwen3.5-9B` as a caveat-heavy path

There are recent AutoRound discussions around dense Qwen3.5 variants with DeltaNet layers, including reports of some tensors staying in higher precision during export. That does not mean the model is unusable. It means you should not use Qwen3.5-9B as the first example in a tutorial that is supposed to feel predictable.

Optional packages are optional

flash-linear-attention and causal-conv1d can help quantization speed. They are not the conceptual heart of the workflow. Keep them optional in your setup instructions unless performance is the main point of the post.

Do not reshuffle the loaded quantized model casually

AutoRound’s own guidance warns against manually moving a quantized model across devices after load, for example with model.to("cpu"). Let the initial loading configuration decide placement.

Practical Guidance

If you are using this workflow in real life, keep the process boring:

Validate the stack with Qwen/Qwen3.5-0.8B first.
Use auto-round for W4A16 unless you have a specific reason not to.
Use auto_round export format when your next step is transformers.
Prefer a short Python load-and-generate script as your first smoke test; the core API tends to be a steadier target than the serving CLI surface.
Only move to Qwen/Qwen3.5-4B or Qwen/Qwen3.5-35B-A3B after the small model loads and generates correctly.
Treat NVFP4 and llm_compressor as a separate branch of the decision tree, not as part of the default tutorial.

Sanity check

Before you scale up, verify these:

transformers.__version__ is 5.12.1
auto_round.__version__ is 0.13.1
auto-round -h runs successfully
AutoConfig.from_pretrained("Qwen/Qwen3.5-0.8B").model_type resolves to qwen3_5
AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B") loads successfully

Summary

The core update is simple. A modern Qwen3.5 + AutoRound guide does not need Git installs, a nightly transformers checkout, or auto-round-best as the default answer. The cleaner path is a stable package install, auto-round for W4A16, a small official Qwen3.5 checkpoint to validate the flow, and a plain transformers reload of the quantized output.

That is the version of this tutorial worth keeping.

Next Steps

Run the 0.8B recipe first and confirm you can load the quantized output locally.
Scale the same command to Qwen/Qwen3.5-4B once the workflow is stable.
Move to Qwen/Qwen3.5-35B-A3B only when your hardware budget matches the checkpoint.