Let AutoScheme Pick the GGUF: Mixed Quantization with AutoRound and Qwen3.6

TL;DR

The old notebook worked by patching around fast-moving internals. The better 2026 version is a CLI-first AutoRound workflow.
Use transformers==5.12.1, auto-round==0.13.1, and a current llama.cpp checkout for GGUF export.
Validate with Qwen/Qwen3.5-0.8B, then point the same AutoScheme recipe at Qwen/Qwen3.6-27B when you have the hardware.

Abstract

GGUF export is where quantization stops being an experiment and starts becoming something you can run locally. The original draft showed the right idea, using AutoRound with AutoScheme to choose a mixed GGUF layout, but it carried too much notebook residue: Git installs, duplicated monkey patches, and a final Python call that no longer matches the cleanest current API surface. This post rebuilds the workflow around the supported AutoRound CLI, a current Qwen checkpoint, and a small validation model you can use before spending hours on a larger export. By the end, you will have a reproducible path for generating mixed-quant GGUFs that can be loaded by llama.cpp-family runtimes.

Requirements

Use this stack for the examples below:

Python 3.10+
torch for your CPU or CUDA setup
transformers==5.12.1
auto-round==0.13.1
a current llama.cpp checkout for the GGUF Python package
enough disk space for the source checkpoint, calibration cache, and exported GGUF

Install PyTorch first with the selector on pytorch.org, then install the packages used by the tutorial:

uv pip install "transformers==5.12.1" "auto-round==0.13.1" sentencepiece
git clone https://github.com/ggml-org/llama.cpp.git
uv pip install ./llama.cpp/gguf-py

Optional speedups:

uv pip install flash-linear-attention causal-conv1d

Prerequisites

You know what GGUF is and why llama.cpp-style runtimes use it.
You are comfortable with post-training quantization tradeoffs.
You have already confirmed that the base model fits on the machine doing calibration.

Problem
Why Qwen3.6 Now
Background
Approach
Example
Failure Modes & Caveats
Practical Guidance
Summary
Next Steps
Resources

Problem

The old draft was useful as a debugging notebook, not as a durable tutorial.

It did three things that make a guide age quickly:

installed transformers and AutoRound directly from GitHub,
patched AutoRound and GGUF converter internals before explaining the main workflow,
and used a Python API call as the final step even though the current CLI is clearer for GGUF export.

Those patches made sense while the stack was moving. They are not the shape you want to teach. A good GGUF tutorial should make the ordinary path obvious first, then tell readers what to check when the converter complains.

Why Qwen3.6 Now

If you want a current official dense Qwen text model for this workflow, use Qwen/Qwen3.6-27B as the target. Its model config reports the qwen3_5 architecture family, which is supported by current transformers releases, and it is newer than the Qwen3.5 checkpoints used in the original notebook.

That said, a tutorial should not make you prove the workflow on a 27B model first.

Use this progression:

Qwen/Qwen3.5-0.8B to validate installs, tokenizer handling, and GGUF export.
Qwen/Qwen3.5-4B if you want a practical medium checkpoint.
Qwen/Qwen3.6-27B when you are ready to spend the time and memory on the current target.

The important part is that the command structure stays the same. Only the model ID, output path, and resource budget change.

Background

AutoRound can export many quantized formats. GGUF is the one you use when your next step is a llama.cpp-compatible runtime such as llama.cpp, llama-cpp-python, Ollama, or other local inference tools that consume GGUF files.

AutoScheme solves a different problem: instead of choosing one quantization type for every layer, you give AutoRound a menu of GGUF formats and a target average bit budget. AutoScheme then chooses a mixed assignment that tries to keep the model near that budget.

For example, instead of saying “make every layer Q4_K_M,” you can say:

Target average: about 3 bits
Allowed GGUF types: Q2_K_S, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_M, Q5_K_M, Q6_K, Q8_0

That gives AutoRound room to spend more bits where they matter and fewer where they hurt less.

Approach

The clean workflow is:

Install stable package releases.
Confirm AutoRound sees the GGUF export format.
Run a small model through AutoScheme.
Load the exported GGUF with a llama.cpp-style runtime.
Scale the same recipe to Qwen/Qwen3.6-27B.

This keeps the tutorial centered on the thing you actually want: a reproducible GGUF file, not a collection of compatibility patches.

Example

1. Confirm the current stack

Start by checking that you are on the intended versions and that AutoRound can see GGUF export support:

python - <<'PY'
import auto_round
import transformers

print("transformers", transformers.__version__)
print("auto_round", getattr(auto_round, "__version__", "unknown"))
PY

auto-round list format

In auto-round==0.13.1, the format list includes gguf with options such as GGUF:Q2_K_S, GGUF:Q3_K_M, GGUF:Q4_K_M, GGUF:Q5_K_M, GGUF:Q6_K, and GGUF:Q8_0.

2. Validate the workflow on a small checkpoint

Use the small Qwen3.5 checkpoint first. It is not the final recommendation; it is the quickest way to find install, tokenizer, or converter problems before you scale.

MODEL_ID="Qwen/Qwen3.5-0.8B"
OUTPUT_DIR="./Qwen3.5-0.8B-autoscheme-gguf"

auto-round \
  --model "$MODEL_ID" \
  --format gguf \
  --avg_bits 3.0 \
  --options "GGUF:Q4_K_M,GGUF:Q2_K_S,GGUF:Q3_K_S,GGUF:Q3_K_M,GGUF:Q3_K_L,GGUF:Q5_K_M,GGUF:Q6_K,GGUF:Q8_0" \
  --ignore_scale_zp_bits \
  --enable_alg_ext \
  --iters 200 \
  --nsamples 64 \
  --seqlen 1024 \
  --output_dir "$OUTPUT_DIR"

For the first pass, keep the calibration small. You are checking the pipeline, not trying to publish the final quant.

3. Scale to the current target model

Once the small export works, point the same structure at Qwen/Qwen3.6-27B:

MODEL_ID="Qwen/Qwen3.6-27B"
OUTPUT_DIR="./Qwen3.6-27B-autoscheme-gguf"

auto-round \
  --model "$MODEL_ID" \
  --format gguf \
  --avg_bits 3.0 \
  --options "GGUF:Q4_K_M,GGUF:Q2_K_S,GGUF:Q3_K_S,GGUF:Q3_K_M,GGUF:Q3_K_L,GGUF:Q5_K_M,GGUF:Q6_K,GGUF:Q8_0" \
  --ignore_scale_zp_bits \
  --enable_alg_ext \
  --iters 500 \
  --nsamples 128 \
  --seqlen 2048 \
  --low_gpu_mem_usage \
  --output_dir "$OUTPUT_DIR"

The options list is intentionally broad. Q8_0 and Q6_K let AutoScheme preserve sensitive layers, while the Q2 and Q3 choices give it room to hit a low average bit target.

If your hardware is tight, use Qwen/Qwen3.5-4B as the practical model and keep the rest of the command unchanged:

MODEL_ID="Qwen/Qwen3.5-4B"
OUTPUT_DIR="./Qwen3.5-4B-autoscheme-gguf"

4. Find the exported GGUF

AutoRound writes the quantized files into the output directory. After export, inspect the generated files:

ls "$OUTPUT_DIR"

You should see one or more .gguf files. If you see only intermediate files, rerun auto-round list format and confirm that gguf appears in the supported formats.

5. Run a local inference check

Build llama.cpp if you have not already:

cmake -S llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build --config Release -j

Then use llama.cpp to confirm that the file loads:

./llama.cpp/build/bin/llama-cli \
  -m "$OUTPUT_DIR"/*.gguf \
  -p "Explain mixed quantization in one paragraph." \
  -n 128

If your llama.cpp binary is named differently, use the equivalent local runner for your build. The point is not benchmark quality yet; it is simply that the exported GGUF opens and generates.

Failure Modes & Caveats

Do not start with the 27B checkpoint

Large-model quantization failures are expensive to debug because every iteration costs time. Start with Qwen/Qwen3.5-0.8B, then scale.

Keep `llama.cpp` current

GGUF export depends on converter support for the tokenizer and architecture. If AutoRound fails during GGUF conversion, update your llama.cpp checkout and reinstall gguf-py before patching Python internals.

cd llama.cpp
git pull
cd ..
uv pip install --upgrade ./llama.cpp/gguf-py

Tokenizer errors usually mean converter drift

The old notebook patched the GGUF converter to force tokenizer.ggml.pre = "qwen2" when the pre-tokenizer was not recognized. That is a useful debugging clue, but it should not be the default tutorial path. With a current llama.cpp checkout, try upgrading the converter first.

AutoScheme is not magic

A lower average bit target gives smaller files, but it can also hurt model quality. Treat avg_bits=3.0 as an aggressive local-serving target, not as a universal best setting. If outputs degrade, try avg_bits=4.0 or narrow the options around Q4_K_M, Q5_K_M, and Q6_K.

Calibration settings matter

The small validation command uses --iters 200, --nsamples 64, and --seqlen 1024 to keep the first run approachable. For the final export, increase calibration work if you care about quality. The example uses --iters 500, --nsamples 128, and --seqlen 2048 as a stronger starting point.

Practical Guidance

Use this decision path:

If you are testing the stack, use Qwen/Qwen3.5-0.8B.
If you want a practical local model, use Qwen/Qwen3.5-4B.
If you want a current dense Qwen text checkpoint in this family, use Qwen/Qwen3.6-27B.
If the GGUF converter fails, update llama.cpp before changing AutoRound code.
If quality is poor, raise avg_bits before adding complicated per-layer overrides.

Sanity check

Before you publish or share the GGUF, verify:

transformers.__version__ is 5.12.1.
auto_round.__version__ is 0.13.1.
auto-round list format includes gguf.
The source model config resolves to model_type: qwen3_5.
The exported .gguf loads in your target llama.cpp-style runtime.

Summary

The cleaner version of this workflow is not a bigger patch. It is a smaller shape: stable package versions, current Qwen checkpoints, AutoRound’s CLI, AutoScheme’s mixed GGUF options, and a small model used as the first proof that the pipeline works.

Use Qwen/Qwen3.6-27B when you want the current target. Use Qwen/Qwen3.5-0.8B when you want to make sure the path is sound before paying the full quantization cost.

Next Steps

Run the small-model command and confirm the GGUF loads locally.
Increase avg_bits and compare output quality before settling on a final quant.
Scale the same recipe to Qwen/Qwen3.6-27B once the validation export works.