From Prompts to Practice: Instruction Tuning Qwen3 with Transformers 5

TL;DR

This tutorial replaces an older DeciLM notebook workflow with a current transformers 5.x and trl 1.x supervised fine-tuning pipeline.
The model is Qwen/Qwen3-4B-Instruct-2507, a practical Apache-2.0 model for QLoRA instruction tuning.
You will format chat data with the tokenizer’s own chat template, fine-tune with PEFT adapters, run a quick generation check, and optionally push the adapter to the Hugging Face Hub.

Abstract

Instruction tuning turns a base language model into a more useful assistant by training it on examples of user requests and helpful responses. Older tutorials often depend on hand-written prompt templates, pinned 2023-era libraries, and model-specific code that breaks as the Hugging Face stack evolves. This guide shows a cleaner path with current transformers, trl, peft, and datasets: use the model’s chat template, train QLoRA adapters, and keep the training script small enough to understand. By the end, you will have a working instruction-tuning baseline that you can adapt to your own dataset.

Requirements

Python 3.10+
A CUDA GPU for QLoRA training
A Hugging Face account if you want to push the adapter
Basic familiarity with language-model fine-tuning and GPU memory limits

Install the libraries used in the examples:

pip install -U "transformers>=5.12.1" "trl>=1.7.0" "datasets>=5.0.0" \
  "peft>=0.19.1" "accelerate>=1.12.0" "bitsandbytes>=0.49.2" torch

If you plan to upload the result:

huggingface-cli login

Why This Tutorial Needed an Update

The original notebook used Deci/DeciLM-6b, transformers==4.31.0, trl==0.4.7, and a custom Alpaca-style string template:

### Instruction:
...

### Input:
...

### Response:
...

That pattern worked at the time, but it has three problems today:

Modern instruct models already define their expected conversation format in the tokenizer.
Current TRL uses SFTConfig, processing_class, and updated dataset handling.
A hard-coded prompt format can train the model on text that does not match how you will use it at inference time.

The better invariant is simple: train on the same chat format you will use during generation.

Model Choice

The strongest current open Qwen model family on relevant Hugging Face leaderboards is Qwen 3.6. In the official TIGER-Lab/MMLU-Pro leaderboard, Qwen/Qwen3.6-27B appears near the top open-weight models with an MMLU-Pro score of 86.2; it also appears in harborframework/terminal-bench-2.0 with a score of 59.3. It is Apache-2.0 and very capable, but at roughly 27.8B parameters it is not the best default for a tutorial that readers should be able to adapt quickly.

For this walkthrough, use:

MODEL_ID = "Qwen/Qwen3-4B-Instruct-2507"

Why this model:

It is small enough for QLoRA experiments on a single modern GPU.
It uses the standard AutoModelForCausalLM and AutoTokenizer path in transformers.
It ships with a chat template, so you do not need to invent a prompt format.
It is Apache-2.0, which makes it friendlier for tutorials and downstream experiments.

If you have larger hardware, the same structure can be adapted to a larger Qwen chat model that supports the standard causal language model API. Keep the dataset and chat-template logic the same; adjust batch size, context length, and LoRA targets only when the model architecture requires it.

The Training Plan

The workflow has five stages:

Load a public instruction dataset.
Render each conversation with the model’s tokenizer chat template.
Load the model in 4-bit with BitsAndBytes.
Train LoRA adapters with SFTTrainer.
Generate a response from the tuned adapter.

For a first run, train on a small slice. Once the script works, increase the dataset size and run a real evaluation.

Load and Format the Dataset

Use HuggingFaceH4/ultrachat_200k because it already stores examples as chat messages. The important move is converting those messages into the exact string format expected by Qwen3.

from datasets import load_dataset
from transformers import AutoTokenizer

MODEL_ID = "Qwen/Qwen3-4B-Instruct-2507"
DATASET_ID = "HuggingFaceH4/ultrachat_200k"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token


def render_chat(example: dict) -> dict:
    """Render one conversation using the model's native chat template."""
    text = tokenizer.apply_chat_template(
        example["messages"],
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": text}


raw_dataset = load_dataset(DATASET_ID, split="train_sft[:6000]")
split_dataset = raw_dataset.train_test_split(test_size=500, seed=42)

train_dataset = split_dataset["train"].map(
    render_chat,
    remove_columns=split_dataset["train"].column_names,
)
eval_dataset = split_dataset["test"].map(
    render_chat,
    remove_columns=split_dataset["test"].column_names,
)

You now have a dataset with one text column. That keeps the trainer configuration explicit and avoids depending on hidden dataset-format inference.

Load Qwen3 in 4-bit

QLoRA fine-tunes small adapter weights while the base model stays quantized. This gives you a practical way to tune a 4B model without full-precision training memory.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig


def preferred_compute_dtype() -> torch.dtype:
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported():
        return torch.bfloat16
    return torch.float16


quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=preferred_compute_dtype(),
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=quantization_config,
    device_map="auto",
    dtype=preferred_compute_dtype(),
)

model.config.use_cache = False

use_cache=False matters during training because gradient checkpointing and cached key/value states do not mix well.

Configure LoRA

For Qwen-style decoder models, LoRA usually works well on attention projections and MLP projections.

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
)

Start with r=16. If the adapter underfits and you have memory headroom, try r=32 or r=64.

Train with Current TRL

Current TRL uses SFTConfig for supervised fine-tuning arguments. The precision flags below are intentionally derived from the machine so the config is valid on different hardware.

import torch
from trl import SFTConfig, SFTTrainer

OUTPUT_DIR = "qwen3-4b-ultrachat-qlora"


def precision_flags() -> dict:
    if not torch.cuda.is_available():
        return {"bf16": False, "fp16": False, "use_cpu": True}

    use_bf16 = torch.cuda.is_bf16_supported()
    return {"bf16": use_bf16, "fp16": not use_bf16, "use_cpu": False}


training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    dataset_text_field="text",
    max_length=2048,
    packing=True,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=25,
    optim="paged_adamw_8bit",
    gradient_checkpointing=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="epoch",
    report_to="none",
    **precision_flags(),
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    peft_config=lora_config,
    processing_class=tokenizer,
)

trainer.train()
trainer.save_model(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

For a real run, change train_sft[:6000] to a larger slice or the full split, then train for more than one epoch only if validation loss and qualitative samples justify it.

Try the Adapter

After training, load the adapter and generate with the same tokenizer chat template.

import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

OUTPUT_DIR = "qwen3-4b-ultrachat-qlora"

tokenizer = AutoTokenizer.from_pretrained(OUTPUT_DIR, use_fast=True)
inference_dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float16
)
model = AutoPeftModelForCausalLM.from_pretrained(
    OUTPUT_DIR,
    device_map="auto",
    dtype=inference_dtype,
)
model.eval()

messages = [
    {
        "role": "user",
        "content": "Explain why chat templates matter when instruction tuning a language model.",
    }
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=180,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )

new_tokens = output_ids[0, input_ids.shape[-1] :]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Compare this output with the base model on the same prompt. You are looking for better instruction following, not just longer answers.

Push the Adapter to the Hub

For most experiments, push the adapter instead of merging the full model. It is smaller, faster to upload, and makes the relationship to the base model clear.

trainer.model.push_to_hub("your-username/qwen3-4b-ultrachat-qlora")
tokenizer.push_to_hub("your-username/qwen3-4b-ultrachat-qlora")

If you need a standalone merged model, merge only when you have enough CPU or GPU memory:

merged_model = trainer.model.merge_and_unload()
merged_model.push_to_hub("your-username/qwen3-4b-ultrachat-merged")

Sanity Checks

Before starting a long run, verify these:

tokenizer.apply_chat_template(...) returns Qwen-style chat markup.
The training dataset has one text column after mapping.
SFTConfig initializes without precision errors on your machine.
A tiny run over a few dozen examples completes before you scale up.
Generation uses add_generation_prompt=True; training examples use add_generation_prompt=False.

That last point is easy to miss. During training, the assistant answer is already present. During inference, you want the template to end at the assistant turn so the model knows to continue.

Failure Modes and Caveats

QLoRA makes tuning cheaper, but it does not make evaluation optional. Watch for:

Template mismatch: If training and inference use different formats, the model may learn behavior you never trigger at inference time.
Overfitting small slices: A 5,000-example run is useful for validating the pipeline, not for claiming a generally better assistant.
Low-quality instruction data: The model learns tone, refusal style, verbosity, and mistakes from the dataset.
Memory pressure: Increase gradient_accumulation_steps, reduce max_length, or disable packing experiments before changing several knobs at once.
Merged-model uploads: Merging adapters can require much more memory than adapter training.

Practical Guidance

Use the smallest experiment that tells you something:

Run the tokenizer and dataset formatting code.
Train on 50 examples to validate the trainer path.
Train on 5,000 examples to check qualitative behavior.
Add a task-specific evaluation set before training on the full dataset.
Only then tune rank, learning rate, context length, and dataset mixture.

If your real goal is a domain assistant, replace UltraChat with your own instruction data, but keep the messages structure:

example = {
    "messages": [
        {"role": "user", "content": "Summarize this support ticket in one sentence: ..."},
        {"role": "assistant", "content": "The customer cannot reset their password because ..."},
    ]
}

That structure lets the tokenizer handle the exact chat format for the model you choose.

Summary

Instruction tuning is no longer about hand-assembling prompt strings and hoping they match the model. With current Hugging Face libraries, the cleaner path is to keep conversations structured, render them with the tokenizer’s chat template, and let TRL train on that final text. Qwen/Qwen3-4B-Instruct-2507 is a practical default for this workflow: current, permissively licensed, small enough for QLoRA, and compatible with the standard transformers causal language model API.

Next Steps

Swap in your own instruction dataset using the same messages schema.
Add a small held-out evaluation set that matches your real task.
Try a larger Qwen model only after the 4B pipeline is stable.
Push the adapter to the Hub so your training config, tokenizer, and model card stay together.