Train a Modern VLM on One GPU: Qwen3-VL with Unsloth and Transformers 5.12.1

TL;DR

• The original notebook was built around Qwen2-VL-7B, duplicated code, and mixed blog prose with Colab-only scaffolding.
• A better current baseline is Qwen3-VL-8B-Instruct: it is newer, directly supported by transformers==5.12.1, and still fits the same efficient LoRA-style fine-tuning story.
• For the current Hugging Face API, the clean multimodal path is AutoProcessor plus Qwen3VLForConditionalGeneration, with chat-template formatting handled inside transformers.

Abstract

The hard part of maintaining model tutorials is not getting them to work once. It is keeping them honest after the ecosystem moves. The original draft here had the right instinct, but it was frozen around an older Qwen2-VL notebook export, repeated the same preprocessing twice, and relied on an inference path that no longer reflects the cleanest modern transformers workflow. This rewrite keeps the same goal, efficient vision-language fine-tuning with Unsloth, but updates the core model to Qwen3-VL-8B-Instruct, tightens the training scaffold, and replaces the brittle notebook flow with a reusable blog post and script.

By the end, you will have a current DocumentVQA fine-tuning recipe, a matching inference example for the saved adapter, and a clearer mental model for what actually matters when adapting a modern vision-language model (VLM).

Requirements

Use this setup if you want to reproduce the example as written:

• python>=3.10
• transformers==5.12.1
• a current PyTorch 2.12.x build for your CUDA or CPU setup
• unsloth
• trl
• peft
• accelerate
• datasets
• Pillow

Install PyTorch first with the selector on pytorch.org, then install the Python packages used by the examples:

pip install "transformers==5.12.1" "datasets>=3.0.0" "trl>=0.19.0" "peft>=0.13.0" "accelerate>=1.0.0" pillow unsloth

If your machine and CUDA stack support FlashAttention 2, you can add it later for faster inference. The examples below do not require it.

Prerequisites

• Basic familiarity with Hugging Face model loading and LoRA adapters
• A CUDA GPU with enough memory for 4-bit loading plus training overhead
• Comfort working with image-question-answer data such as DocumentVQA
• Enough local disk space for the base checkpoint, dataset cache, and adapter checkpoints

Table of Contents

• Problem
• Why Qwen3-VL
• Approach
• Example
• Inference with the Adapter
• Failure Modes & Caveats
• Practical Guidance
• Summary
• Next Steps
• Resources

Problem

Notebook exports often preserve the wrong things. They keep transient setup cells, repeated helper functions, and one-off measurements, but they hide the actual reusable structure.

That is what happened in the original draft:

• it was tied to Qwen2-VL-7B,
• it duplicated the conversation-building logic for training and inference,
• it mixed old-style dependency cells into the main narrative,
• and it used an inference path that can now be simplified with the built-in multimodal processor flow in transformers.

The result was not terrible, but it was harder to trust than it needed to be.

Why Qwen3-VL

If you are updating this tutorial in mid-2026, the main question is simple: should you keep the older Qwen2-VL baseline, move to Qwen2.5-VL, or use the newer Qwen3-VL family?

For this post, Qwen3-VL-8B-Instruct is the best fit:

1. It is materially newer than Qwen2-VL and Qwen2.5-VL.
2. It stays in roughly the same practical size class as the original 7B tutorial.
3. transformers==5.12.1 resolves its config, processor, and multimodal chat path cleanly.
4. Unsloth documents Qwen3-VL support through FastVisionModel, so the fine-tuning story still makes sense.

If you need a smaller starting point, Qwen/Qwen3-VL-4B-Instruct is a reasonable drop-in alternative for the inference side.

Approach

The updated workflow is intentionally boring:

1. Load unsloth/Qwen3-VL-8B-Instruct in 4-bit mode.
2. Add LoRA adapters across the vision and language stack.
3. Convert DocumentVQA examples into chat-style messages records.
4. Train with SFTTrainer plus UnslothVisionDataCollator.
5. Load the base Qwen/Qwen3-VL-8B-Instruct checkpoint for inference and attach the saved adapter with PeftModel.
6. Use AutoProcessor.apply_chat_template(...) and processor(..., images=[...]) for the modern multimodal inference path.

Two choices matter more than they first appear:

• The dataset should already be shaped like a conversation, so the trainer does not need custom prompt assembly inside the loss loop.
• For VLM fine-tuning, max_length=None is safer than forcing an arbitrary text-centric limit that may cut away image tokens.

Example

The script below keeps the same task as the original draft, fine-tuning on HuggingFaceM4/DocumentVQA, but modernizes the model choice and removes duplicated logic. It is written like a real script on purpose: named constants, compact helpers, and one clear training flow.

from __future__ import annotations

from typing import Final

from datasets import Dataset, load_dataset
from PIL import Image
from trl import SFTConfig, SFTTrainer
from unsloth import FastVisionModel, is_bf16_supported
from unsloth.trainer import UnslothVisionDataCollator


MODEL_NAME: Final[str] = "unsloth/Qwen3-VL-8B-Instruct"
DATASET_NAME: Final[str] = "HuggingFaceM4/DocumentVQA"
OUTPUT_DIR: Final[str] = "qwen3-vl-documentvqa-lora"
MAX_IMAGE_EDGE: Final[int] = 896
MAX_STEPS: Final[int] = 200
SEED: Final[int] = 3407
SYSTEM_PROMPT: Final[str] = (
    "You are an expert in document analysis. "
    "Answer the user's question using the provided document image."
)


def resize_long_edge(image: Image.Image, max_edge: int) -> Image.Image:
    """Resize an image while preserving aspect ratio."""
    width, height = image.size
    long_edge = max(width, height)
    if long_edge <= max_edge:
        return image.convert("RGB")

    scale = max_edge / long_edge
    new_size = (int(width * scale), int(height * scale))
    return image.convert("RGB").resize(new_size)


def format_sample(sample: dict) -> dict:
    """Convert one DocumentVQA record into a multimodal chat example."""
    image = resize_long_edge(sample["image"], MAX_IMAGE_EDGE)
    answer = sample["answers"][0] if sample["answers"] else ""

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {
                    "type": "text",
                    "text": f"{SYSTEM_PROMPT}\n\nQuestion: {sample['question']}",
                },
            ],
        },
        {
            "role": "assistant",
            "content": [{"type": "text", "text": answer}],
        },
    ]
    return {"messages": messages}


def build_dataset(limit: int = 1000) -> Dataset:
    """Load and convert a small training slice for a quick experiment."""
    raw_dataset = load_dataset(DATASET_NAME, split=f"train[:{limit}]")
    return Dataset.from_list([format_sample(sample) for sample in raw_dataset])


def main() -> None:
    train_dataset = build_dataset(limit=1000)

    model, processor = FastVisionModel.from_pretrained(
        MODEL_NAME,
        load_in_4bit=True,
        use_gradient_checkpointing="unsloth",
    )

    model = FastVisionModel.get_peft_model(
        model,
        finetune_vision_layers=True,
        finetune_language_layers=True,
        finetune_attention_modules=True,
        finetune_mlp_modules=True,
        r=16,
        lora_alpha=16,
        lora_dropout=0,
        bias="none",
        random_state=SEED,
        use_rslora=False,
        loftq_config=None,
        target_modules="all-linear",
        modules_to_save=["lm_head", "embed_tokens"],
    )

    FastVisionModel.for_training(model)

    trainer = SFTTrainer(
        model=model,
        train_dataset=train_dataset,
        processing_class=processor.tokenizer,
        data_collator=UnslothVisionDataCollator(model, processor),
        args=SFTConfig(
            output_dir=OUTPUT_DIR,
            per_device_train_batch_size=1,
            gradient_accumulation_steps=8,
            warmup_steps=5,
            max_steps=MAX_STEPS,
            learning_rate=2e-4,
            fp16=not is_bf16_supported(),
            bf16=is_bf16_supported(),
            logging_steps=10,
            save_steps=100,
            save_total_limit=2,
            optim="adamw_8bit",
            weight_decay=0.01,
            lr_scheduler_type="linear",
            seed=SEED,
            report_to="none",
            remove_unused_columns=False,
            dataset_text_field="",
            dataset_kwargs={"skip_prepare_dataset": True},
            max_length=None,
        ),
    )

    trainer.train()
    trainer.save_model()
    processor.save_pretrained(OUTPUT_DIR)


if __name__ == "__main__":
    main()

There are four important updates here relative to the old notebook:

• Qwen3-VL-8B-Instruct replaces the stale Qwen2-VL-7B baseline.
• The dataset conversion happens once, not once for training and again for inference.
• The trainer uses processing_class=processor.tokenizer and UnslothVisionDataCollator(model, processor) instead of treating the vision processor like plain text-only tokenization.
• max_length=None avoids text-style truncation logic that can quietly break multimodal batches.

Inference with the Adapter

Once training finishes, you can attach the saved LoRA adapter to the base model and run a quick prediction on a validation example.

The main API change to notice is that you no longer need extra helper glue for a simple single-image flow. The modern path is chat template -> processor -> generate.

from __future__ import annotations

from typing import Final

from datasets import load_dataset
from peft import PeftModel
import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration


BASE_MODEL: Final[str] = "Qwen/Qwen3-VL-8B-Instruct"
ADAPTER_PATH: Final[str] = "qwen3-vl-documentvqa-lora/checkpoint-200"
PROMPT: Final[str] = (
    "You are an expert in document analysis. "
    "Answer the user's question using the document image.\n\n"
)


def move_to_model_device(batch: dict[str, torch.Tensor], model) -> dict[str, torch.Tensor]:
    """Move processor outputs onto the model's device."""
    return {key: value.to(model.device) for key, value in batch.items()}


def main() -> None:
    device = "cuda" if torch.cuda.is_available() else "cpu"
    dtype = torch.bfloat16 if device == "cuda" else torch.float32

    processor = AutoProcessor.from_pretrained(BASE_MODEL)
    model = Qwen3VLForConditionalGeneration.from_pretrained(
        BASE_MODEL,
        torch_dtype=dtype,
        device_map="auto" if device == "cuda" else None,
    )
    model = PeftModel.from_pretrained(model, ADAPTER_PATH)

    if device == "cpu":
        model = model.to(device)

    model.eval()

    sample = load_dataset("HuggingFaceM4/DocumentVQA", split="validation[:1]")[0]
    image = sample["image"].convert("RGB")
    question = sample["question"]

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": f"{PROMPT}Question: {question}"},
            ],
        }
    ]

    chat_text = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    inputs = processor(
        text=[chat_text],
        images=[image],
        padding=True,
        return_tensors="pt",
    )
    inputs = move_to_model_device(inputs, model)

    with torch.inference_mode():
        generated_ids = model.generate(
            **inputs,
            max_new_tokens=64,
            do_sample=False,
        )

    trimmed_ids = generated_ids[:, inputs["input_ids"].shape[1] :]
    answer = processor.batch_decode(
        trimmed_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]

    print("Question:", question)
    print("Prediction:", answer)
    print("Reference answers:", sample["answers"])


if __name__ == "__main__":
    main()

The multimodal processor path above was smoke-tested against transformers==5.12.1, including:

• AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
• processor.apply_chat_template(...)
• processor(..., images=[image], return_tensors="pt")

That matters because this is the piece most likely to drift when the underlying model family changes.

Failure Modes & Caveats

This workflow is cleaner than the original notebook, but it is still easy to trip over a few realities:

• DocumentVQA answers are not always perfectly clean, so a model can appear wrong when the label itself is noisy.
• Vision-language fine-tuning is much more sensitive to sequence construction than plain text SFT. A bad messages format can silently poison training.
• Qwen3-VL-8B-Instruct is practical for modern single-GPU experimentation, but it is not lightweight in the way a small text-only model is lightweight.
• If you hard-cap sequence length the way you would for a text-only SFT job, you can truncate image tokens and get confusing failures.
• A short train[:1000] slice is a debugging and iteration choice, not a serious full-data training strategy.

Practical Guidance

If you want this post to stay useful beyond one run, start with these habits:

• Keep the base model current, but keep the training recipe simple.
• Test the processor path separately from the full training loop.
• Resize images deliberately instead of letting resolution explode sequence cost.
• Start with a small subset and verify one end-to-end batch before launching a longer run.
• Save the adapter and processor together so your inference path matches your training assumptions.

One more practical point: for blog code, boring wins. The point is not to show every tunable switch Unsloth exposes. The point is to leave the reader with a small shape they can actually adapt.

Sanity check

Before trusting the setup, verify that:

• AutoProcessor resolves for Qwen/Qwen3-VL-8B-Instruct,
• the processor returns both input_ids and pixel_values,
• one multimodal batch can be built from a PIL image and a chat template,
• and the LoRA adapter path loads back into the base model without changing the prompt format.

Summary

The original notebook was worth updating because the ecosystem around it moved. Qwen2-VL-7B is no longer the best anchor for a “current” Unsloth VLM tutorial, and the old inference flow carried more notebook residue than reusable structure.

Qwen3-VL-8B-Instruct is a better center of gravity for this post. It is newer, it stays in a practical model-size range, and it works with the modern transformers multimodal chat path. Once you strip away the clutter, the core recipe is not complicated: build clean messages, fine-tune with Unsloth and the vision data collator, then run inference through AutoProcessor and Qwen3VLForConditionalGeneration.

Next Steps

• Increase the training slice beyond train[:1000] once the small run behaves the way you expect.
• Compare Qwen3-VL-8B-Instruct against Qwen3-VL-4B-Instruct if you care more about cost than absolute capacity.
• Add exact-match or ANLS-style evaluation if you want a more systematic DocumentVQA validation pass.

Related reading:

• For a more general VQA fine-tuning path, read Modern VQA Fine-Tuning.
• For preference tuning after supervised adaptation, read Preference-Aligning Vision Models.

If your team is adapting a VLM to internal documents or workflows, talk to us. We can help shape the dataset, training loop, and evaluation path before the model becomes expensive to unwind.

Resources

• Qwen3-VL documentation in Transformers
• Transformers multimodal chat templates documentation
• Unsloth vision fine-tuning documentation
• Qwen3-VL-8B-Instruct model page
• DocumentVQA dataset page