Modern Visual Question Answering Fine-Tuning: Qwen3-VL-4B with QLoRA in Transformers 5.12.1

TL;DR

• The original PaliGemma draft was really a Colab export, not a durable tutorial.
• A better current baseline is Qwen/Qwen3-VL-4B-Instruct: newer, public, supported by transformers==5.12.1, and still close to the original model size class.
• The clean modern path is AutoProcessor plus Qwen3VLForConditionalGeneration, with multimodal chat formatting handled by the processor instead of hand-built prompt glue.

Abstract

Tutorials like this age in two ways at once: the model gets old, and the surrounding code picks up notebook residue that makes it harder to trust than it should be. That happened here. The original draft used google/paligemma-3b-pt-224, mixed blog prose with Colab-only setup, and relied on a processor flow that no longer matches the most natural transformers workflow for current vision-language models.

This rewrite keeps the original goal, fine-tuning a compact visual question answering (VQA) model with QLoRA, but updates the center of gravity to Qwen/Qwen3-VL-4B-Instruct. By the end, you will have a small VQAv2 training recipe, a matching inference path for the saved adapter, and a simpler mental model for how modern multimodal fine-tuning should look in transformers.

Requirements

Use this setup if you want to run the example as written:

• python>=3.10
• torch for your CUDA or CPU setup
• transformers==5.12.1
• datasets
• peft
• bitsandbytes
• accelerate
• Pillow

Install PyTorch first with the selector on pytorch.org, then install the Python packages used below:

pip install "transformers==5.12.1" "datasets>=3.0.0" "peft>=0.15.0" "bitsandbytes>=0.46.0" "accelerate>=1.0.0" pillow

Prerequisites

• Basic familiarity with Hugging Face model loading
• A CUDA GPU if you want practical QLoRA training speed
• Comfort working with image-question-answer datasets
• Enough local disk space for the base checkpoint, dataset cache, and adapter checkpoints

Table of Contents

• Problem
• Why This Model Now
• Approach
• Example
• Inference with the Adapter
• Failure Modes & Caveats
• Practical Guidance
• Summary
• Next Steps
• Resources

Problem

The old notebook had the right ambition but the wrong shape.

Its real problems were not subtle:

• it was tied to an older PaliGemma checkpoint,
• it carried a large amount of notebook boilerplate into the main tutorial,
• it assumed a very specific processor and prompt style,
• and it did not clearly separate the reusable training logic from the one-off Colab setup.

That is the kind of draft that still works once, but stops teaching well.

Why This Model Now

If you are refreshing a VQA tutorial in mid-2026, the natural question is not whether PaliGemma is unusable. It is whether it is still the best default teaching model.

For this post, the answer is no.

Qwen/Qwen3-VL-4B-Instruct is the better center of gravity:

1. It is materially newer than PaliGemma and Qwen2.5-VL.
2. It stays close to the original tutorial’s “small enough to fine-tune with QLoRA” spirit.
3. It is public and ungated, which matters for reproducibility.
4. transformers==5.12.1 exposes a clean processor and model path for it.

There are two nearby alternatives worth knowing about:

• Qwen/Qwen3-VL-2B-Instruct is even lighter and newer, but gives up some headroom.
• Qwen/Qwen3-VL-8B-Instruct is stronger, but asks for more memory and overlaps too much with a larger-model fine-tuning story.

For a modern replacement of a 3B-class PaliGemma walkthrough, 4B is the sweet spot.

Approach

The updated workflow is deliberately small:

1. Load Qwen3-VL-4B-Instruct in 4-bit mode.
2. Freeze the vision tower so QLoRA stays focused on the language side.
3. Build VQA prompts with the processor’s multimodal chat template.
4. Apply LoRA to the language projection layers, not to the vision encoder.
5. Mask the user-side prompt tokens so the loss focuses on answer generation.
6. Save the adapter and reuse the same prompt format at inference time.

Two choices matter more than they first appear:

• Do not let the LoRA target selection spill into the visual stack just because some module names match.
• Do not train on the entire prompt as if it were assistant output; mask the prompt tokens and train the answer continuation.

Example

The script below keeps the original spirit, fine-tuning a compact VLM on a small VQAv2 slice, but replaces the PaliGemma-specific pieces with a cleaner Qwen3-VL recipe.

from __future__ import annotations

from typing import Final

import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import (
    AutoProcessor,
    BitsAndBytesConfig,
    Qwen3VLForConditionalGeneration,
    Trainer,
    TrainingArguments,
)


MODEL_NAME: Final[str] = "Qwen/Qwen3-VL-4B-Instruct"
DATASET_NAME: Final[str] = "merve/vqav2-small"
OUTPUT_DIR: Final[str] = "qwen3-vl-4b-vqav2-qlora"
SEED: Final[int] = 42


def build_splits():
    """Reuse the validation split as a tiny train/test sandbox."""
    dataset = load_dataset(DATASET_NAME, split="validation")
    split = dataset.train_test_split(test_size=0.05, seed=SEED)
    return split["train"], split["test"]


def build_question_text(question: str) -> str:
    """Create a short-answer VQA instruction."""
    return (
        "Answer the visual question using a short phrase.\n"
        "If the image does not contain enough information, say so.\n\n"
        f"Question: {question}"
    )


def build_prompt_messages(question: str) -> list[dict]:
    """Create the user-side multimodal prompt."""
    return [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": build_question_text(question)},
            ],
        }
    ]


def build_full_messages(question: str, answer: str) -> list[dict]:
    """Append the reference answer as the assistant turn."""
    return build_prompt_messages(question) + [
        {
            "role": "assistant",
            "content": [{"type": "text", "text": answer}],
        }
    ]


def select_lora_target_modules(model) -> list[str]:
    """Select only language-side linear layers for LoRA injection."""
    allowed_suffixes = {
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    }

    target_modules: list[str] = []
    for name, module in model.named_modules():
        if name.startswith("visual"):
            continue
        if isinstance(module, torch.nn.Linear) and name.split(".")[-1] in allowed_suffixes:
            target_modules.append(name)

    if not target_modules:
        raise ValueError("No language-side LoRA targets were found.")

    return sorted(set(target_modules))


def build_collate_fn(processor):
    """Tokenize full conversations and mask the prompt-side loss."""
    pad_token_id = processor.tokenizer.pad_token_id
    image_token_id = getattr(processor, "image_token_id", None)

    def collate_fn(examples: list[dict]) -> dict[str, torch.Tensor]:
        images = [example["image"].convert("RGB") for example in examples]

        prompt_texts = [
            processor.apply_chat_template(
                build_prompt_messages(example["question"]),
                tokenize=False,
                add_generation_prompt=True,
            )
            for example in examples
        ]

        full_texts = [
            processor.apply_chat_template(
                build_full_messages(
                    question=example["question"],
                    answer=example["multiple_choice_answer"],
                ),
                tokenize=False,
                add_generation_prompt=False,
            )
            for example in examples
        ]

        prompt_batch = processor(
            text=prompt_texts,
            images=images,
            padding=True,
            return_tensors="pt",
        )
        full_batch = processor(
            text=full_texts,
            images=images,
            padding=True,
            return_tensors="pt",
        )

        labels = full_batch["input_ids"].clone()
        labels[labels == pad_token_id] = -100

        if image_token_id is not None:
            labels[labels == image_token_id] = -100

        for index in range(len(examples)):
            prompt_length = int(prompt_batch["attention_mask"][index].sum().item())
            labels[index, :prompt_length] = -100

        full_batch["labels"] = labels
        return full_batch

    return collate_fn


def main() -> None:
    train_dataset, eval_dataset = build_splits()

    quantization_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

    processor = AutoProcessor.from_pretrained(MODEL_NAME)
    model = Qwen3VLForConditionalGeneration.from_pretrained(
        MODEL_NAME,
        quantization_config=quantization_config,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )

    # Keep the visual encoder fixed and adapt the language stack.
    for param in model.visual.parameters():
        param.requires_grad = False

    model = prepare_model_for_kbit_training(model)

    lora_config = LoraConfig(
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules=select_lora_target_modules(model),
    )

    model = get_peft_model(model, lora_config)
    model.print_trainable_parameters()

    trainer = Trainer(
        model=model,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=build_collate_fn(processor),
        args=TrainingArguments(
            output_dir=OUTPUT_DIR,
            num_train_epochs=2,
            per_device_train_batch_size=1,
            gradient_accumulation_steps=8,
            learning_rate=2e-4,
            warmup_steps=5,
            logging_steps=10,
            save_steps=100,
            save_total_limit=2,
            eval_steps=100,
            eval_strategy="steps",
            remove_unused_columns=False,
            bf16=torch.cuda.is_available() and torch.cuda.is_bf16_supported(),
            fp16=torch.cuda.is_available() and not torch.cuda.is_bf16_supported(),
            optim="paged_adamw_8bit",
            report_to="none",
            dataloader_pin_memory=False,
            seed=SEED,
        ),
    )

    trainer.train()
    trainer.save_model()
    processor.save_pretrained(OUTPUT_DIR)


if __name__ == "__main__":
    main()

Four updates matter in this version:

• Qwen3-VL-4B-Instruct replaces the stale PaliGemma baseline.
• The collator uses the processor’s chat template instead of a model-specific prompt string hack.
• Prompt-side tokens are masked out of the loss, so the model learns the answer continuation instead of re-predicting the prompt.
• LoRA targets are chosen explicitly from language-side linear layers, which avoids quietly adapting the wrong part of the multimodal stack.

Inference with the Adapter

Once training finishes, you can attach the saved adapter and run a quick held-out prediction.

The modern inference path is simple on purpose: chat template, processor, generate, decode.

from __future__ import annotations

from typing import Final

import torch
from datasets import load_dataset
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration


BASE_MODEL: Final[str] = "Qwen/Qwen3-VL-4B-Instruct"
ADAPTER_PATH: Final[str] = "qwen3-vl-4b-vqav2-qlora/checkpoint-100"


def move_to_model_device(batch: dict[str, torch.Tensor], model) -> dict[str, torch.Tensor]:
    """Move processor outputs onto the model device."""
    return {key: value.to(model.device) for key, value in batch.items()}


def main() -> None:
    device = "cuda" if torch.cuda.is_available() else "cpu"
    dtype = torch.bfloat16 if device == "cuda" else torch.float32

    processor = AutoProcessor.from_pretrained(BASE_MODEL)
    model = Qwen3VLForConditionalGeneration.from_pretrained(
        BASE_MODEL,
        torch_dtype=dtype,
        device_map="auto" if device == "cuda" else None,
    )
    model = PeftModel.from_pretrained(model, ADAPTER_PATH)

    if device == "cpu":
        model = model.to(device)

    model.eval()

    sample = load_dataset("merve/vqav2-small", split="validation[9:10]")[0]
    image = sample["image"].convert("RGB")
    question = sample["question"]

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {
                    "type": "text",
                    "text": (
                        "Answer the visual question using a short phrase.\n\n"
                        f"Question: {question}"
                    ),
                },
            ],
        }
    ]

    chat_text = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    inputs = processor(
        text=[chat_text],
        images=[image],
        padding=True,
        return_tensors="pt",
    )
    inputs = move_to_model_device(inputs, model)

    with torch.inference_mode():
        generated_ids = model.generate(
            **inputs,
            max_new_tokens=32,
            do_sample=False,
        )

    trimmed_ids = generated_ids[:, inputs["input_ids"].shape[1] :]
    prediction = processor.batch_decode(
        trimmed_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]

    print("Question:", question)
    print("Prediction:", prediction)
    print("Reference answer:", sample["multiple_choice_answer"])


if __name__ == "__main__":
    main()

That same prompt shape should stay consistent across training and inference. If you change it later, evaluate whether the adapter is still aligned with the style it saw during fine-tuning.

Failure Modes & Caveats

This tutorial is cleaner than the original draft, but modern VLM fine-tuning still has real edge cases:

• If your LoRA target selection accidentally reaches into the vision stack, training can become harder to reason about.
• If you do not mask the prompt tokens, the loss looks better than it should because the model is rewarded for re-predicting the prompt.
• If you let image resolution grow unchecked, memory usage rises fast.
• VQAv2 answers are short and sometimes noisy, so qualitative inspection still matters.
• A tiny train/test split from merve/vqav2-small is useful for iteration, not for claiming serious benchmark performance.

Practical Guidance

If you want this post to stay useful after one run, keep the workflow boring:

• Start with a very small slice and confirm one batch end to end.
• Inspect the chosen LoRA targets before launching training.
• Keep the answer style short so the task stays close to VQAv2 supervision.
• Freeze the vision tower first; only unfreeze it later if you have a good reason.
• Save the processor beside the adapter so inference keeps the same multimodal formatting assumptions.

If you need to reduce memory pressure further, you have three clean levers before redesigning the whole recipe:

1. Switch from Qwen3-VL-4B-Instruct to Qwen3-VL-2B-Instruct.
2. Reduce batch size and increase gradient accumulation.
3. Lower the processor image resolution constraints.

Sanity check

Before trusting the setup, verify that:

• AutoProcessor resolves for Qwen/Qwen3-VL-4B-Instruct,
• the processor returns input_ids and pixel_values for one image-question example,
• the collator masks the prompt prefix out of labels,
• and the saved adapter reloads into the same base model with the same prompt format.

Summary

The original PaliGemma notebook was not wrong so much as dated. It was tied to a model and workflow that no longer make the best default teaching example for a current VQA fine-tuning post.

Qwen3-VL-4B-Instruct is a better replacement. It is newer, public, supported by transformers==5.12.1, and close enough to the original model size that the QLoRA story still feels practical. Once the notebook clutter is removed, the real recipe is compact: build clean multimodal prompts, freeze the vision tower, apply LoRA to the language projections, mask the prompt-side loss, and keep inference on the same processor path.

Next Steps

• Replace the toy split with a larger VQAv2 or DocumentVQA slice once the training path is stable.
• Compare Qwen3-VL-2B-Instruct, 4B, and 8B if you want a clearer quality-versus-cost tradeoff.
• Add an exact-match or VQA-style scoring pass if you want more than anecdotal validation.

Related reading:

• For a document-heavy variant of VLM tuning, read Train a Modern VLM on One GPU.
• For preference tuning after supervised fine-tuning, read Preference-Aligning Vision Models.

If your VQA workflow needs to work beyond a benchmark slice, talk to us. We can help with data selection, training shape, and evaluation that reflects the questions users actually ask.

Resources

• Qwen3-VL model page
• Qwen3-VL documentation in Transformers
• Transformers multimodal chat templates documentation
• PEFT documentation
• Bitsandbytes quantization documentation
• VQAv2 small dataset page