Fine-Tuning Qwen3.6-VL for Brain Tumor MRI Detection with Transformers 5

TL;DR

• This tutorial replaces an older PaliGemma2 notebook-style workflow with a current transformers 5.x multimodal training pipeline.
• The model stack uses Qwen/Qwen3.6-27B, AutoProcessor, AutoModelForMultimodalLM, and QLoRA through PEFT.
• You will prepare a brain MRI dataset, fine-tune for binary tumor detection (yes/no), and run before/after inference with the same chat-template API.

Abstract

If you want a modern, maintainable way to fine-tune a vision-language model for medical-image triage tasks, older notebook-export workflows quickly become hard to reproduce. This post shows a clean rewrite based on current transformers multimodal APIs and a model family that is newer than the original PaliGemma2 setup. You will convert the brain tumor object-detection dataset into a binary diagnostic prompt/response dataset, train with QLoRA, and run inference with the same processor interface used during training. By the end, you will have a reproducible baseline you can extend to richer labels, larger datasets, and stricter evaluation protocols.

Requirements

• Python 3.10+
• torch
• transformers>=5.12.0
• datasets
• peft
• bitsandbytes
• accelerate
• pillow

pip install -U "transformers>=5.12.0" datasets peft bitsandbytes accelerate pillow

Prerequisites

• You know basic Hugging Face training loops (Trainer, TrainingArguments)
• You are comfortable with GPU memory constraints and LoRA/QLoRA tradeoffs
• You understand this is a research workflow, not a clinical diagnostic system

Problem

The original draft was tied to a notebook export and an older model workflow:

• heavy notebook boilerplate
• duplicated code paths for inference/fine-tuning
• older model-specific classes scattered across cells
• weaker portability to current transformers releases

For a production-quality tutorial, you want:

1. one modern multimodal API surface
2. minimal glue code
3. clear train/infer parity
4. explicit caveats for medical use

Why Qwen3.6-VL Now

To keep the model choice current, official Hugging Face benchmark leaderboards for multimodal reasoning (MMMU/MMMU_Pro) were checked alongside top candidate model metadata on the Hub.

Key candidates:

• Qwen/Qwen3.6-27B: MMMU Pro 75.8, updated 2026-04-24, ~27.8B params
• Qwen/Qwen3.6-35B-A3B: MMMU Pro 75.3, updated 2026-04-24, ~36.0B params
• Qwen/Qwen2.5-VL-7B-Instruct: MMMU Pro 34.3, updated 2025-04-06, ~8.3B params

For this tutorial, Qwen/Qwen3.6-27B is the best “current + strong benchmark” default. If your hardware is tighter, use Qwen/Qwen3-VL-8B-Instruct with the same code pattern.

Approach

The workflow has four stages:

1. Load mmenendezg/brain-tumor-object-detection
2. Convert each sample to a binary prompt/answer pair (yes if tumor label exists, else no)
3. Fine-tune Qwen3.6-VL with QLoRA using chat-template tokenization
4. Compare predictions before and after fine-tuning

This keeps the training target simple and easy to evaluate before moving to localization or richer medical labels.

Key Details

1) Prepare a binary VLM dataset

from datasets import DatasetDict, load_dataset

QUESTION = (
    "You are assisting MRI triage. Does this brain MRI show a tumor? "
    "Answer with exactly one word: yes or no."
)


def to_binary_example(example: dict) -> dict:
    """Convert object labels to a binary yes/no answer."""
    object_labels = [int(x) for x in example["objects"]["label"]]
    has_tumor = any(label == 1 for label in object_labels)
    return {
        "image": example["image"],
        "question": QUESTION,
        "answer": "yes" if has_tumor else "no",
    }


raw_ds: DatasetDict = load_dataset("mmenendezg/brain-tumor-object-detection", name="full")
train_ds = raw_ds["train"].map(to_binary_example)
val_ds = raw_ds["validation"].map(to_binary_example)
test_ds = raw_ds["test"].map(to_binary_example)

2) Load model + processor with QLoRA

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForMultimodalLM, AutoProcessor, BitsAndBytesConfig

MODEL_ID = "Qwen/Qwen3.6-27B"

processor = AutoProcessor.from_pretrained(MODEL_ID)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="sdpa",
)

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj", "gate_proj"],
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.config.use_cache = False
model.print_trainable_parameters()

3) Build a chat-template collator

The collator creates a user+assistant conversation and masks loss on non-assistant tokens.

from typing import Any
import torch

ASSISTANT_HEADER = "<|im_start|>assistant\n"
assistant_tokens = processor.tokenizer.encode(ASSISTANT_HEADER, add_special_tokens=False)


def find_subsequence(haystack: list[int], needle: list[int]) -> int:
    """Return start index of needle in haystack, or -1."""
    n = len(needle)
    for idx in range(len(haystack) - n + 1):
        if haystack[idx : idx + n] == needle:
            return idx
    return -1


def build_conversation(example: dict[str, Any]) -> list[dict[str, Any]]:
    return [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": example["image"]},
                {"type": "text", "text": example["question"]},
            ],
        },
        {
            "role": "assistant",
            "content": [{"type": "text", "text": example["answer"]}],
        },
    ]


def collate_fn(batch: list[dict[str, Any]]) -> dict[str, torch.Tensor]:
    conversations = [build_conversation(example) for example in batch]

    encoded = processor.apply_chat_template(
        conversations,
        tokenize=True,
        add_generation_prompt=False,
        return_dict=True,
        return_tensors="pt",
        processor_kwargs={"padding": True},
    )

    labels = encoded["input_ids"].clone()
    for row_idx, input_ids in enumerate(encoded["input_ids"]):
        tokens = input_ids.tolist()
        start = find_subsequence(tokens, assistant_tokens)
        if start == -1:
            labels[row_idx].fill_(-100)
            continue

        cutoff = start + len(assistant_tokens)
        labels[row_idx, :cutoff] = -100
        labels[row_idx, encoded["attention_mask"][row_idx] == 0] = -100

    encoded["labels"] = labels
    return encoded

4) Train with Trainer

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="qwen3_6_brain_tumor_qlora",
    num_train_epochs=2,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    warmup_ratio=0.03,
    weight_decay=0.01,
    logging_steps=20,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=2,
    bf16=True,
    gradient_checkpointing=True,
    remove_unused_columns=False,
    report_to="none",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    data_collator=collate_fn,
)

trainer.train()
trainer.save_model("qwen3_6_brain_tumor_qlora/final")
processor.save_pretrained("qwen3_6_brain_tumor_qlora/final")

Example Inference (Before vs After Fine-Tuning)

import torch
from peft import PeftModel
from transformers import AutoModelForMultimodalLM, AutoProcessor

BASE_MODEL_ID = "Qwen/Qwen3.6-27B"
FT_MODEL_ID = "qwen3_6_brain_tumor_qlora/final"

processor = AutoProcessor.from_pretrained(BASE_MODEL_ID)

def load_base_model():
    return AutoModelForMultimodalLM.from_pretrained(
        BASE_MODEL_ID,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )

example = test_ds[0]
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": example["image"]},
            {"type": "text", "text": QUESTION},
        ],
    }
]


def predict(model, processor, conv):
    inputs = processor.apply_chat_template(
        conv,
        tokenize=True,
        add_generation_prompt=True,
        return_dict=True,
        return_tensors="pt",
    )
    inputs = {k: v.to(model.device) for k, v in inputs.items()}
    generated = model.generate(**inputs, max_new_tokens=8)
    trimmed = [out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs["input_ids"], generated)]
    return processor.batch_decode(trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0].strip()


print("Ground truth:", example["answer"])
base_model = load_base_model()
print("Base model :", predict(base_model, processor, conversation))
del base_model
torch.cuda.empty_cache()

adapter_base = load_base_model()
ft_model = PeftModel.from_pretrained(adapter_base, FT_MODEL_ID)
print("Fine-tuned :", predict(ft_model, processor, conversation))

Failure Modes & Caveats

• This tutorial predicts binary yes/no, not bounding boxes or segmentation masks.
• MRI domain shift is real: scanners, protocols, and institutions can break generalization quickly.
• Label quality controls matter more than model size if the supervision signal is noisy.
• A high offline score does not imply clinical safety.
• This is not medical advice and must not be used as a standalone diagnostic tool.

Practical Guidance

• Start with a tiny training subset and confirm loss decreases before scaling.
• Keep prompts fixed during initial experiments; prompt drift can invalidate comparisons.
• Log confusion matrix and per-class recall on yes and no, not just aggregate accuracy.
• Save exact package versions with your checkpoints for reproducibility.
• If GPU memory is limited, swap to Qwen/Qwen3-VL-8B-Instruct first, then scale up.

Sanity check

• processor.apply_chat_template(...) returns input_ids, pixel_values, and image_grid_thw.
• Training loss decreases over the first several hundred steps.
• Generated responses are constrained to yes or no for most test samples.
• Fine-tuned output outperforms base output on the same held-out subset.

Summary

This rewrite moves the old PaliGemma2 notebook workflow to a cleaner, current training stack centered on Qwen3.6-VL and transformers 5.x. The pipeline is intentionally simple: binary question answering over MRI images with QLoRA fine-tuning and chat-template-aligned tokenization. That gives you a strong baseline you can trust, extend, and evaluate rigorously.

Next Steps

• Add calibrated confidence estimation and threshold-based abstention behavior.
• Expand from binary labels to structured outputs (tumor type, location hints, uncertainty).
• Add institution-split validation to measure domain robustness explicitly.

Related reading:

• For a general visual question answering fine-tuning pattern, read Modern VQA Fine-Tuning.
• For preference tuning multimodal models after supervised fine-tuning, read Preference-Aligning Vision Models.

If medical or domain-specific vision models are entering a decision workflow, talk to us. We can help define evaluation boundaries, failure cases, and the claims the model should not be allowed to make.

Resources

• Qwen3.5/3.6 Transformers model docs
• Qwen/Qwen3.6-27B model card
• Brain tumor dataset used here
• PEFT documentation
• Transformers multimodal chat templates