Preference-Aligning Vision Models: DPO Fine-Tuning Qwen3.5-4B in Transformers 5.12.1

TL;DR

• The original SmolVLM draft was a raw notebook export with outdated dependencies and noisy structure.
• A better current baseline for this workflow is Qwen/Qwen3.5-4B: newer than early SmolVLM checkpoints, broadly adopted, and supported by the modern multimodal API in transformers==5.12.1.
• For vision-language DPO with TRL, keep your dataset in images, prompt, chosen, and rejected format, pass an AutoProcessor via processing_class, and avoid text-style truncation that can drop image tokens.

Abstract

Preference tuning is where multimodal models start behaving less like generic captioners and more like assistants with judgment. Instead of training on one “correct” answer, Direct Preference Optimization (DPO) trains on comparisons: which answer is better for the same image and prompt.

This post rebuilds an older SmolVLM notebook into a clean Markdown tutorial using a newer model (Qwen/Qwen3.5-4B), current transformers APIs, TRL’s DPOTrainer, and PEFT LoRA adapters. By the end, you will have a practical training script and an adapter inference check you can adapt to your own preference data.

Requirements

Use this setup if you want to run the examples as written:

• python>=3.10
• torch (your CUDA/CPU build)
• transformers==5.12.1
• trl>=0.19.0
• peft>=0.13.0
• accelerate>=1.0.0
• datasets
• Pillow
• bitsandbytes (optional but recommended for 4-bit loading)

Install PyTorch first with the selector on pytorch.org, then:

pip install "transformers==5.12.1" "trl>=0.19.0" "peft>=0.13.0" "accelerate>=1.0.0" datasets pillow bitsandbytes

Prerequisites

• You already know basic Hugging Face model loading and dataset mapping.
• You have a GPU if you want practical training speed.
• You are comfortable with preference data shaped like prompt + chosen + rejected.

Table of Contents

• Problem
• Why This Model
• Background
• Approach
• End-to-End Example
• Inference with the Adapter
• Failure Modes and Caveats
• Practical Guidance
• Summary
• Next Steps
• Resources

Problem

The old draft had good intent but bad shape:

• it embedded notebook metadata and binary noise directly in the source,
• it pinned an older Transformers version,
• and it mixed reusable training logic with notebook-specific clutter.

That combination makes a tutorial hard to trust, hard to maintain, and hard to reuse.

The right update is not cosmetic. You need a cleaner model baseline, current APIs, and a script structure that you can actually run outside a notebook.

Why This Model

For this rewrite, the target changed from an older SmolVLM checkpoint to Qwen/Qwen3.5-4B.

Why this one:

1. It is newer than early SmolVLM and Qwen2.5-era baselines.
2. It is strong enough to be interesting but still realistic for LoRA-style adaptation.
3. It uses the current multimodal path (AutoProcessor + AutoModelForMultimodalLM) in Transformers 5.x.
4. It keeps the tutorial practical instead of drifting into huge-model-only territory.

There are larger and newer options on the Hub, but a good tutorial chooses a model people can iterate on, not just admire on a benchmark card.

Background

DPO trains from pairwise preferences.

Each row contains:

• one prompt (for VLMs: prompt + image),
• one preferred completion (chosen),
• one less preferred completion (rejected).

For multimodal DPO in TRL, your examples should preserve multimodal structure. A good default is:

• images: list of PIL.Image
• prompt: chat-style user turn including image placeholder
• chosen: assistant completion
• rejected: assistant completion

One crucial VLM detail: avoid text-only truncation habits. Setting max_length too aggressively can cut image tokens and quietly degrade training.

Approach

This workflow stays intentionally compact:

1. Load a preference dataset (openbmb/RLHF-V-Dataset in this tutorial).
2. Convert rows into VLM DPO format.
3. Load Qwen/Qwen3.5-4B with an AutoProcessor.
4. Add LoRA adapters to language-side projection modules.
5. Train with DPOTrainer.
6. Save and run a quick adapter inference check.

End-to-End Example

The script below is the cleaned replacement for the original notebook export.

from __future__ import annotations

import json
from typing import Any, Final

import torch
from datasets import Dataset, load_dataset
from peft import LoraConfig
from PIL import Image
from transformers import AutoProcessor, BitsAndBytesConfig
from trl import DPOConfig, DPOTrainer


MODEL_ID: Final[str] = "Qwen/Qwen3.5-4B"
DATASET_ID: Final[str] = "openbmb/RLHF-V-Dataset"
OUTPUT_DIR: Final[str] = "qwen3-5-4b-rlhf-v-dpo-lora"
MAX_IMAGE_EDGE: Final[int] = 896
SEED: Final[int] = 42


def pick_dtype() -> torch.dtype:
    """Choose a practical compute dtype for available hardware."""
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported():
        return torch.bfloat16
    if torch.cuda.is_available():
        return torch.float16
    return torch.float32


def resize_image(image: Image.Image) -> Image.Image:
    """Normalize image mode and cap the largest edge."""
    image = image.convert("RGB")
    image.thumbnail((MAX_IMAGE_EDGE, MAX_IMAGE_EDGE), Image.Resampling.LANCZOS)
    return image


def format_preference_example(example: dict[str, Any]) -> dict[str, Any]:
    """Convert RLHF-V row into VLM DPO format."""
    payload = json.loads(example["text"])
    question = payload["question"].strip()
    chosen = payload["chosen"].strip()
    rejected = payload["rejected"].strip()

    prompt = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": question},
            ],
        }
    ]

    return {
        "images": [resize_image(example["image"])],
        "prompt": prompt,
        "chosen": [{"role": "assistant", "content": [{"type": "text", "text": chosen}]}],
        "rejected": [{"role": "assistant", "content": [{"type": "text", "text": rejected}]}],
    }


def load_preference_splits() -> tuple[Dataset, Dataset]:
    """Load small train/eval slices for a first alignment run."""
    train_dataset, eval_dataset = load_dataset(
        DATASET_ID,
        split=["train[:1000]", "train[-100:]"],
    )

    train_dataset = train_dataset.map(
        format_preference_example,
        remove_columns=train_dataset.column_names,
        desc="Formatting train preferences",
    )
    eval_dataset = eval_dataset.map(
        format_preference_example,
        remove_columns=eval_dataset.column_names,
        desc="Formatting eval preferences",
    )
    return train_dataset, eval_dataset


def main() -> None:
    train_dataset, eval_dataset = load_preference_splits()
    dtype = pick_dtype()

    processor = AutoProcessor.from_pretrained(MODEL_ID)
    processor.tokenizer.padding_side = "left"
    if processor.tokenizer.pad_token is None:
        processor.tokenizer.pad_token = processor.tokenizer.eos_token

    model_init_kwargs: dict[str, Any] = {
        "dtype": dtype,
        "device_map": "auto" if torch.cuda.is_available() else None,
    }
    if torch.cuda.is_available():
        model_init_kwargs["quantization_config"] = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_use_double_quant=True,
            bnb_4bit_compute_dtype=torch.bfloat16 if dtype == torch.bfloat16 else torch.float16,
        )

    peft_config = LoraConfig(
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
            "gate_proj",
            "up_proj",
            "down_proj",
        ],
        init_lora_weights="gaussian",
    )

    training_args = DPOConfig(
        output_dir=OUTPUT_DIR,
        seed=SEED,
        bf16=dtype == torch.bfloat16,
        fp16=dtype == torch.float16,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=8,
        gradient_checkpointing=True,
        learning_rate=1e-5,
        max_steps=200,
        logging_steps=5,
        eval_strategy="steps",
        eval_steps=50,
        save_strategy="steps",
        save_steps=50,
        save_total_limit=2,
        remove_unused_columns=False,
        max_length=None,
        report_to="none",
        model_init_kwargs=model_init_kwargs,
    )

    trainer = DPOTrainer(
        model=MODEL_ID,
        ref_model=None,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        peft_config=peft_config,
        processing_class=processor,
    )

    trainer.train()
    trainer.save_model(OUTPUT_DIR)
    processor.save_pretrained(OUTPUT_DIR)


if __name__ == "__main__":
    main()

Two details make this script more robust than the old draft:

• max_length=None avoids accidental image-token truncation in multimodal DPO.
• The dataset keeps structured message content instead of flattening everything into plain text too early.

Inference with the Adapter

After training, load the base model, attach the adapter, and run one sample.

from __future__ import annotations

import json

import torch
from datasets import load_dataset
from peft import PeftModel
from transformers import AutoModelForMultimodalLM, AutoProcessor


MODEL_ID = "Qwen/Qwen3.5-4B"
ADAPTER_DIR = "qwen3-5-4b-rlhf-v-dpo-lora"


def pick_dtype() -> torch.dtype:
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported():
        return torch.bfloat16
    if torch.cuda.is_available():
        return torch.float16
    return torch.float32


def main() -> None:
    dtype = pick_dtype()
    processor = AutoProcessor.from_pretrained(ADAPTER_DIR)
    model = AutoModelForMultimodalLM.from_pretrained(
        MODEL_ID,
        dtype=dtype,
        device_map="auto" if torch.cuda.is_available() else None,
    )
    model = PeftModel.from_pretrained(model, ADAPTER_DIR)
    if not torch.cuda.is_available():
        model = model.to("cpu")
    model.eval()

    row = load_dataset("openbmb/RLHF-V-Dataset", split="train[-1]")[0]
    payload = json.loads(row["text"])
    question = payload["question"].strip()
    image = row["image"].convert("RGB")

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": question},
            ],
        }
    ]

    text = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=False,
    )

    inputs = processor(text=text, images=[image], return_tensors="pt")
    inputs = {key: value.to(model.device) for key, value in inputs.items()}

    with torch.inference_mode():
        output_ids = model.generate(
            **inputs,
            do_sample=False,
            max_new_tokens=128,
        )

    generated_ids = output_ids[:, inputs["input_ids"].shape[-1] :]
    answer = processor.batch_decode(
        generated_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0].strip()

    print("Question:", question)
    print("Generated:", answer)


if __name__ == "__main__":
    main()

Failure Modes and Caveats

DPO can still fail even when code runs.

Common pitfalls:

• weak preference pairs (easy chosen vs obviously broken rejected),
• hidden truncation of multimodal context,
• over-reading one scalar loss as proof of alignment quality,
• and assuming public preference data transfers directly to your product behavior.

Small models can align surprisingly well on narrow tasks, but they still hit limits on deep visual reasoning and long-context grounding.

Practical Guidance

Keep iteration tight:

• Start with 100-1,000 examples and overfit on purpose to validate data formatting.
• Save fixed qualitative prompts and compare base vs adapter every checkpoint.
• Tune LoRA rank or training steps only after the pipeline is correct.
• Move to your own preference data early; public datasets are for mechanics, not product fit.

Sanity check

Before you trust results, verify:

• AutoProcessor.from_pretrained("Qwen/Qwen3.5-4B") resolves,
• one image+prompt sample tokenizes with pixel_values present,
• DPOTrainer starts with your processed dataset,
• and adapter inference uses the same prompt style as training.

Summary

The biggest improvement over the old SmolVLM draft is not a minor syntax update. It is a structural rewrite: cleaner data shape, current multimodal APIs, and a model choice that is both modern and practical for iterative alignment work.

If you keep the pipeline simple and the preference data honest, DPO gives you a direct path to improving vision-language behavior without building a separate reward model stack.

Next Steps

• Replace the public RLHF-V slice with preference pairs from your own application logs.
• Add a small evaluation harness with blind human preference votes.
• Compare this 4B adapter workflow against a smaller 2B or larger 8B model for your latency/cost envelope.

Related reading:

• For a smaller-model DPO path, read Human Preference Tuning for Small VLMs.
• For a supervised VQA baseline before preference tuning, read Modern VQA Fine-Tuning.

If you are using preference data to change model behavior, talk to us. We can help design evaluations that show whether the alignment step improved the right behavior or just moved the failure around.

Resources

• Qwen3.5-4B model page
• Transformers multimodal chat template docs
• TRL DPOTrainer docs
• PEFT LoRA docs
• Bitsandbytes quantization docs
• RLHF-V dataset