Skip to content

LoRA training

imagehub.training fine-tunes a LoRA adapter on your own images. It ships its own LoRA layers and training loop (no PEFT or Accelerate) and is an optional install; the core package stays PyTorch-only.

1. Install and download

pip install "imagehub[training]"
python -m imagehub.pretrained sd-v1-5 checkpoints/sd-v1-5

2. Prepare the images

Put the images in one folder with a metadata.jsonl file that has one line per image:

my_images/
  metadata.jsonl
  0001.png
  0002.jpg
{"file_name": "0001.png", "text": "a photo of sks dog on the beach"}
{"file_name": "0002.jpg", "text": "a photo of sks dog in the snow"}

Missing files, empty captions and unreadable images are reported when the dataset is created. Any sized dataset whose items are {"image": PIL image, "text": str} also works, for example a Hugging Face datasets dataset; imagehub does not depend on that package.

3. Train

from imagehub.training import ImageFolderDataset, Trainer, TrainingArguments

trainer = Trainer(
    model="sd-v1-5",
    checkpoint="checkpoints/sd-v1-5",
    args=TrainingArguments(output_dir="runs/my-dog", max_steps=1000),
    train_dataset=ImageFolderDataset("my_images"),
)
result = trainer.train()
print(result.training_loss)
trainer.save_model("runs/my-dog/final")

The loss is printed every logging_steps. Every save_steps steps, and at the end, the trainer writes output_dir/checkpoint-N/ with the adapter and the full training state. If a run stops, continue it from the latest checkpoint:

trainer.train(resume_from_checkpoint=True)

A resumed run gives exactly the same adapter as an uninterrupted one.

4. Use the adapter

from imagehub import pipeline

with pipeline(
    "t2i",
    backend="sd-v1-5",
    model="checkpoints/sd-v1-5",
    lora="runs/my-dog/final/adapter.safetensors",
) as pipe:
    image = pipe("a photo of sks dog in a bucket", seed=0).images[0]

adapter.safetensors uses the diffusers layout, so diffusers load_lora_weights reads it too.

Evaluation and sample images

trainer = Trainer(
    model="sd-v1-5",
    checkpoint="checkpoints/sd-v1-5",
    args=TrainingArguments(
        output_dir="runs/my-dog",
        max_steps=1000,
        eval_strategy="steps",
        eval_steps=200,
        validation_prompts=["a photo of sks dog in a bucket"],
    ),
    train_dataset=ImageFolderDataset("my_images"),
    eval_dataset=ImageFolderDataset("my_val_images"),
)
trainer.train()
print(trainer.evaluate())  # {"eval_loss": 0.1234}

eval_loss is the training loss without gradients over the evaluation images. Diffusion loss depends on the random noise and timestep, so these are seeded by image position and the images are center-cropped: evaluating the same adapter twice gives the same number, and the curve over training follows the adapter, not the dice. At every evaluation point (and at the end) each validation prompt is rendered with the current adapter to output_dir/samples/step-N/<i>.png, exactly as pipeline(..., lora=...) would render it with seed, validation_num_inference_steps and validation_guidance_scale.

Custom batches

data_collator turns a list of dataset items into a batch. The default resizes, crops, flips and normalizes the images and tokenizes the captions. A custom one returns {"pixel_values": (B, 3, H, W) in [-1, 1], "input_ids": (B, 77)}:

trainer = Trainer(..., data_collator=my_collator)

Exact resume is guaranteed for the default collator, whose random state the checkpoint holds.

Several GPUs

Run the same script with torchrun; nothing in it changes:

torchrun --nproc_per_node=4 train.py

Each process trains on its own share of every batch and the adapter gradients are averaged, so all processes keep the same adapter. The effective batch is per_device_train_batch_size × gradient_accumulation_steps × number of processes. Only the first process prints, writes checkpoints, samples and save_model output. A checkpoint resumes with the same number of processes. Distributed runs use CUDA (NCCL) or the CPU (gloo); MPS cannot.

Settings

Only output_dir and max_steps are required. The defaults follow diffusers' train_text_to_image_lora.py.

Argument Default Meaning
learning_rate 1e-4 AdamW learning rate
per_device_train_batch_size, per_device_eval_batch_size 1, 1 images per micro-batch on each process
gradient_accumulation_steps 1 micro-batches per optimizer step
lr_scheduler, lr_warmup_steps "constant", 0 also constant_with_warmup, linear, cosine
max_grad_norm 1.0 gradient clipping; 0 turns it off
adam_beta1, adam_beta2, adam_weight_decay, adam_epsilon 0.9, 0.999, 1e-2, 1e-8 AdamW settings
mixed_precision "no" "fp16" or "bf16"; the adapter and optimizer stay float32
resolution, center_crop, random_flip 512, False, False image transform
lora_rank, lora_alpha, lora_dropout model's script default, model's script default, 0.0 adapter size and scale (alpha / rank); 4 / 4 except Qwen-Image 2.1 (16 / 16) and Krea 2 (32 / 32)
lora_target_modules the family's layers that get an adapter: to_q, to_k, to_v, to_out.0 for U-Nets, Qwen-Image, Z-Image and Lumina 2; FLUX.2 klein also uses to_qkv_mlp_proj and its literal first-24 single-block attn.to_out paths; to_q, to_k, to_v, to_out for HiDream-I1 image attention; the eight joint-attention projections for SD 3; and to_k, to_q, to_v for Sana
gradient_checkpointing False recompute U-Net or transformer blocks in the backward pass to save memory
noise_offset, snr_gamma, prediction_type 0.0, None, the scheduler's U-Net (DDPM) objective options
weighting_scheme, logit_mean, logit_std, mode_scale the family's ("logit_normal" for SD 3, "none" for FLUX.1, FLUX.2 klein, Krea 2, Sana, Qwen-Image, Z-Image, Lumina 2 and HiDream-I1), 0.0, 1.0, 1.29 flow-matching timestep sampling: also mode, sigma_sqrt, cosmap, none
precondition_outputs True flow matching: compare the preconditioned output with the latents instead of noise - latents
max_sequence_length 77 (SD 3), 512 (FLUX.1, FLUX.2 klein, Krea 2, Qwen-Image and Z-Image), 300 (Sana), 256 (Lumina 2), 128 (HiDream-I1) text prompt length
system_prompt None Lumina 2's optional Gemma system prompt; other families reject it
complex_human_instruction None Sana's optional prompt-enhancement instruction; other families reject it
guidance_scale 3.5 FLUX.1 and FLUX.2 klein: the guidance embedding's value in training, on checkpoints that have one
logging_steps, save_steps 10, 500 how often to log and to checkpoint
eval_strategy, eval_steps "no", logging_steps "steps" evaluates every eval_steps
validation_prompts none prompts rendered at each evaluation point
validation_num_inference_steps, validation_guidance_scale 25, 7.5 sampling settings for those images
report_to, run_name "none", None "wandb", "clearml" or both as a list: logs train_loss, lr, eval_loss and the sample images (the package must be installed)
seed, device 0, "cpu" random seed; "cuda" or "mps" to train on a GPU

Choosing rank and alpha

The adapter's update is scaled by lora_alpha / lora_rank. Diffusion tools use small ranks and scales near 1, while language and speech trainers tend to use larger ones:

Source Rank Alpha Scale
imagehub default, diffusers training scripts (SD 1.5, SDXL, FLUX.1) 4 4 1
kohya sd-scripts default 4 1 0.25
PEFT default 8 8 1

Rank 4 is enough for a style or a simple subject. For more detail, raise the rank to 16-32 and keep alpha equal to the rank, so the scale and a good learning rate stay the same.

Supported models

get_training_spec(model_id) tells whether a model can be trained; list_training_specs() lists every catalog model.

from imagehub.training import get_training_spec

spec = get_training_spec("sd-v1-5")
print(spec.support.value, spec.methods)  # native ('lora',)

native means the model trains from images and captions with an objective checked against the reference training script; inference_only means there is no training path yet. methods lists what can be trained (lora today).

Model Support Methods
qwen-image native lora
sd-v1-1 native lora
sd-v1-2 native lora
sd-v1-3 native lora
sd-v1-4 native lora
sd-v1-5 native lora
dreamshaper-8 native lora
segmind-small-sd native lora
segmind-tiny-sd native lora
bk-sdm-base native lora
bk-sdm-small native lora
bk-sdm-tiny native lora
sd-v2-1 native lora
bk-sdm-v2-base native lora
bk-sdm-v2-small native lora
bk-sdm-v2-tiny native lora
sdxl-base-1.0 native lora
juggernaut-xl-v9 native lora
animagine-xl-4.0 native lora
dreamshaper-xl-1.0 native lora
z-image native lora
qwen-image-2.1 native lora
sd3-medium native lora
sd3.5-medium native lora
sd3.5-large native lora
flux2-klein-4b native lora
flux2-klein-9b native lora
flux1-dev native lora
flux1-krea-dev native lora
flux-1-lite-8b native lora
flux1-kontext-dev native lora
sana-1600m-1024 native lora
sana-600m-1024 native lora
sana-1.5-4.8b-1024 native lora
lumina-image-2.0 native lora
krea2-turbo native lora
ssd-1b native lora
segmind-vega native lora
koala-700m native lora
koala-1b native lora
playground-v2 native lora
hidream-i1-dev native lora
hidream-i1-full native lora

Every other model is inference_only for now: the few-step distilled checkpoints (sd-turbo, lcm-dreamshaper-v7, sdxs-512-0.9, SDXL Lightning, Hyper-SD, DMD2, sdxl-turbo, KOALA Lightning), the SDXL refiner (image to image), Playground v2.5 (EDM, not DDPM) and sd3.5-large-turbo (few-step distilled), sana-sprint-1.6b-1024 (distilled), and the Qwen-Image-Edit checkpoints, which are conditioned on an input image and have no diffusers training script.

Sana trains its transformer only, using the script's to_k, to_q, to_v targets. Its DC-AE has deterministic latents, so the training random stream draws noise then the flow timestep; the autoencoder stays float32 under mixed precision. Gemma 2 prompt conditioning uses Sana's attention mask and 300-token default; complex_human_instruction is optional and defaults to None as in the script. Gradient checkpointing recomputes the Sana transformer blocks.

Qwen-Image and Qwen-Image 2.1 train transformer-only LoRA adapters on to_k, to_q, to_v and to_out.0. Both use the flow target noise - latents, their scripts' "none" density default and VAE posterior normalization. Original Qwen-Image uses its 512-token descriptive chat prompt, a one-frame latent and 2×2 spatial packing; 2.1 removes its system prompt tokens and trains the joint prompt/image sequence with its image-slot mask. max_sequence_length=512 applies to the original release only. When rank and alpha are unset, Qwen-Image uses its script's 4 / 4 defaults and Qwen-Image 2.1 uses 16 / 16. Both backends load transformer adapters through lora=, lora_scale= and lora_fuse=.

Krea 2 trains transformer-only adapters with rank / alpha 32 / 32 and the script's "none" weighting, 512-token prompt and 5-D posterior-latent packing. Turbo itself is an 8-step distilled checkpoint and is never trained. Accept the Krea 2 Community License, then download the gated krea/Krea-2-Raw training checkpoint at revision 6b0ece7fffb640c5e3bcbe0a7f10f66b8e60a603; adapters trained on Raw are bound by that license and load on Turbo through lora=, lora_scale= and lora_fuse=. Validation uses the training snapshot with 8 steps and guidance 0.

Ideogram 4 remains inference_only because every official checkpoint is quantized. Its fp8 backend nevertheless loads transformer adapters with lora=, lora_scale= and lora_fuse=. Diffusers-layout transformer files use attention.to_q, to_k, to_v and to_out.0; the backend maps the three split attention updates onto the corresponding row blocks of its fused attention.qkv weight and maps to_out.0 to attention.o. Fusion first dequantizes the fp8 weight to the compute dtype, then applies the same updates.

Z-Image trains transformer-only LoRA adapters on to_k, to_q, to_v and to_out.0, with the script's 4 / 4 rank and alpha, "none" weighting and 512-token chat-template prompts. Its KL VAE uses the deterministic posterior mean, then applies the published shift and scale before the flow step. The model receives (1000 - timestep) / 1000 and the negated prediction is compared with noise - latents; gradient checkpointing recomputes every Z-Image transformer block. Its backend loads transformer adapters with lora=, lora_scale= and lora_fuse=. z-image-turbo is distilled and remains inference_only.

FLUX.2 klein 4B and 9B train transformer-only LoRA adapters with the script's literal targets: to_k, to_q, to_v, to_out.0, to_qkv_mlp_proj, and single_transformer_blocks.0 through .23 attn.to_out. Their rank / alpha defaults are 4 / 4, weighting is "none", and Qwen prompts use 512 tokens and layers 9, 18 and 27, the layers the official FLUX.2 code and the klein pipeline use at inference. The training script's --text_encoder_out_layers default, 10, 20 and 30, is the FLUX.2 [dev] Mistral set; the reference test runs the script with 9, 18 and 27. The deterministic VAE posterior is patchified and normalized with the VAE batch-normalization statistics before flow matching against noise - latents; gradient checkpointing covers both double- and single-stream blocks. A guidance value is passed only to checkpoints with a guidance embedding; the other klein checkpoints accept only the default training value. The backend loads transformer adapters with lora=, lora_scale= and lora_fuse=.

Lumina-Image 2.0 trains transformer-only LoRA adapters on to_k, to_q, to_v and to_out.0, with the script's 4 / 4 rank and alpha, "none" weighting, a 256-token Gemma prompt and its default system prompt. Its float32 KL VAE posterior is shifted and scaled before reversed flow matching: the transformer receives timestep / 1000 and predicts latents - noise. Gradient checkpointing recomputes Lumina 2 transformer blocks. Its backend loads transformer adapters with lora=, lora_scale= and lora_fuse=; Lumina-Next and Lumina-DiMOO remain inference_only. Reference tests exactly compare the script defaults, image transform, default-system-prompt token ids and attention mask, per-weighting step draws/loss/gradients, and random or three-step-trained adapter final latents and uint8 pixels after diffusers loading.

HiDream-I1 Dev and Full train transformer-only LoRA adapters on the image-stream to_k, to_q, to_v and to_out projections, with rank / alpha 4 / 4, "none" weighting and a 128-token prompt for both T5 and Llama. All four text encoders and the VAE use the training weight dtype. The sampled, shifted and scaled VAE latent is mixed as (1 - sigma) * latents + sigma * noise; the raw timestep is passed to the inference-MoE transformer and its negated prediction is compared with noise - latents. Gradient checkpointing covers both double- and single-stream blocks. The backend loads transformer adapters through lora=, lora_scale= and lora_fuse=. Reference tests load random single, stacked (0.8 / 0.5) and fused adapters, plus a 3-step trained adapter, in diffusers and compare registered targets, changed outputs, final latents and uint8 pixels from the same explicit starting latents. hidream-i1-fast is distilled and remains inference_only.

For SDXL the transform resizes, flips and then crops, as diffusers' SDXL script does, and records each image's original size and crop corner as the U-Net's size conditioning. The autoencoder stays in float32 under mixed precision. gradient_checkpointing=True recomputes the U-Net's transformer and resnet blocks in the backward pass to save memory; the gradients are the same.

SD 3 / 3.5 train with the flow-matching objective of diffusers' train_dreambooth_lora_sd3.py: timesteps drawn by weighting_scheme from the shifted schedule, the noisy latents (1 - sigma) * latents + sigma * noise, and preconditioned outputs compared with the latents. The prompt joins the CLIP-L and CLIP-G penultimate states with T5, as at inference. DDPM options (snr_gamma, prediction_type, noise_offset) are refused for SD 3, and the flow-matching options for the U-Net families.

FLUX.1 dev, Krea dev and Lite train with the corresponding diffusers FLUX DreamBooth step: float32 VAE latents are shifted and scaled, packed with FLUX image ids, and compared directly with noise - latents. The CLIP pooled projection and T5 prompt use a 512-token default. The guidance embedding uses guidance_scale only on checkpoints that expose it. The adapter targets cover both double-stream and single-stream blocks. flux1-kontext-dev trains the same step, as the Kontext script does without condition images, and also targets the single-stream proj_mlp layers; image-conditioned Kontext training is not included. flux1-schnell is few-step distilled and stays inference_only.

trainer = Trainer(
    model="sd3.5-medium",
    checkpoint="checkpoints/sd3.5-medium",
    args=TrainingArguments(
        output_dir="runs/my-dog-sd35",
        max_steps=1000,
        resolution=1024,
        mixed_precision="bf16",
        gradient_checkpointing=True,
        weighting_scheme="logit_normal",
    ),
    train_dataset=ImageFolderDataset("my_images"),
)

What is checked

On CPU in float32, the first step's loss and every LoRA gradient equal the diffusers training step exactly, on tiny models and on sd-v1-5; four trainer steps with warmup, a cosine schedule and clipping equal a diffusers + PEFT loop; and a saved adapter gives the same image in imagehub and diffusers. See the validation record.