Skip to content

LlamaGen

Model Source Revision Weight license Parity (CPU, fp32)
llamagen-t2i-xl-256 peizesun/llamagen_t2i 276f5c5 MIT Verified
llamagen-t2i-xl-512 peizesun/llamagen_t2i 276f5c5 MIT Verified

Catalog: llamagen · Models: llamagen-t2i-xl-256, llamagen-t2i-xl-512 · Task: t2i

Architecture. LlamaGen is an autoregressive image model: the image is a grid of discrete tokens and a decoder-only transformer predicts them one at a time, like a language model. A VQ-16 tokenizer (a codebook of 16384 entries of dimension 8, 16× downsampling) maps between tokens and pixels, so the 256 model writes 16×16 = 256 tokens and the 512 model 32×32 = 1024 tokens. The transformer (GPT-XL: 36 layers, width 1280, 20 heads, RMSNorm, SwiGLU, two-dimensional rotary position embeddings over the token grid) is conditioned on the prompt by a prefix: the prompt is encoded by the Flan-T5-XL encoder into 120 vectors of width 2048, a small MLP projects them to the model width, and they are placed in front of the image tokens. Each new token is drawn from the softmax of the logits after temperature, top-k and top-p filtering. Classifier-free guidance runs a second pass in which the caption is replaced by a learned "no caption" embedding and combines the logits as uncond + cfg_scale * (cond - uncond).

Usage.

python -m imagehub.pretrained llamagen-t2i-xl-256 checkpoints/llamagen-t2i-xl-256
from imagehub import pipeline

with pipeline(
    "t2i", backend="llamagen-t2i-xl-256", model="checkpoints/llamagen-t2i-xl-256", device="mps"
) as pipe:
    result = pipe("a blue Porsche 356 parked in front of a yellow brick wall", seed=42)
    result.images[0].save("porsche.png")

The snapshot holds vq_ds16_t2i.pt and the GPT checkpoint (t2i_XL_stage1_256.pt or t2i_XL_stage2_512.pt) at its root, and the Flan-T5-XL files (config, tokenizer and the two safetensors shards) in text_encoder/. The two image sizes are separate checkpoints; the size is fixed by the checkpoint.

Settings. The defaults are those of the official sampling script: cfg_scale 7.5, temperature 1.0, top_k 1000, top_p 1.0 and cfg_interval -1. The supported options are seed, cfg_scale, temperature, top_k, top_p, cfg_interval, width and height. width and height can only repeat the size of the checkpoint (256 or 512). Guidance is used only when cfg_scale is above 1.0. With cfg_interval of n (0 or larger), guidance is dropped for every token after the first n + 2 generated tokens; -1 keeps it for all. top_k of 0 and top_p of 1.0 disable the respective filter. Unrecognized settings raise an error.

The seed starts a CPU generator that supplies every random draw, in the same order as the reference after torch.manual_seed(seed) immediately before sampling, so the same seed gives the same tokens on every device up to numerical differences. Images have a batch size of one.

Prompt handling. The prompt is lower-cased and stripped and encoded to 120 tokens (longer prompts are truncated), which is what the reference's text embedder does when its text preprocessing is switched off. The reference enables that preprocessing by default: it also removes URLs, HTML, some punctuation, file names and similar noise with the ftfy and BeautifulSoup packages. imagehub does not implement this caption cleaning, so a prompt that contains such text can give different tokens from the default reference script; plain prompts give identical tokens.

Memory. In float32 the GPT needs about 3.1 GB, the Flan-T5-XL encoder about 4.9 GB (the decoder in the downloaded shards is never loaded) and the VQ decoder about 0.3 GB. The download holds the full Flan-T5-XL checkpoint, about 11 GB.

Seed compatibility. On CPU, the same seed and settings give the same token ids and the same image as the official sample_t2i.py driver run on CPU in float32 (validation). MPS and CUDA runs are smoke tests only; there is no claim of exact equality on these devices.

Out of scope. The class-conditional (ImageNet) checkpoints, the caption cleaning described above, torch.compile, fp16/bf16 parity, several prompts or images per call, and caller-supplied tokens or caption embeddings are not supported.

License. The repository declares the MIT license and the text encoder is Apache-2.0. imagehub does not distribute weights.