aMUSEd¶
| Model | Source | Revision | Weight license | Parity (CPU, fp32) |
|---|---|---|---|---|
amused-256 |
amused/amused-256 | 09b6259 |
OpenRAIL++ | Verified |
amused-512 |
amused/amused-512 | 6f3f309 |
OpenRAIL++ | Verified |
Catalog: amused ·
Models: amused-256, amused-512 · Task: t2i
Architecture. aMUSEd is a masked image-token model, not a diffusion model. A VQGAN with a codebook of 8192 entries turns an image into a grid of token ids at 16× downsampling (16×16 tokens for the 256 model, 32×32 for the 512 model, which downsamples and upsamples the grid inside the network). A U-ViT predicts the ids of masked tokens. It has a ConvNeXt stem, a stack of 22 transformer layers of width 1024 with cross-attention to the text, and a ConvNeXt head, and it is conditioned on the CLIP ViT-L/14 text encoder: the projected pooled embedding and the second-to-last hidden state. Micro-conditioning (original size, crop coordinates and aesthetic score) is added to the pooled embedding. Generation starts from a grid of mask tokens. At every step the transformer predicts all tokens, one id is sampled for each masked position, and the least confident tokens are masked again, following a cosine schedule. The final tokens are looked up in the VQGAN codebook and decoded to pixels.
Usage.
from imagehub import pipeline
with pipeline("t2i", backend="amused-512", model="checkpoints/amused-512", device="mps") as pipe:
result = pipe("a photo of an astronaut riding a horse on mars", seed=42)
result.images[0].save("astronaut.png")
Settings. The models default to 12 steps, guidance 10.0, an empty negative prompt and the size
of their token grid (256×256 or 512×512). The supported options are seed, steps, guidance,
width, height, negative_prompt, aesthetic_score (default 6), crop_coord (default (0, 0))
and temperature (default (2, 0)). width and height must be multiples of 16 for amused-256
and of 32 for amused-512. temperature is either a number, which falls linearly from that number
to 0.01 over the steps, or a (start, end) pair, which is interpolated linearly. It scales the random
noise that is added to the token confidences before re-masking. Guidance is applied to the token
logits as uncond + guidance * (cond - uncond); guidance 1.0 or lower skips the unconditional pass.
The seed drives a CPU generator that is used for the token sampling and the confidence noise, so
the same seed gives the same tokens on every device up to numerical differences. Unrecognized
settings raise an error.
Memory. In float32 the transformer needs about 2.4 GB, the CLIP text encoder about 0.5 GB and the VQGAN about 0.6 GB.
Seed compatibility. On CPU, the same seed and settings give the same token ids and the same image
as diffusers' AmusedPipeline with generator=torch.Generator("cpu").manual_seed(seed)
(validation). MPS and CUDA runs are smoke tests only; there is no claim of exact
equality on these devices.
Out of scope. LoRA, fp16/bf16 parity, batches of several prompts or images per call, image inpainting and variation, and caller-supplied tokens or embeddings are not supported.
License. Both repositories declare the OpenRAIL++ license; check its use restrictions before redistributing generated images or weights. imagehub does not distribute weights.
Reference and imagehub, side by side¶
Drag the slider. Left is the recorded reference, right is imagehub, from the same seed, on CPU in float32.

referenceimagehub

referenceimagehub