Ana içeriğe geç

aMUSEd

Model Source Revision Weight license Parity (CPU, fp32)
amused-256 amused/amused-256 09b6259 OpenRAIL++ Verified
amused-512 amused/amused-512 6f3f309 OpenRAIL++ Verified

Catalog: amused · Models: amused-256, amused-512 · Task: t2i

Architecture. aMUSEd is a masked image-token model, not a diffusion model. A VQGAN with a codebook of 8192 entries turns an image into a grid of token ids at 16× downsampling (16×16 tokens for the 256 model, 32×32 for the 512 model, which downsamples and upsamples the grid inside the network). A U-ViT predicts the ids of masked tokens. It has a ConvNeXt stem, a stack of 22 transformer layers of width 1024 with cross-attention to the text, and a ConvNeXt head, and it is conditioned on the CLIP ViT-L/14 text encoder: the projected pooled embedding and the second-to-last hidden state. Micro-conditioning (original size, crop coordinates and aesthetic score) is added to the pooled embedding. Generation starts from a grid of mask tokens. At every step the transformer predicts all tokens, one id is sampled for each masked position, and the least confident tokens are masked again, following a cosine schedule. The final tokens are looked up in the VQGAN codebook and decoded to pixels.

Usage.

python -m imagehub.pretrained amused-512 checkpoints/amused-512
from imagehub import pipeline

with pipeline("t2i", backend="amused-512", model="checkpoints/amused-512", device="mps") as pipe:
    result = pipe("a photo of an astronaut riding a horse on mars", seed=42)
    result.images[0].save("astronaut.png")

Settings. The models default to 12 steps, guidance 10.0, an empty negative prompt and the size of their token grid (256×256 or 512×512). The supported options are seed, steps, guidance, width, height, negative_prompt, aesthetic_score (default 6), crop_coord (default (0, 0)) and temperature (default (2, 0)). width and height must be multiples of 16 for amused-256 and of 32 for amused-512. temperature is either a number, which falls linearly from that number to 0.01 over the steps, or a (start, end) pair, which is interpolated linearly. It scales the random noise that is added to the token confidences before re-masking. Guidance is applied to the token logits as uncond + guidance * (cond - uncond); guidance 1.0 or lower skips the unconditional pass. The seed drives a CPU generator that is used for the token sampling and the confidence noise, so the same seed gives the same tokens on every device up to numerical differences. Unrecognized settings raise an error.

Memory. In float32 the transformer needs about 2.4 GB, the CLIP text encoder about 0.5 GB and the VQGAN about 0.6 GB.

Seed compatibility. On CPU, the same seed and settings give the same token ids and the same image as diffusers' AmusedPipeline with generator=torch.Generator("cpu").manual_seed(seed) (validation). MPS and CUDA runs are smoke tests only; there is no claim of exact equality on these devices.

Out of scope. LoRA, fp16/bf16 parity, batches of several prompts or images per call, image inpainting and variation, and caller-supplied tokens or embeddings are not supported.

License. Both repositories declare the OpenRAIL++ license; check its use restrictions before redistributing generated images or weights. imagehub does not distribute weights.

Reference and imagehub, side by side

Drag the slider. Left is the recorded reference, right is imagehub, from the same seed, on CPU in float32.