• Diffusion (DDPM)

  • Fréchet inception distance (FID) score: lower is better

    • Compare the activations of

  • DALL-E 3, openAI 2023

    • Recaptioning technique
    • Key hypothesis: existing models struggle to follow detailed image descriptions and often ignore words or confuse prompt meanings. Just need more detailed and accurate image captions to train. Our ability to generate captions should be able to help.
    • At sample time: use GPT-4 to “upsample” prompt
    • Some SD model
      • 256x256 → 32x32 latents + diffusion (not VAE!) decoder
      • Might have SR diffusion

  • Stable Diffusion 3, Stability 2024

    Untitled

    • Diffusion transformer architecture, but with separate MLPs for text and image embeddings, but feeding into same attn. Reminiscent of MoEs (no routing).

      • Use ensemble of text embeddings (2 CLIPs and 1 T5), but aggressive dropout of 46.3% [dropping entire models?] means at inference time you can just load one model, don’t need all 3
      • Dropping T5 causes more misspellings
      • They parameterize a bunch of different sizes in the model in terms of number of DiT layers $d$

    • Uses “flow matching” new general approach

    • Rectified flow for straight paths

      • Start from ODE

        Untitled

      • They formulate all other design points within the same framework

      • Fortunately it’s also one of the simpler models, compared to EDM (Karras) and cosine (Nichol) and log-normal and uniform (actually this is the simplest)

      • Classic flows are curvy, need to take more steps than if the path was straighter:

        Untitled

        Untitled

    • Trained on ImageNet, CC12M. Recaptioned with CogVLM. Deduped with faiss.

    • Autoencoder increased hidden dimensions—important since it’s an upper bound on how good your generated images are

    • Pretrain on low res, fine tune on high res, same as in SD2

    • Pretrain → fine tune → DPO on 128 captions from Partiprompt (simple but realistic captions → high-quality images)

    • Comprehensive paper did grid search over design space

    • Paper tweets

    • hu-po

    • Rectified flow

    • Sander on diffusion distillation

  • Diffusion Transformers (DiT) 2023

    Untitled

    • Start with ViT
    • Operate in noised latent space
    • Patchify with strided 2x2 or 4x4 convolutions
    • Naive method
      • Must condition on timestep, class. So in addition to patch sequence, could concat timestep embedding and class.
      • But want text, not class. So concat the text embedding. But now it’s messy, doesn’t work as well. [Why?]
    • Also can do cross-attention on the text embeddings
      • Works well but computationally expensive because need another layer of attention (in addition to self-attention)
    • So, introduced adaLN-Zero: conditioning on timestep embedding TODO
      • Let conditioning define the scale/shift params, just via MLP
      • Additional conditioned scale determines how important the attention result is (the skip sum)
    • Transformers just more scalable
    • https://www.youtube.com/watch?v=fWUwDEi1qlA

  • VQVAE

  • ControlNet, 2023

    • Train with pair data, e.g. edges, depth maps, normal maps, p ose, style, etc.

      Untitled

      Untitled

  • DDIM

    • Actually feels cleaner and simpler than DDPM—people seem to prefer this formulation
    • Start with standard forward. WARNING: they change notations/variable names, somewhat confusing!

    Untitled

    • Pretty straightforward: rewind x_t to x_0 then jump forward to x_{t-1}

    Untitled

    Untitled

    • Can thus choose different sampling steps/skip steps

    • Altogether:

      Untitled

      • We’re taking small steps

      Untitled

      Untitled

      Untitled

      • Notice in below how: x_0 is weighted alpha, noise is weighted sigma, x_t is weighted the rest (both those subtracted from 1).

      Untitled

    • Also parameterizes the stochasticity (readded noise) with $\sigma$. $\eta$ is knob to control this: 0 means $\sigma=0$, 1 means $\sigma$ is its original DDPM value.

    • Code—notice how we parameterize the skip size:

      def ddim_step(x_t, t, noise, abar_t, abar_t1, bbar_t, bbar_t1, eta):
    vari = ((bbar_t1/bbar_t) * (1-abar_t/abar_t1))
    sig = vari.sqrt()*eta
    x_0_hat = ((x_t-bbar_t.sqrt()*noise) / abar_t.sqrt())
    x_t = abar_t1.sqrt()*x_0_hat + (bbar_t1-sig**2).sqrt()*noise
    if t>0: x_t += sig * torch.randn(x_t.shape).to(x_t)
    return x_t
@torch.no_grad()
def sample(f, model, sz, n_steps, skips=1, eta=1.):
    tsteps = list(reversed(range(0, n_steps, skips)))
    x_t = torch.randn(sz).to(model.device)
    preds = []
    for i,t in enumerate(progress_bar(tsteps)):
        abar_t1 = abar[tsteps[i+1]] if t > 0 else torch.tensor(1)
        noise = model(x_t,t).sample
        x_t = f(x_t, t, noise, abar[t], abar_t1, 1-abar[t], 1-abar_t1, eta)
        preds.append(x_t.float().cpu())
    return preds

    • fastai

  • Muse 2023

    Untitled

  • Cold diffusion: none of the technicalities matter, could use anything outside of Gaussian noise too (blurring, masking, etc.)

    Untitled

    Untitled

    • https://arxiv.org/pdf/2208.09392.pdf

  • Classifier free guidance

    • Issues with classifier guidance:

      • Can’t use pretrained classifier since it must be trained on noisy data
      • Hard to justify

    • Train a joint model over labeled images as well as unlabeled—basically, randomly decide whether to train the current sample with or without label (all absorbed into the UNet)

    • Start with classifier guidance term, then you’re left with both conditional generator p(x|y) and unconditional p(x)—we’re training both in the model:

      Untitled

    • Incorporate both labeled and unlabeled noise predictions in the sampling process

    • Over sample timesteps, linearly interpolate away from unconditional to conditional on the class

  • GLIDE

    • Condition over text instead of labels; otherwise similar to classifier free
    • Also introduced a super resolution model
    • Tried CLIP guided (more similar to classifier guidance), but didn’t work as well

  • VAE

    • Make autoencoders generative by adding Gaussian noise in latent space. Why? So you can chop off encoder, randomly sample Gaussian, and decode an image!
    • Add aux loss that encourages Gaussian mean/sd of 0/1.

  • VQ-VAE

    Untitled

    • Quantization: force the latent representation to use a codebook

    • Quantize latent space into closest (by say L2 norm) entry in learnable codebook (a certain set of vectors)—basically a set of fixed values.

      • Why? Learn discrete representations. Why? IDK

    • Add way to do backprop. In Torch, how to ensure the gradient of the input is copied over:

      Untitled

  • UNet: just a popular CNN architecture with wide resnet

    • Residual connection between corresponding down and up blocks
    • Timestep is encoded with transformer style positional embedding
    • Can also take context embedding
    • Can apply attention/transformer layers. Pixels are big/expensive, so do most transformers in lower dim latent space

    Untitled

    Untitled

  • Google Imagen, 2022

    • Simpler architecture than DALL-E (2), scaled up the right way

    • Basically: you start with a frozen LLM (they use T5) rather than CLIP, so you’re not limited to training on image,text pairs. Feed the text encoding to a text-to-image diffusion model (UNet).

      Untitled

    • TODO is the text encoding a fixed-size embedding? So the first model is T5 adapted as an embedding model somehow?

    • Use memory efficient U-Net: more model params at low res where things are more semantically processed

      • Architecture does use attention

    • Cascading DDPMs generate and super-res.

    • TODO something about using positional encoders in the timesteps, and also pooling the text encoding into the diffusion timesteps, and classifier free guidance with static/dynamic thresholding

  • Dall-E 2 aka unCLIP, OpenAI 2022

  • DALL-E, OpenAI 2021

  • Cascaded diffusion models, 2022

  • Stable Diffusion, Stability 2022