• Surveys
  • https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/
• Possibly of interest
  • DenseFormers: just add skip connections from internal representations, not just x0
  • Universal transformers, 2019
  • Memformer
  • Transformer XL (RMT)
  • RWKV
  • H4
  • Multiple independent sample heads
• State space models
  • Two views of SSMs: "linear RNN" or long convolutions
• Recurrent Memory Transformer, 2022

Byte Latent Transformers, Meta 2024

• blog
• code
• Yannic
• Gabriel

My summary: key idea is to define “tokens” “dynamically”. Define patches as groups of bytes that are runs of low-entropy and end in high-entropy byte, as determined by a small byte level LM. Once you have these patches, still need to figure out their embeddings/encoding. This is done by a “Local Encoder” that is just N cross attention layers over the byte level embeddings (also produced by that small byte level LM).

Let's analyze how BLT processes the example shown in the paper for the sequence "Daenerys Targaryen is in Game of Thrones, a fantasy epic by George R.R. Martin."

Here's how it works step by step:

1. Initial Byte Processing
   • The sequence is first processed at the byte level, which looks at raw bytes instead of predefined tokens
2. Entropy-Based Patching
   • A small byte-level language model estimates the entropy (uncertainty) of predicting each next byte
   • The model identifies high-entropy points where prediction is more difficult, for example:
     • The "G" in "George" has high entropy because it's the start of a new name
     • The "e" later in the name also shows high entropy
   • These high-entropy points become patch boundaries when they exceed a threshold
3. Dynamic Patch Creation
   • Between these boundaries, the bytes are grouped into patches of varying sizes:
     • "G" becomes a single-byte patch because it's a high-uncertainty prediction
     • "eorge" becomes a larger patch since the following characters have lower entropy
     • This creates efficient, context-aware groupings rather than fixed-size tokens
4. Patch Representation
   • It groups bytes into patches based on entropy
   • It uses cross-attention to create patch representations from byte representations
   • The cross-attention is part of the Local Encoder module
5. Patch Processing
   • The Latent Transformer processes these patches as units
   • More compute is allocated to the difficult predictions (like the start of names)
   • Less compute is needed for more predictable sequences
6. Byte-Level Decoding
   • The Local Decoder converts the patch representations back to bytes
   • This maintains byte-level precision while being computationally efficient