Surveys
Possibly of interest
- DenseFormers: just add skip connections from internal representations, not just x0
- Universal transformers, 2019
- Memformer
- Transformer XL (RMT)
- RWKV
- H4
- Multiple independent sample heads
Byte Latent Transformers, Meta 2024 (blog)
Let me walk through a concrete example to illustrate how BLT (Byte Latent Transformer) works, focusing on its key innovation of dynamic patching based on entropy.
Let's analyze how BLT processes the example shown in the paper for the sequence "Daenerys Targaryen is in Game of Thrones, a fantasy epic by George R.R. Martin."
Here's how it works step by step:
- Initial Byte Processing
- The sequence is first processed at the byte level by the Local Encoder, which looks at raw bytes instead of predefined tokens
- Entropy-Based Patching
- A small byte-level language model estimates the entropy (uncertainty) of predicting each next byte
- The model identifies high-entropy points where prediction is more difficult, for example:
- The "G" in "George" has high entropy because it's the start of a new name
- The "e" later in the name also shows high entropy
- These high-entropy points become patch boundaries when they exceed a threshold
- Dynamic Patch Creation
- Between these boundaries, the bytes are grouped into patches of varying sizes:
- "G" becomes a single-byte patch because it's a high-uncertainty prediction
- "eorge" becomes a larger patch since the following characters have lower entropy
- This creates efficient, context-aware groupings rather than fixed-size tokens
- Between these boundaries, the bytes are grouped into patches of varying sizes:
- Patch Processing
- The Latent Transformer processes these patches as units
- More compute is allocated to the difficult predictions (like the start of names)
- Less compute is needed for more predictable sequences
- Byte-Level Decoding
- The Local Decoder converts the patch representations back to bytes
- This maintains byte-level precision while being computationally efficient
The key insight is that BLT dynamically adjusts its processing granularity based on the complexity of the text, unlike fixed tokenizers that always process predefined units regardless of their predictability. This allows it to be both efficient (using larger patches for predictable sequences) and precise (breaking into smaller units when needed for complex predictions).
This approach helps solve issues with traditional tokenizers while maintaining computational efficiency - it can handle unusual inputs robustly since it works at the byte level, but still processes predictable sequences efficiently by grouping them into larger patches.
Mamba, 2023
- https://github.com/johnma2006/mamba-minimal
- See also all the open source Small models / scaling down LLMs
State space models
- Two views of SSMs: "linear RNN" or long convolutions
Memorizing transformers, 2022
-
With fixed context size of standard transformers, must chunk up attention when dealing with long docs:
-
Instead, allow attending over full history (past chunks), but with some sparser sort of attention, using knn search that’s O(log n) for total of O(n log n)
-
On first chunk, start stashing memory, but runs identically to standard attention
-
On next chunk, keep stashing, but now can attend over first chunk
-
Perform top-k attention over memory
-
Simply use a sigmoid gate (per attn head) to switch between context and memory. [Not sure what this gate is a function of—imagine it would be at least the token]
-
Overall architecture
-
Their initial TPU impl (bounded by 16GB/s mem bandwidth) handles 500k tokens. Next version will be “much larger”, can use external stores/web scale data/etc.
-
Improves perf—here, context window is 500 tokens
-
You can just add memory as fine tuning step! (Or upgrade to larger mem)
State space models
- Two views of SSMs: "linear RNN" or long convolutions
State space models
- Two views of SSMs: "linear RNN" or long convolutions
Transformers as search index
- Simply train (fine tune T5) on doc text → doc ID.
- Works much better than typical dual encoder embedding search (with contrastive training) on smallest corpus, slightly better on larger corpus
- Tried a lot of permutations: variations on inputs (stopwords, bag of words, etc.), outputting hierarchical IDs, inverted (docid2text) and bidir, random span masking,
- https://www.youtube.com/watch?v=qlB0TPBQ7YY
$\infty$-former, 2022
-
Idea: compress all the logits at the same dimension but across all tokens with curves (they use a bunch of RBFs, but imagine it’s a degree-k polynomial)—i.e. uses a fixed number of params (k polynomial degrees) to capture, say, the 3rd logit across all tokens (thus any number of tokens)
-
Can also re-encode [historical encoding ++ new data]—this starts feeling much more like an RNN/LSTM
-
Also need updated attention that attends over this. Augment your (”current”) discrete tokens with tokens from your RBFs history. (Note your attention is now non-square, and has different input vs output sizes.) This gives you keys. For values, integrate over Gaussian regions.
-
“Sticky memory” is simply adjusting the resampling from the historical re-encoding + concatenation. Instead of uniform, oversample the areas that have had a lot of attention (a lot of the Gaussians).
Longformer, Allen AI 2021
-
Introduces sliding window attention, sparse global attention
-
Global “pins” special tokens like [CLS] and [SEP]—anyone else can read this and it can read from anyone else. CLS for instance is the special reserved token for the class label output from a classification transformer.
Sparse transformer, 2019
- Reduce mem/time to $O(n \sqrt{n})$
