• Meta learning, i.e. learning to learn. E.g. neural architecture search

  

MuP, Greg Yang TODO

• Paper
• By parameterizing your model with µP, you can cheaply find the optimal hyperparams on small models and directly transfer them to larger models

Twisted SMC

• Twisted SMC

Residual vector quantization (RVQ)

Residual Vector Quantization (RVQ) is a data compression technique commonly used in machine learning and signal processing, particularly for compressing large neural networks and high-dimensional vectors. Let me explain how it works:

The basic idea of RVQ is to approximate a vector using multiple codebooks in a sequential manner, where each codebook tries to encode the residual (error) left by the previous codebooks. Here's the step-by-step process:

1. First Codebook:
   • The original vectors are clustered (often using k-means) to create the first codebook
   • Each vector is approximated using its nearest centroid from this codebook
   • The difference between the original vector and this first approximation is called the residual
2. Subsequent Codebooks:
   • The residual vectors are clustered to create the next codebook
   • This process continues for a predetermined number of codebooks
   • Each subsequent codebook tries to capture the remaining error

The final approximation of a vector is the sum of the selected centroids from all codebooks. For example, if we use 3 codebooks, a vector x would be approximated as: x≈c1+c2+c3 where c_i is the selected centroid from the i-th codebook.

Key advantages of RVQ:

• Better compression ratio compared to regular vector quantization
• Maintains good reconstruction quality
• Efficient for similarity search and nearest neighbor lookups
• Particularly effective for compressing large language models and neural networks

RVQ has gained significant attention in recent years for model compression, especially in the context of making large language models more efficient and deployable on resource-constrained devices.

Applications

1. Audio AI/ML Models:
   • Audio tokens/neural codecs like EnCodec, SoundStream, and AudioLDM use RVQ to compress raw audio into discrete tokens
   • Helps convert continuous audio waveforms into discrete sequences that language models can work with
   • Crucial for music generation models like MusicLM and AudioCraft
   • The residual nature of RVQ helps capture both broad structure and fine details of audio