ViVIT
Meta Chameleon
CogVLM, 2023
Llava-Interactive: visual prompting
Llava-Plus:
Obsidian: 3B, MLLM vision, based on StableLM
- Pretrain takes around 2.5 hours for Obsidian-3B-V0.5 on 4x A100 (80G), at 336px for the vision module. Training script with DeepSpeed ZeRO-2: pretrain.sh.
TinyGPT-V: 2.7B, MLLM vision, based on Phi 2
- Based on MiniGPT-4
- They also train a post-LM normalization layer and LoRA adapters
- (3) TinyGPT-V: Small but Mighty Multimodal Large Language Model - YouTube
- Training in multiple stages
-
It stands out by requiring merely a 24G GPU for training and an 8G GPU or CPU for inference.
-
We use a single 3090 GPU(24G).
Stage 1 About 8 hours of training, Stage 2 About 4 hours, Stage 3 About 20 mins, Stage 4 About 8 hours or more.
Currently Stage 4 is still in a testing state, as it is not actually performing well (although it is able to perform well on the assessment results), and it is recommended that you only study up to Stage 3.
BakLLaVA: By implementing better base models, modified training process, custom datasets, and significant architecture changes to the original LLaVA implementation.
LLaVA: 7B, MLLM vision
-
Vision encoder = CLIP/BLIP. LM = Llama. First train the projection W. Then unfreeze LM, train ‘end-to-end’ (but vision encode always frozen).
LLaVA: LLaVA utilizes the LLaMA model, renowned for its efficacy in open-source language-only instruction-tuning projects. For visual content processing, LLaVA relies on the pre-trained CLIP visual encoder ViT-L/14, which excels in the realm of visual comprehension. The encoder extracts visual features from input images and connects them to language embeddings through a trainable projection matrix. This projection effectively translates visual features into language embedding tokens, thereby bridging the gap between text and images.
-
LLaVA-1.5 achieves SoTA on 11 benchmarks, with just simple modifications to the original LLaVA, utilizes all public data, completes training in ~1 day on a single 8-A100 node, and surpasses methods that use billion-scale data.
MiniGPT-4: 7B, MLLM vision
- MiniGPT-4: How does MiniGPT-4 work? MiniGPT-4 aligns a frozen visual encoder (EVA ViT → BLIP-2 Q-Former) with a frozen LLM called Vicuna using just one projection layer.
- The visual encoder consists of pretrained ViT and Q-Former models that are connected to an advanced Vicuna large language model via a single linear projection layer
BLIP-2 / Q-Former TODO
BLIP TODO
CLIP (Contrastive Language-Image Pre-Training) is a binary classifier trained on (image, text) pairs and negative pairs.
- You have text encoder and image encoder, and do contrastive loss within each batch
- does not use existing models; it fully pretrains both vision and text transformers
- Can use to predict next most relevant text, and thus use for downstream tasks such as classification similarly to the zero-shot capabilities of GPT-2 and 3
- OpenCLIP: includes larger and independently trained CLIP models up to ViT-G/14
- outline
- get text embedding
- get image embedding
- project each to same dimensional space (objective is for image / text to map to same representation)
- Linear → GELU → Linear → dropout → residual with projected → layer norm
- all-pairs similarity is
I_e @ I_t.T, also scale by temperature $e^t$ - labels is
arange(N) loss = (crossent(sim, labels, axis=0) + crossent(sim, labels, axis=1)) / 2
- OpenAI’s CLIP explained! | Examples, links to code and pretrained model - YouTube
- OpenAI CLIP: ConnectingText and Images (Paper Explained) - YouTube
- OpenAI CLIP - Connecting Text and Images | Paper Explained - YouTube
KOSMOS-1: multimodal (image and text) (HN)
- arbitrarily interleaved text and images, image-caption pairs, and text data
- https://medium.com/@amol-wagh/whats-new-in-gpt-4-an-overview-of-the-gpt-4-architecture-and-capabilities-of-next-generation-ai-900c445d5ffe