Applied LLMs | Notion

This is mostly about building things using LLMs, and about fine-tuning.

References
- Concise FT guide: https://medium.com/@mayaakim/complete-guide-to-llm-fine-tuning-for-beginners-c2c38a3252be
- Concise FT videos
  - https://www.deeplearning.ai/short-courses/finetuning-large-language-models/
  - Fine-tuning Large Language Models (LLMs) | w/ Example Code - YouTube
- HF Course to learn the libs
Leaderboards (caveat issues)
- TODO
Applications
Composition tools and libraries
- https://github.com/hwchase17/langchain
  - Overengineered, but consolidates up-to-date techniques
    - Prompt templates = string interpolation, LLMs are just thin API wrappers around LLMs, chains = function composition
  - Agents: dynamic chain with overarching prompt pattern that registers tools
  - When processing docs, use “combine chains”: either “stuff” (concat), map-reduce, “refine” (fold), and map-rerank (score each chunk)
- https://github.com/jerryjliu/llama_index: Easier than LangChain for IR/QA tasks (source)
- https://github.com/deepset-ai/haystack
- https://github.com/chroma-core/chroma: simple document embeddings database, can use pluggable models, has in-mem (duckdb) mode for testing
- https://github.com/TimDettmers/bitsandbytes: The bitsandbytes is a lightweight wrapper around CUDA custom functions, in particular 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization functions.
- xformers: Meta’s building blocks for transformers
- https://einops.rocks/ - Flexible and powerful tensor operations for readable and reliable code. Supports numpy, pytorch, tensorflow, jax, and others.
- huggingface
  - datasets
  - transformers[sentencepiece]: AutoTokenizer, AutoModel*, Trainer, DataCollatorWithPadding, etc
  - evaluate: load/compute metrics
  - https://pypi.org/project/accelerate/: avoid hardcoding device details
  - peft
- gradio, streamlit: for building simple web UIs in python
RLHF how-to’s
- https://huggingface.co/blog/trl-peft
Implementations
- https://github.com/Lightning-AI/lit-gpt
- mingpt
- nanogpt
- ‣: minimal code to train 1-10B model
- https://github.com/lucidrains
- Llama and Alpaca
  - 2048 tokens
  - https://github.com/ggerganov/llama.cpp
  - https://github.com/antimatter15/alpaca.cpp
  - https://github.com/cocktailpeanut/dalai frontend UI
  - https://github.com/nsarrazin/serge frontend UI
  - https://news.ycombinator.com/item?id=35378683
Other modality models
- https://news.ycombinator.com/item?id=35352452
- Proprietary research
  - https://news.ycombinator.com/item?id=35365399
- Services
  - https://news.ycombinator.com/item?id=35358873
Pretrained embeddings
- GPT embeddings
- all-MiniLM-L6-v2: fastest
- all-mpnet-base-v2: best quality
- “The OpenAI text similarity models perform poorly and much worse than the state of the art (all-mpnet-base-v2 / all-roberta-large-v1).” (source)
- List from Chroma
File formats
- Pytorch .bin: most uploaded to HF in this format which pytorch dumps (sometimes called .pk e.g. by llama2.c)
- safetensors: HF lib, instead of normal (unsafe) pickle files for pytorch tensors
- GGUF: llama.cpp format. TheBloke’s quantized outputs are in this format. Conversion script.
  
  It is a successor file format to GGML, GGMF and GGJT, and is designed to be unambiguous by containing all the information needed to load a model. It is also designed to be extensible, so that new features can be added to GGML without breaking compatibility with older models.
- GPTQ is the same quanitized file format for models that runs on GPU
- Quote
  Source
  - Llama.cpp (by Georgi Gerganov)
    - GGUF (new)
    - GGML (old)
  - Transformers (by Huggingface)
    - bin (unquantized)
    - safetensors (safer unquantized)
    - safetensors (quantized using GPTQ algorithm via AutoGPTQ)
  - AutoGPTQ (quantization library based on GPTQ algorithm, also available via Transformers)
    - safetensors (quantized using GPTQ algorithm)
  - koboldcpp (fork of Llama.cpp)
    - bin (using GGML algorithm)
  - ExLlama v2 (extremely optimized GPTQ backend for LLaMA models)
    - safetensors (quantized using GPTQ algorithm)
  - AWQ (low-bit quantization (INT3/4))
    - safetensors (using AWQ algorithm)
  - GGUF contains all the metadata it needs in the model file (no need for other files like tokenizer_config.json) except the prompt template
  - *llama.cpp has a script to convert *.safetensors model files into .gguf
  - Transformers & Llama.cpp support both CPU, GPU and MPU inference
- Quote
  Source
  - GGML: used with llama.cpp, outdated, support is dropped or will be soon. cpu+gpu inference
  - GGUF: "new version" of the GGML file format, used with llama.cpp. cpu+gpu inference. offers 2-8bit quantization
  - GPTQ: pure gpu inference, used with AutoGPTQ, exllama, exllamav2, offers only 4 bit quantization
  - EXL2: pure gpu inference, used with exllamav2, offers 2-8bit quantization
Generic inference platforms
- replicate
- huggingface
- textsynth
- nlpcloud
- banana.dev
- forefront has some open source models for playing with
Quantization
- Quality: “the q4 34B code llama (which is used here) performs quite poorly in my experience. Using a high quantised larger model gives you an unrealistic impression that smaller models and larger models are roughly equivalently capable… but it’s a trade off. The larger codellama model is categorically better, if you don’t lobotomise it” (HN)