A GitHub repository showcasing a custom decoder-only transformer with a custom architecture and tokenizer. It includes features like sparse Mixture-of-Experts routing, distributed training, DPO alignment, and interpretability analysis. The repository also provides scripts for reproducibility and transparency.
From the README
From-Scratch LLM: Pretraining, MoE, Alignment, Interpretability, Compression, and Serving A decoder-only transformer built from scratch — custom architecture, custom BPE tokenizer, custom training loop — extended with sparse Mixture-of-Experts routing, distributed (FSDP) training, DPO alignment, KV-cached and padded-batch inference, causal interpretability analysis, two independent compression techniques, and a real batched-serving layer. Every claim below is backed by a script you can re-run and a log file with real numbers — including the failures and honest null results, which are reported alongside the successes rather than edited out. Live demo (Hugging Face Spaces) — type a prompt,...
Skills
- Python
