This repository presents a small-scale language model designed to operate within a strict hardware constraint of 8GB VRAM. It aims to train and run efficiently on consumer hardware, prioritizing architectural control and system-level optimization.
From the README
MAYA Problem Statement Modern language models are designed to run on computationally expensive hardware. MAYA takes the opposite approach. The goal is to design and train a small language model that can run locally on everyday machines, with a strict hardware constraint of 8GB VRAM. --- Vision Build an efficient, small-scale language model that: • Trains on limited consumer hardware • Runs smoothly on the same hardware it was trained on • Does not rely on external API wrappers • Prioritizes architectural control and system-level optimization --- Technical Constraints • Target Hardware: 8GB VRAM (NVIDIA 4070 Laptop GPU) • Parameter Budget: ≤ 100 million parameters • Dataset:...
Skills
- Python