MayaTok-BPE is a Byte Pair Encoding (BPE) tokenizer implemented in Rust, focusing on performance and extensibility. It supports multithreaded training, persistent merges, and checkpoint saving, with Python bindings available.
From the README
MayaTok MayaTok is a Byte-Pair Encoding (BPE) tokenizer written in Rust. Built with performance and extensibility in mind. I made this project just because I wanted to study how Byte Pair Encoding Works. Version: 2.1.4 ⚡️ Features • Multithreaded training for fast vocab generation • Persistent merges • Checkpoint saving • Focus on raw speed — built for performance benchmarking 🚀 Installation Prerequisites • Rust (required for 'from source' approach) • Python 3.9+ (for Python bindings) PIP Installation From Source Use maturin for building wheels. Quick Start Using with Python To use MayaTok with Python: Output of the sample code above If you want to create your own Vocab If you...
Skills
- Rust
- Python
- bytepairencoding
- nlp
- pyo3-rust-bindings
- rust-lang
- tokenization