40%

Complete your profile to find better job opportunities

MayaTok-BPE
GitHubRepository

MayaTok-BPE

MayaTok is a Byte Pair Encoding based Tokenizer.

  • 1 stars
  • 0 forks
  • Rust language
View on GitHub
Published
17 Jul 2025
Updated
17 Apr 2026
AI summary

MayaTok-BPE is a Byte Pair Encoding (BPE) tokenizer implemented in Rust, focusing on performance and extensibility. It supports multithreaded training, persistent merges, and checkpoint saving, with Python bindings available.

From the README

MayaTok MayaTok is a Byte-Pair Encoding (BPE) tokenizer written in Rust. Built with performance and extensibility in mind. I made this project just because I wanted to study how Byte Pair Encoding Works. Version: 2.1.4 ⚡️ Features • Multithreaded training for fast vocab generation • Persistent merges • Checkpoint saving • Focus on raw speed — built for performance benchmarking 🚀 Installation Prerequisites • Rust (required for 'from source' approach) • Python 3.9+ (for Python bindings) PIP Installation From Source Use maturin for building wheels. Quick Start Using with Python To use MayaTok with Python: Output of the sample code above If you want to create your own Vocab If you...

Skills

  • Rust
  • Python
  • bytepairencoding
  • nlp
  • pyo3-rust-bindings
  • rust-lang
  • tokenization

More work by Ashwin Rajhans