Let's Learn Byte Latent Transformer - A New Direction in LLM Development
Are you curious about why every large language model (LLM) relies on tokenizers like BPE, despite their inherent flaws? What if there's a better way?
Join us for this meetup to explore Byte Latent Transformer (BLT) - a new architectural paradigm for LLMs that eliminates fixed tokenizers by learning directly from raw bytes using dynamic, context-aware "patches".
In this concise and accessible session, we will take a rapid journey through the key breakthroughs that are making "tokenizer-free" language models a practical reality:
- The Core Problem: We'll quickly cover the fundamentals of subword tokenization (BPE) and its fundamental limitations—such as the vocabulary bottleneck and rigid compute allocation—that motivate the search for better alternatives.
- The Foundational Architecture: We'll discuss the groundbreaking paper, "Byte Latent Transformer: Patches Scale Better Than Tokens" (Meta AI, 2024). Discover how BLT intelligently groups bytes into variable-length patches based on entropy, achieving performance parity with Llama 3 while using up to 50% fewer inference FLOPs and unlocking a new axis for model scaling.
- The Practical Revolution: We'll examine how the follow-up work, "Bolmo: Byteifying the Next Generation of Language Models" (AllenAI, 2025), makes this technology instantly accessible. Its innovative "byteifying" method converts an existing subword model into a byte-level one for less than 1% of the typical pre-training cost, significantly outperforming prior byte-level models like BLT in specific tasks.
- Ecosystem & Future: We'll take a brief look at active research repositories, including the BitStream Foundation Models Research repo, to see how the community is building upon these foundations and where this new paradigm is headed.
This meetup is for anyone who wants to understand the substance behind the "tokenizer-free LLM" trend and what the next evolutionary step for large models might be. No advanced expertise is required-we will explain all core concepts from the ground up.
We will not cover all papers, but for people that are just interested we mentioning other papers important for BLT topic.
