This is an automated archive made by the Lemmit Bot.
The original was posted on /r/machinelearning by /u/bminixhofer on 2025-04-24 10:34:11+00:00.
Hello r/MachineLearning !
I’ve been experimenting with a method called ALM to distill language models across tokenizers. This enables, for example, transferring LLMs to a new tokenizer and distilling knowledge from a model with one tokenizer into a model with a different tokenizer (see our paper for details).
I’ve released tokenkit, a library implementing ALM among other methods, to make this easy to use.
One neat application of ALM is distilling subword-based LLMs into byte-level models. I've applied this to two instruction-tuned models:
- Gemma2-2B-IT-Byte:
- Llama3-2-3B-IT-Byte:
Even though the distillation phase is very short (just 1.2B bytes ≈ 330M subword tokens), the models perform competitively (for example 57.0% MMLU of the byte-level Llama vs. 62.4% MMLU of the original Llama3-3B-Instruct).
This approach opens up an interesting direction: we can potentially keep subword tokenization for pretraining (to still squeeze as much text into the model in as little time as possible), but then change to a more user-friendly tokenization afterwards.
These models aren’t yet optimized for efficiency, but if you would add self-speculative decoding plus a BLT/DTP-style hierarchical architecture and/or linearized attention, they might also be able to replace subword-based models when speed matters.
If you want to train your own models, this guide on tokenizer transfer via tokenkit should make it easy. The model cards of the transfers above also contain the exact command used to train them. I’ve been training on fairly limited hardware, so effective transfer is possible even in a (near) consumer-grade setup.
I'd love to get feedback on the method, the models, or tokenkit itself. Happy to discuss or answer questions!
