Cross-Family Speculative Decoding Boosts Polish LLM Performance on Apple Silicon

Researchers have developed a method to accelerate inference for Polish language models on Apple Silicon using cross-family speculative decoding. By extending the MLX-LM framework with Universal Assisted Generation (UAG), they enabled speculative decoding between models with mismatched tokenizers. The study focused on Bielik-11B-Instruct, a Mistral-based model, paired with three draft models to evaluate performance gains.

This advancement addresses a critical gap in speculative decoding, which has primarily been studied for same-tokenizer pairs on high-bandwidth GPUs. The new approach demonstrates that significant speedups are achievable even with consumer-grade unified memory and mismatched tokenizers. This could make large language models more practical for deployment on consumer devices, particularly for languages like Polish that benefit from specialized models.

The researchers plan to further optimize the UAG-Extended MLX-LM framework for additional language pairs and hardware configurations. Future work will also explore the impact of different draft model architectures on inference speed and quality. The study highlights the potential for speculative decoding to bridge the performance gap between high-end GPUs and consumer-grade hardware, making advanced AI capabilities more accessible.