Cactus v2 Brings On-Device AI with Cloud Fallback to Any Arm Device

Roman and Henry from Cactus have shipped the biggest upgrade to their on-device inference platform, Cactus v2. The platform is designed to run AI models directly on your device, with a built-in confidence-based routing system that automatically hands off inference runs to the cloud when the on-device model is uncertain.

Key features of Cactus v2 include: - **Confidence-based routing**: The on-device model decides when it's unsure and sends the query to a cloud model for a more accurate answer. - **Converter for any PyTorch model**: You can convert and run virtually any PyTorch model on your device. - **Lossless 4-bit quantization**: Reduces model size without sacrificing accuracy (evals are available on the GitHub README). - **GPU acceleration**: Compatible devices can leverage GPU acceleration, starting with Apple Metal. - **Minimal RAM footprint**: Designed to run efficiently even on devices with limited memory. - **Runs on any Arm device**: Including iOS, Android, Mac, DGX Spark, Raspberry Pi, and more.

Performance benchmarks show that a Gemma 4 E2B class model runs at 169 tokens per second on an M5 Max, taking up only 2.7GB of disk space.

This matters because it means you can use AI tools without worrying about privacy or slow internet. For example, you could use an AI assistant that runs entirely on your phone, keeping your data local. Or you could use AI tools in places with spotty internet, like a hiking trip or a remote work location.

If you want to try it out, head to the Cactus GitHub page (github.com/cactus-compute/cactus) and follow the instructions to install it on your device. You can start by running a small AI model and seeing how it performs on your device.